Downloading a web page with telnet

[big chopper]

Hey all,
Anyone here remember the commands needed to download a web page with telnet?

TAI.
--bc,

[TheCotMan]

Hey all,
Anyone here remember the commands needed to download a web page with telnet?

Did you try:

Really basic selection without special extra options can work:

$ telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
GET /PATH/TO/TXT/OR/HTML/FILE.txt.or.html HTTP/1.0^M^M

Though, you should be able to use ^J^J instead of ^M^M or press the "RETURN" key twice.

But why telnet? Why not use "wget" or "sitecopy" ? They have many more features for conversion of links to absolut/dynamic name of referring site (for check of internal referral) and web client browser version.

wget and sitecopy will also work on servers that have multiple domains sharing a single IP address.

[Added content:]
More advanced features when connecting:

$ telnet 127.0.0.1 80
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
GET /PATH/TO/TXT/OR/HTML/FILE.txt.or.html HTTP/1.0
Host: NAMEOFHOST.YOU.WANT.TO.CONNECT.IF.MULTI.HOMED.TLD
Accept: text/html, text/plain, text/*
Accept: ANY_OTHER_MIME_TYPES_COMMA_SEPARATED
Accept: MULTIPLE_ACCEPT_LINES_OK
Accept-Language: en
User-Agent: I-AM-CORNHOLIO/1.0 IP-FOR-MY-TCP NCSA-Mosaic^M^M

You can end each line with a ^M or ^J, but you signify you are done entering options with a ^M^M or ^J^J or try pressing the return key twice.
Other options exist. Related docs:
http://www.w3.org/Protocols/rfc1945/rfc1945
http://www.w3.org/Protocols/rfc2616/rfc2616.html

[big chopper]

For what ever reason, that doesn't work. I need to specify a full URI
telnet http://www.domain.com 80
GET http://www.domain.com/index.php
--bc,

[TheCotMan]

For what ever reason, that doesn't work. I need to specify a full URI
telnet http://www.domain.com 80
GET http://www.domain.com/index.php
--bc,

It is likely they rely upon the client to pass the name of the server for which they want to see content. Apache's option for virtualhost uses this method to allow a single IP address to run a single web server that serves many different domains. However,this means any client not specifying the "Host:" option will likely fail to get its content.

Your telnet client appears to have modified your "http://www.domain.com" to "www.domain.com" for you. It is good that the server respected the GET as you formatted it without a specification of what version HTTP you were using and the conventional "Host:" option, but instead relying upon the host as handed int he GET..:-)

(I added the content on some of the extra options to the first reply as well as the RFC that relate to this. This is stated so people know you replied before the [added content] in my first reply was included and it was my fault for not editing faster :-)

[hackajar]

For what ever reason, that doesn't work. I need to specify a full URI
telnet http://www.domain.com 80
GET http://www.domain.com/index.php
--bc,

Is this going to be part of a script? Why not use curl/lwp/fetch/wget?

[big chopper]

Is this going to be part of a script? Why not use curl/lwp/fetch/wget?

OK, my actual goal is to use telnet to get mail from a pop3 mail server. I wanted to download a web page first, for a basic understanding of telnet.

That (telnet) exercise was groundwork for a PHP/CURL script that programatically reads/deletes email from a POP3 mail server. I had to learn the POP3 commands first, telnet was a good place to experiment.

Thanks everyone for your help.
--bc,

[hackajar]

Pop3

1.) Just ask, this question could have been aswered days ago

2.) These are the droids you are looking for

3.)
%telent 127.0.0.1 110

>user myuser
USER OK
>pass mypass
LOGGED IN
>list
1 512
2 86
3 9806
> top 1 512
Header....bla bla bla
<Subject>your V|@gra request
Thankyou for buying our great product!!!


KEY:
% = shell command line (remove % and type that command)
> = POP3 Command line (remove > and type that command)
Anything else is returned results

[hackajar]

php pop3

by hand....

<?php

$Array = array("user"=>"user myuser", "pass"=>"pass pass");

//We love d socket!
$socket = socket_create(AF_INET, SOCK_STREAM, SOL_TCP);
//We open d socket!
socket_connect($socket, "127.0.0.1", "110");
//Send username
socket_write($socket, $Array['user'], strlen($Array['user']));
//Check result came back
while ($out = socket_read($socket, 2048)) {
$result .= $out;
}
//Send Pass
if($result)socket_write($socket, $Array['pass'], strlen($Array['pass']));
$result = false;
while ($out = socket_read($socket, 2048)) {
$result .= $out;
}
//Check Auth OK
if(!preg_match("/OK/", $result)) die("Authentication faild!");

//Start Getting Mail

bla bla bla

?>

[big chopper]

by hand....
<?php
<snip> ... </snip>
?>

Yep, that's pretty much what I did.

When you execute a POP3 LIST command, it returns a message id and a size.
LIST
+OK
1 2400
2 5432
(etc...)

Initially I used the returned size value to let me know when the entire message was read. In actual practice, however, I found the size value returned by LIST has little resemblance to the actual size of the message. (Is the value in octal or something?) A better technique is to use the POP3 end of multi-line message indicator (a ".\n")--which is what I should have done from the beginning, I suppose.

--bc,