Hand-Crafted HTTP with Telnet: Exploring the protocol
If we want to make an HTTP request by hand, the most important place to start is Section 5 of RFC 2616, which explains how a client makes a request to a server. The format is, roughly:
Request-Line Header line(s) CRLF (empty line) Message body (optional; for example if POSTing data)What does Request-Line mean? The spec explains. Here it is, verbatim, where "SP" means whitespace.
Method SP Request-URI SP HTTP-Version CRLFYou're already familiar with the possible values for "Method" (GET/POST/PUT/etc). Request-URI is basically the url path that you are requesting (i.e., if you are asking for the root url, it is "/"). The http version is going to be 1.1, which is written as "HTTP/1.1".
So, to request the root URL at the server via GET, you would use this as the request line:
GET / HTTP/1.1So far so good. The next part of the request is the headers. The full range of possible headers are described in Section 14. Suffice it to say for now that the headers are where you can include additional metadata about your request, such as your client's user-agent, the content-type, the language, caching options, and so on. The only one that is required by HTTP/1.1 is the "Host" header, which must match the address of the server that you are requesting the resource from. To request the homepage from google, then, this is the HTTP request text we would need to use.
GET / HTTP/1.1 Host: google.com CRLF (empty line)For example, say we have this very simple "Hello, World" node server, running on localhost at port 5000:
Here is what it looks like to connect via telnet and request the url "/":
You may notice a couple things. First of all, as soon as an empty line (CRLF) is entered after the last header, we get back the server response. HTTP also defines a specific way for an HTTP server to respond, and it is fairly similar to the format of a client request. There's an initial line that includes the HTTP version and the status code ("HTTP/1.1 200 OK"), then some header lines, then a blank line (CRLF), then the message.
But why does it say "c" before the "Hello, World" text and "0" afterward? That wasn't part of our server's response, was it? That looks like random junk. Well, if you look at the headers again you'll notice that one of them specifies "Transfer-Encoding: chunked". Chunked Transfer Encoding wasn't in the HTTP/1.0 spec but has been added to the HTTP/1.1 spec. Chunked transfer encoding allows a server to send responses in, well, chunks, rather than doing it all at once. The HTTP spec says that before each chunk the server must send a line with the length of the following chunk (in hexadecimal), and a line with "0" on it after the chunk. If you count up the letters in "Hello, World" you'll get 12, which happens to equal "c" in hexadecimal.
Finally, if you're paying close attention you may further notice that the telnet session didn't end after the server sent us the "Hello, World" response. This is due to another change that was introduced in HTTP/1.1: persistent connections by default. In order to cut down on the latency overhead that is incurred by the TCP connection handshake, in HTTP/1.1 all connections are, by default, kept alive so that the client can request another resource without needing to re-connect. The server indicates that it intends to keep the connection alive by adding the "Connection: Keep-Alive" header to its response.
If we do not want this behavior, we can explicitly tell the server to close the connection by adding the "Connection: Close" header to our initial request. Here's an example of that:
This time, the connection is immediately dropped by the server after it finishes responding.
Remember how I mentioned that the Host header is required by HTTP/1.1? Why would it be required? Shouldn't the server know what Host is being requested? The answer is, surprisingly, that it does not. The HTTP/1.0 standard assumed that one IP address was equivalent to one server, but we now have the concept of a multi-homed server, which just means that a single server at a single IP address can serve multiple domains. Without the Host header, a server running HTTP/1.0 can't tell which of the possible hosts should handle the incoming request.
In the follow-up to this post I'll look into how to hand-craft an HTTP request that actually sends data to the server. If you've ever wondered how the data in a form actually gets to the server, we'll go into that.