Skip to content
Apr 6 14

Hand-Crafted HTTP with Telnet: Exploring the protocol

by cory
I've been reading Ilya Grigorik's truly excellent High Performance Browser Networking lately and as I went through the HTTP section I realized that there were some gaps in my understanding of the protocol.

Fortunately, the HTTP 1.1 protocol is text-based, so we can create HTTP requests very easily by hand using telnet! The HTTP version 1.1 protocol is laid out in RFC 2616. It can look fairly intimidating at first but it's actually quite readable. At a high level, HTTP, which stands for Hypertext Transfer Protocol, outlines a mechanism for clients and servers to communicate to transfer resources (the R in URL) from one server to another. Your web browser is an HTTP client and it retrieves resources (HTML documents, CSS files, javascript files, images, etc) from HTTP servers.
If we want to make an HTTP request by hand, the most important place to start is Section 5 of RFC 2616, which explains how a client makes a request to a server. The format is, roughly:
Request-Line
Header line(s)
CRLF (empty line)
Message body (optional; for example if POSTing data)
What does Request-Line mean? The spec explains. Here it is, verbatim, where "SP" means whitespace.
Method SP Request-URI SP HTTP-Version CRLF
You're already familiar with the possible values for "Method" (GET/POST/PUT/etc). Request-URI is basically the url path that you are requesting (i.e., if you are asking for the root url, it is "/"). The http version is going to be 1.1, which is written as "HTTP/1.1".

So, to request the root URL at the server via GET, you would use this as the request line:
GET / HTTP/1.1
So far so good. The next part of the request is the headers. The full range of possible headers are described in Section 14. Suffice it to say for now that the headers are where you can include additional metadata about your request, such as your client's user-agent, the content-type, the language, caching options, and so on. The only one that is required by HTTP/1.1 is the "Host" header, which must match the address of the server that you are requesting the resource from. To request the homepage from google, then, this is the HTTP request text we would need to use.
GET / HTTP/1.1
Host: google.com
CRLF (empty line)
For example, say we have this very simple "Hello, World" node server, running on localhost at port 5000:

Here is what it looks like to connect via telnet and request the url "/":

telnetsimple
You may notice a couple things. First of all, as soon as an empty line (CRLF) is entered after the last header, we get back the server response. HTTP also defines a specific way for an HTTP server to respond, and it is fairly similar to the format of a client request. There's an initial line that includes the HTTP version and the status code ("HTTP/1.1 200 OK"), then some header lines, then a blank line (CRLF), then the message.

But why does it say "c" before the "Hello, World" text and "0" afterward? That wasn't part of our server's response, was it? That looks like random junk. Well, if you look at the headers again you'll notice that one of them specifies "Transfer-Encoding: chunked". Chunked Transfer Encoding wasn't in the HTTP/1.0 spec but has been added to the HTTP/1.1 spec. Chunked transfer encoding allows a server to send responses in, well, chunks, rather than doing it all at once. The HTTP spec says that before each chunk the server must send a line with the length of the following chunk (in hexadecimal), and a line with "0" on it after the chunk. If you count up the letters in "Hello, World" you'll get 12, which happens to equal "c" in hexadecimal.

Finally, if you're paying close attention you may further notice that the telnet session didn't end after the server sent us the "Hello, World" response. This is due to another change that was introduced in HTTP/1.1: persistent connections by default. In order to cut down on the latency overhead that is incurred by the TCP connection handshake, in HTTP/1.1 all connections are, by default, kept alive so that the client can request another resource without needing to re-connect. The server indicates that it intends to keep the connection alive by adding the "Connection: Keep-Alive" header to its response.

If we do not want this behavior, we can explicitly tell the server to close the connection by adding the "Connection: Close" header to our initial request. Here's an example of that:

telnetsimple-close

This time, the connection is immediately dropped by the server after it finishes responding.

Remember how I mentioned that the Host header is required by HTTP/1.1? Why would it be required? Shouldn't the server know what Host is being requested? The answer is, surprisingly, that it does not. The HTTP/1.0 standard assumed that one IP address was equivalent to one server, but we now have the concept of a multi-homed server, which just means that a single server at a single IP address can serve multiple domains. Without the Host header, a server running HTTP/1.0 can't tell which of the possible hosts should handle the incoming request.

In the follow-up to this post I'll look into how to hand-craft an HTTP request that actually sends data to the server. If you've ever wondered how the data in a form actually gets to the server, we'll go into that.
Feb 24 14

How does Release Date Affect Oscars Best Picture Wins?

by cory
I was looking for an excuse to play around with d3, and since the Oscars are coming up next weekend, I put together a little d3-powered chart showing the number of best picture wins by US release month.

If you're just here for the answer to the title question, it is: Quite a bit. About 25% of the winners were released in December, and over half were released in the 3 months at the end of the year.

I used the enjoyable and optimistic-sounding Cheerio npm module (DOM traversal/selection a la jQuery, but in node.js) to scrape , and followed along with Mike Bostock's gentle d3 bar chart tutorial to put the data together in jsbin.

Here are the results:
JS Bin

I was hoping to do a little more explanation of how this was generated, but I'll save that for a future post. The code is pretty quick-and-dirty, anyway. If you'd like to do something with the cleaned-up data, the movies.json file is on github.

What did I learn? Well, not a whole lot. My initial expectation was that the movies were released toward the end of the year. In retrospect, even though it seemed official, the Oscars.org best picture winners page was probably not the best one to use. I should have been clued in by the raw php code in the source HTML there. It seemed marginally easier to parse than Wikipedia, though, so I went with it, but the extra time spent cleaning up data by hand offset any gains.

I was a bit surprised to realize how complicated release dates can be. The same movie gets released multiple times in the same country, and also in other countries, and also sometimes gets re-released. I had to put together some simple heuristics to choose the most appropriate date from lists like this one for The Sting.

Not-too-surprising fact: Lots of movies sneak in a limited release somewhere late in December but aren't available widely in the US until the following year (like Million Dollar Baby, which had a limited release in mid-December and the wide release about 7 weeks later).

Most surprising thing: Casablanca appears to be the only film listed to win for a year other than the one in which it was released. According to IMDB it was released in 1942 yet somehow won the 16th Academy Awards in 1943. Perhaps the notion of 'release date' is even looser than I realized.
Feb 17 14

Ember Controller versus ObjectController

by cory
When doing Ember.js training, I commonly see confusion surrounding the different controller types that Ember offers. In this post I'll show a couple examples that illustrate the differences between the two.

Every time you you display a template in an ember application, you can use Handlebar's interpolation expressions to display property values in the rendered HTML (often called data binding). The source of these property values is the controller that Ember is using as the context for the handlebars template. Every time you display a template, its context is bound to some controller, either one that you have explicitly created or one that ember has auto-generated for you (you can see the generated controllers logged to the console if you create your app with "LOG_ACTIVE_GENERATION: true").

Ember.Controller



For example, if you have the template "Hello, {{name}}.", and a controller with the property `name: "Cory"`, then you'll see "Hello, Cory." You can see that below

Ember Starter Kit

Any property that is on the controller can be displayed by referencing it in the template. Properties can be simple key-value pairs as above, or the result of a function. When using a function as a property for display, you must add a `.property()` call to the end of the function definition. Pass any dependent keys as arguments to the `property()` call. For example, we can make a more sophisticated controller and template like this:

App.IndexController = Ember.Controller.extend({
  greeting: function(){
    if (this.get('isFancy')) {
      return "Greetings";
    } else {
      return "Hello";
    }
  }.property('isFancy'),
  name: "Cory"
});

Ember.TEMPLATES['index'] = Ember.Handlebars.compile("{{greeting}}, {{name}}");


This uses the controller's property `isFancy` to determine what type of greeting to display. Since we haven't specified `isFancy` explicitly yet, it will be falsy and the output of the template will still be "Hello, Cory".

If we explicitly define `isFancy` to be true, however, the output changes to "Salutations, Cory."

So far these examples all show usage of the standard controller, which we extended from `Ember.Controller`. There's another type of controller Ember offers, called the `ObjectController`, which we'll look at now.

Ember.ObjectController

A typical use case in Ember is to load in model data via an ajax call and display some of it to the user in the template. Typically the data is loaded in and returned from the `model` hook in the route. We'll make that change now, explicitly defining a route in our code instead of letting Ember auto-generate it as before. (If you have `LOG_ACTIVE_GENERATION` set to true, you'd see something like "generated -> route:index" in the console, pointing out that the index route had been automatically generated.)

App.IndexRoute = Ember.Route.extend({
  model: function(){
    var author = Ember.Object.create({name: 'Cory'});
    return author;
  }
});


This example isn't loading in data via ajax, but it will have the same result regardless. The model hook just has to return an object or a promise that resolves to an object.

Since we are planning on displaying model data in our template, it's time to change our controller to an ObjectController. Ember will automatically set the "model" property on our controller to the return value of the matching route's `model` hook. In this case we return the `author` object from `IndexRoute`'s `model` hook, so it will be set to the "model" property of the IndexController. We can now wire up our original starting template using the ObjectController instead. Knowing that its "model" property is the author object, along with what we learned earlier about property functions on a controller, we could get the same behavior we started with by doing this:

App.IndexRoute = Ember.Route.extend({
  model: function(){
    var author = Ember.Object.create({name: 'Cory'});
    return author;
  }
});

App.IndexController = Ember.ObjectController.extend({
  name: function(){
    return this.get('model.name');
  }.property('model.name')
});

Ember.TEMPLATES['index'] = Ember.Handlebars.compile("Hello, {{name}}");


This feels a little awkward and luckily Ember has anticipated this use case already and offers us a much terser option. The ObjectController is so named because it proxies property lookups to its `model` property (its "object", so to speak). When our template references the `name` property, the ObjectController's default behavior is to look up the `name` property on its model, exactly as we have just manually done above. This means we can remove all of this code and instead say:

App.IndexController = Ember.ObjectController.extend({});


The output is unchanged. Ember Starter Kit

Proxying Properties to the ObjectController's Model

Earlier I said that the ObjectController proxied property lookups to its model, and although this is correct there's an important caveat to mention: The ObjectController will first check whether the requested property has been defined on it. Only if the property has not been defined will it look up the property value on the model.

As an example, consider setting a `name` property on the ObjectController explicitly. What would happen in that case?

App.IndexController = Ember.ObjectController.extend({
  name: "impostor"
});
Ember Starter Kit

The index controller will intercept the lookup of the `name` property and return its own `name` property, effectively masking the `name` property on the model. This usually isn't desired behavior and can be the source of some confusion. Most of the time we will want to add properties to an ObjectController it will be to display different data that is not the model's direct concern. A very common example of this would be to display different items in the interface when the controller has been switched into "edit mode" (i.e., had its "isEditing" flag set to true) by the user. The `isEditing` property isn't the model's concern, it's a controller concern.

To continue building on our example, we'll once again make our code show a fancy greeting. This time, though, we'll assume that whether or not a fancy greeting should be displayed is the concern of the user viewing our app, not something that the author object itself ought to be concerned with. A perfect opportunity to augment the controller itself.

Adding the `isFancy` simple boolean property, and the `greeting` property function that we used before, onto the ObjectController allows us to show the fancier greeting once again. Here's the code, with a checkbox to toggle the "isFancy" property

Ember Starter Kit

I hope this illustrates the difference between normal Ember controllers and ObjectControllers, as well as how to use both. There are a few subtle ways to get tripped up when using one or the other that I'll list below.

Gotchas

Using a model hook in the route but not using an ObjectController

If you have set up a route to return a model but not set the corresponding controller to be an ObjectController, the properties you reference in your template will be looked up on the controller directly, resulting in nothing displaying in your template. This can be an easy one to overlook, especially as Ember will happily render your template without raising any errors about the missing properties.

Ember Starter Kit

Overriding setupController but not Setting the Controller's Model

We didn't cover it here, but in a Route you can define a `setupController` hook to do extra setup work on the controller. Ember's default controller setup behavior, as mentioned above, is to set the "model" property on the controller to the result of the "model" hook on the route. If you override the `setupController` hook, however, Ember won't do this default behavior for you anymore.

For example, suppose we decided to add a `setupController` hook to our fancy greeter's IndexRoute to set the "isFancy" property to true when entering this route? We could add this hook to our route like so:

setupController: function(controller, model) {
  controller.set('isFancy', true);
}


Unfortunately, although this does make it so the fancy checkbox is checked, we've lost the display of the name:

Ember Starter Kit

The solution is to either set the "model" property explicitly or just call `this._super(controller, model)` and allow Ember's default `setupController` behavior to happen. Here is is, fixed:

Ember Starter Kit

If you have any related questions, I'm @bantic on twitter. Get in touch. :)
Feb 10 14

Perfect Pricing is the Future, and Uber is its Vanguard

by cory
Uber has, at times, faced plenty of criticism for its vague pricing policies. When I've explained the service to unknowing friends, a common sticking point is the surge pricing. You don't know how much it will cost ahead of time, they ask incredulously? And you even know that it will be some multiple of the normal price, but you don't even know what that is?

If you haven't taken an uber, this is how it works: Uber is a car service coupled with a smartphone app. You hail a car from the app on your phone and get to watch the car's position on nearby streets (like a real-life pac-man game) as it maneuvers its way to pick you up. You get in and tell the driver where you're going, and some time later you've arrived at your destination. You get out, say thanks, close the door, and walk off into the night. At no point do you ask how much it will cost, or take your credit card. As part of the sign-up process you've already connected your credit card, and shortly after your ride ends the app will show you the price that you have just been charged for the ride. Welcome to your destination, you've arrived in the future.

Admittedly it's almost always more expensive to take an Uber than a taxi or another car service, sometimes by a large amount. For the price sensitive the uncertainty over cost can be fairly uncomfortable. For regular users, Uber has built up enough trust capital that the rider will happily take the car, knowing they'll pay a bit of premium but still confident that they'll pay a fair price. Knowing exactly what it will cost ahead of time is not important. Uber's promise to its customer is that they'll charge you a fair rate given the current demand, plus their premium for the high-end service. When there's high demand the app will alert you that they are automatically raising rates to attract more of their drivers to the road. You, the rider, still don't get to know exactly what the price will be but you know it will be some multiple of normal. That's the surge pricing.

This demonstrates two of the key components of seamless transactions that we'll increasingly see in the future. First, Uber charges you without you having to do anything, all through your cell phone. This concept is coming to a store near you, fast. One day soon you'll be able to pay at a coffee shop only with your phone. In the first iteration you'll probably pull up, say, the Starbucks app, select the drink you want, and a few moment later you'll hear a barista calling out your name with that chai latte. As the technology improves you won't even need to pull the phone out of your pocket. You'll walk in and the store will have begun making your usual before you even need to order. You'll just take it off the counter and walk out.

As the friction involved gets smaller and smaller, the part of the transaction that forces you to confront the cost of the item will become an appropriately smaller piece of the exchange. Honestly, how often today do you focus on the specific cost of that cappuccino anyway? Most of the time you just hand over the credit card or $5 bill anyway, because you know Starbucks is going to give you their current price and they're trustworthy enough. Imagine how much less headspace that cost calculation takes up when the mechanism of payment (cash, credit, whatever) never even comes into play. When you're just walking in and taking what you want off the shelf, inserting the friction of a payment process into the exchange is going to be something no one wants. The consumer doesn't want to wait in a line, or deal with a teller. They just want their stuff. The stores will go nuts for this (assuming they can limit theft; and they can) because an impulsive customer is their ideal customer.

The second component of seamless transactions is the pricing, and it's dependent on the first component. Once everyone is comfortable paying transparently, it's much easier for a reputable company to do away with the concept of a fixed price anyway. There will be enough customer backlash to any store that tries to game the system that they'll be incentivized to offer fair pricing anyway. And as a customer with a wallet-ized smartphone, you'll trust that McDonald's, or that corner bodega, will charge you the fair price, and over time the concept of a fixed price itself will simply fade away. In the future you'll just take what you want, when you want it, and the company will exact their expense plus their markup.

At this point, in this future world of frictionless transactions, where the specific prices themselves have faded to the periphery, comes the next phase, which is perfect pricing. Starbucks will never sell as many cups of coffee at 3pm as they do at 9am, and they know it. So why are they selling the same cup for the same price at both those times of day? A cup of coffee at 9am is worth more to their customer than one at 3pm, so the price of the cup in the morning is either too cheap, or the afternoon cup is too expensive, or both. The price should be as fluid as the purchasing experience. Prices should peak in the morning and then decline over the day into the afternoon. Pricing should as perfectly as possible reflect the demand at that moment. This is what happy hour is; a bar wants to attract more customers earlier in the night so they drop their prices for an hour or two to pack the house. In reality the beer at the beginning of the happy hour should be cheaper than the beer at the end of the happy hour, with a smooth transition over that time. In today's world of nickels and dimes this is too complicated to feasibly do, but when your phone is sitting in your pocket doing the paying, and you're just grabbing the beers when you feel like it, the bar can set its beer price to automatically scale up over time.

In a world where no one's looking at the price tags, though, the thought of pricing actually driving customer behavior starts to get a bit vague. But that is a post for another time.
Feb 2 14

You Want to Make an App? Here's my Advice

by cory
Imagine you're a physicist, hanging out in a coffee shop, over-hearing someone talking to another about the research they've been doing on their nights and weekends about Bose-Einstein Condensates. This is a rare occurrence for most people in most professions, but I'm a programmer and it happens to me on at least a weekly basis; it would happen more often if I spent more time in caf├ęs.

Sometimes I find this a little frightening because it feels like everyone is edging in on my territory. Most of the time, though, I remember to feel grateful that I'm in a field that is so popular. It really is amazing to me that people all over the place are getting interested in learning to program. It's pretty validating when so many people want to know what you know.

It's easy to see why, especially from an outsider's perspective, learning to code and making your own app would be so appealing. If you have a computer, an internet connection, a working brain and a little cash you have all the tools you need to get going. It's possible to learn everything you would need to know using free tutorials online. The software you'll use is freely available. No external permission is required: you don't need a license or a certification, and you don't need an agent or a publisher. And of course everyone you know is already talking about it. And if you're good (and lucky&em;or maybe just lucky), you can make something today that is used and enjoyed by millions tomorrow.

As a result, I get emails fairly frequently asking for help on how to get going. Sometimes it feels like everyone wants to make an app. That's a good thing because it means that more people are becoming active problem solvers. Anyway, here's my advice.

Prepare for the fact that it will be difficult



The apps you use every day are almost all created by teams of people, of various specialties, who have hundreds or thousands of hours of experience doing what they do. Furthermore, the apps themselves have invariably been through numerous iterations and improvements by the time they become successful enough that the general public finds them. What that means is that most of the apps on your phone are version 2.0's, or honestly more like version 10.0's. It's unrealistic to expect that your first effort will match theirs. And even though you don't need to be as good as the professionals to make something great, you will have some new skills to learn as you go.

Make it fun, and make it realistic



See the point above. If you're asking for advice about building an app that means you've already got an app idea, and that's important. It's not important because of the idea; even though it seems like the idea's the most important part, it's really not. Everyone's got ideas. It's the execution of them that matters. But having an idea (even a bad one) is great because it gives you focus. Now when you are teaching yourself to code you have a solid reason for doing so. When you learn about a new part of the iOS SDK you'll be able to see how you might want to use it in your app.

This is the part that makes it fun. You get to see your app coming together as you put more time into it. You'll need to find the part that makes it fun for you. If you're lucky enough that you enjoy coding, you'll have a leg up. But if you don't like coding, find the part about the building process that is fun for you. If it's not fun you won't be motivated enough to put the necessary time into it.

Making it more realistic means embracing the point above this one. Your first version is not going to be as awesome as your initial vision. And it won't have the same polish as professional work. But making that first version is the first step to making the version that's in your mind. I'm not saying to aim low and stop once you get there. I'm saying to start simple, and then make it better. But really: start simple.

Ship It



There are two important statements that relate to building an app that are both wholly true and unfortunately totally at odds with one another. They are: No product that was never released was ever a success; and Every product can be improved. Because there's an endless amount of little fixes and tweaks you can make, it can be really hard to actually release something to the public. Once you release it you can no longer tell people about your great idea; your product has to be able to convince them on its own. That's a difficult thing to do, because see the first point above. But I'll reiterate the first statement: No product that was never released was ever a success. Get your product out there. Work like hell to get it over the finish line, even if it's missing 90% of the features you intended. Get it out. Then make it better. Be wary of new features that delay your launch. One of them might be the one that delays your project indefinitely. And a product that was never released...you get the idea.

Pick One, or Pick Zero



Because you're in a death match here, getting your project over the hump and out the door before too many new features weigh it down indefinitely, you have to be ruthless about deciding what you want. Always ask yourself why a particular feature is necessary. Look for ways to cut corners. Lean on established cut-and-copy solutions that you can use for now. Remember, the goal is to get this released. Then make it better.

So every time you have a new feature, choose the simplest option. Want your users to be able to log in using Google, Twitter or Facebook? Don't implement them all. Pick one. Add in the others later. Or, better yet, ask if you really need to have a login system at all. Can you make a working version of your app, just to get v1 out the door, without needing to log in users? Maybe. It's always faster to do nothing than something, if you can. There are vanishingly few apps that were successes because of their login systems. Focus on the features that are most unique to your app. For the rest, Pick one. Or zero.

You Can Do It



I've presented a somewhat bleak picture above, and that's because the truth is that most app ideas wither under the weight of lack of time and experience. But the good news is that some of them do flourish. And even though it is difficult, your first assumption was correct: You just need tenacity and some intelligence to make it. The rest is free or cheap. And everyone who has ever made a successful app started out knowing less than you do.