Where are the REST frameworks?

I like REST. I’ll admit that I didn’t understand all of Dr. Roy Fielding’s Dissertation (he is, after all writing to experts in the field of API design) but I think that I’ve managed to understand enough of it that Rails’ implementation of REST leaves a lot to be desired. Rails is not alone, many services are releasing APIs that they are calling REST. Dr. Fielding recently addressed this in a post on his site. Like his thesis, he is talking to API designers, not to Joe and Jane Developer who needs to get the latest web site off the ground. Perhaps more practically, Subbu Allamaraju gave the issue of REST and hyperlinking a thorough exfoliation over at InfoQ.

If you read Subbu’s article and compare it to the standard Rails 2.x web application you will quickly see that difference. In a rails application, if you GET a JSON representation of our quintessential blog all you are going to be told is that the owner_id is 1232; your code has to know that Owners are Users and that they can be retrieved from http://api.example.com/users/{user_id}. This is like putting an <img img_id=123> in your html and expect the browser to know to retrieve the file from http://www.example/resources/images/123.

There is the comparison that most people who think they are writing REST APIs are forgetting; HTTP. If you are writing a REST API then you are writing something VERY much like HTTP. None of us are likely to need to create a REST API. I work for a financial institution and we might have a need to create a REST API like HTTP that is geared towards exchanging financial transactions between internal systems, our partners and our clients, but I suspect that would be unnecessary.

No most of us have no need for creating a new REST API. What we should be talking about is publishing RESTful media types. If I’m going to build the archetypical Blog application, then I need media types for a Blog, a Post, a Comment, a Tag, an Author and what ever else.

A Note: I’m making a distinction that I don’t think Dr. Fielding would make. Dr. Fielding makes it clear that the linking mechanism is central to a REST API, and since HTTP leaves much of the linking mechanism (but certainly not all) up to the user agent, a true REST API requires a media types. However, the vast majority of the work that we do is over HTTP so we have no need, in general, to worry about that half of a REST API. I think it greatly clarifies things if we simply talk about creating REST Media Types rather than APIs. I just noticed that I’m not the first one to realize this (after Dr. Fielding of course).

What about the Frameworks

I’ve been trying to find a Ruby framework that can handle REST Media Types right out of the gate. Waves, which its resource oriented architecture seems like a good place to start. They are right at the start of putting together what they call foundations for REST services. I’m writing this article in part to influence how they go about that process.

REST ≠ MVC

One of the biggest problems is that people are trying to bolt REST onto MVC frameworks. The problem is it is probably the wrong programming model to use. The controller for a REST service is absolutely trivial; you can only call a maximum of 4 methods on any given resource, and they are always handled the same, so the controller is completely anemic and should fall into the framework itself. Views are also the wrong paradigm since they only deal with out going representations; you really need some mechanism for specifying the transformation of a resources into a representation (that is its media type), and back. All that you really have left are the resources (assuming they’re like models) and the transformations.

REST and Your Web Application

I’m probably going to annoy a lot of people with this. A web browser (as they are currently implemented) is NOT a RESTful client. You can use XHR object supplied by a browser’s JavaScript engine as a REST client, but the browser itself interacting with plain old HTML cannot act RESTfully. In part, its because your browser won’t make PUT or DELETE requests (you have to fake it with extra parameters on a POST). Mostly its a matter of usability.

For a non-trivial use case, you are going to want to something wizard like. That means you are going to have to create a number of intermediate, transient representations to maintain that state. Its an affectation in this case. The transient representations serve no purpose but to maintain this fiction that its REST.

I honestly think that you should really simply leave the your REST API as an API and build a separate application to act as your User Interface. You’ll make it easier on yourself in the end

REST and MVC

Like I said earlier, REST is not MVC. But that doesn’t mean that its not related to MVC. MVC is very well suited to building a User Interface. As I said earlier, you should build a separate application for your User Interface. Well, there is no reason why you can’t use a Resource as the Model in your MVC application. Rails already does this; ActiveResource allows you to use a REST Resource instead of a database table for your model.

This is where things get really interesting.

If you are already using resource representations for the models in your MVC user interface, why limit yourself to a single domain? There is no reason why you couldn’t use multiple domains. This allows for what I call DMVC: Distributed Model View Controller. By distributing your models over a number of applications (via REST media types) you can get get some great long term efficiencies by the serendipitous, and unplanned, reuse of REST APIs.

For example, if you have a non-trivial content management system, you are going to want to move your content changes from the author to an editor to perhaps a legal reviewer and maybe finally to a marketing manager before those changes are “live”. At the same time, you may have defect tracking system where defects move from reported, through any number of stages to completed. Traditionally, you would include something like acts_as_statemachine in each of your rails apps to get this functionality.

But if you apply the REST architectural style, you can create 3 independent applications: Content Management, Defect Management, Workflow. The Workflow applciation would have something like a State resource which would be associated with some other resource without knowing anything about that resource. If the business logic was sufficiently complex (for instance, a change in the State resource would change the associated resource) you could create another application that combined a State resource with a (let’s say) Defect resource to expose StatefulDefect resources.

Also, because you have decoupled your the workflow from both your CMS and your defect tracking, you can reuse both to create a project management application where the content is not work flowed but managed wiki style and defects are not workflowed rather raised and closed since you have hourly builds to the project server (or what ever).

We already know how to optimize and scale web servers, and the same can be done here. Or, if this is a purely internal set of services (or even largely) you could create a more efficient connection mechanisms/schemes than HTTP (I’m not sure what those would be). Or you might try more efficient media types (for example using JSON or even a binary format).

REST, Associations & ORM

One of the more valuable parts of Rails is the ActiveRecord ORM. It allows you to say that a Blog belongs_to a User and a User has_many blogs. Associations between these objects are by ID as they refer to other records in the database.

But REST blows that apart. Your Blog and your User are no longer stored in the same database. We refer to every unique resource by a URL so it may live somewhere totally unaccessible to us. If I have blogs and you have users, I can say that my blog belongs to a user, but I cannot expect your users will know anything about my blogs. Now perhaps the representation of the user that you give me includes a list of blogs that the user owns, in which case I can PUT back a representation that includes the URI of my Blog, but my code cannot rely on that information being known.

You also get some interesting behaviours.

All you have is a URI. You have NO idea what that resource is. As a result, its conceivable that I might try to create a blog that is “owned” by an image. Or the URI point to a resource that is a legitimate owner of the blog but its media types are not those that the blog application understands. For the most part, that’s OK because the blog application is really just an API, its not our User Interface. The User Interface application will need to have coping mechanisms for this but I’m not sure that’s a problem. If all you’re producing is an API, then its not even your problem.

More importantly, you get all sorts of strangely compelling polymorphism. Its liberating not to know what you’re associated with since it guarantees that you’ll become associated with all sorts of unexpected things.

Random Strings

Today I created what is possibly the most efficient way to create a random string:

function randomString(length){
    var max = "";
    for (i=0;i<length; i++){
        max += "z";
    }
    return Math.round(Math.random() * parseInt(max, 36)).toString(36).toUpperCase();
}

So how, you ask, does it work? Well, it uses the fact that you can encode a number with any radix up to 36 which uses 0-9a-z. Perfect for creating a random string. SWe take the number of characters that we want and create our maximum encoded value (by repeating Z as often as necessary), and convert that into an integer. We then take that integer and use it as the upper bounds for a random number generator. We then convert that random number to a string using a radix of 36 and convert it to upper case for fun.

I'm not really thrilled about looping to create my max string but what can you do?

It would be possible to create strings without numbers by using a radix of 26 and mapping numbers to letters after the fact, but that's a lot of extra work.

Things I'd like to see in JavaScript

Recently, someone asked me what features I'd like to see added to JavaScript. In the week or so since then I've had some time to think about it. Here are some of my thoughts:

Code inclusion

Right now, when we want to dynamically includes some javascript on the fly (for instance to ensure that we only load the features that we need) we have to resort to document.writing a script tag. This is exceptionally inefficient. I would be nicer if we could explicitly include javascript files using something like: includeScript('/scripts/myScript.js'). It might be nice to allow a call back function to be passed along too.

Site persistent JavaScript

Currently, every time you unload a page, the compiled javascript is tossed out the door and new javascript is compiled when the next page loads. I don't know about you, but 80% of my code doesn't change page by page. It would be great if we had a way to "save" our compiled JavaScript and simply reuse it on each page. This way, I can load my libraries once and never have to worry about it again. It would also be necessary to identify some code that lasts only for a single session and some that lasts "forever" (same idea as with persistent cookies)

Native set objects

In the past I've run into confusion with some of the new Array methods that are being introduced by Mozilla. Specifically, it turns out that [].any(function (){return true;}) !== [].all(function (){return true;});.

When I chatted with some of the Mozilla guys about this they said that this has to do with set math. That got me to thinking about sets in general. There are some things that one can do with sets that are generally useful.

• You can intersect, union, difference sets. Calculating the intersection of two sets (lets say sets of DOM Nodes) can be exceptionally useful.
• Every member of a set is unique. Again this is a useful feature for working with collection of DOM Nodes.

There are other features of sets that I would like to be able to use. I could build a JavaScript implementation of a set but its MUCH slower than a native implementation would be. Since its generally useful, its worth adding to the language

Date Literals

Since strict JSON only allows literal notation, there is currently no way to include real dates in a JSON construct. Dates are one of the most useful objects we have in JavaScript and having a literal for it would be wonderful.

Those are my ideas. What do you think would be a good addition?

Canvas Loading Indicator

If any of you have been writing Web 2.0 apps for the iPhone, you will have realized by now that it doesn't allow you to use animated gifs. This is a bit of a problem since you want to provide some sort of feedback to the user if you're waiting for some data to load. Normally you would show something like what you find at http://www.ajaxload.info/. The other thing you might know is that you can use the canvas with the iPhone. Clearly the solution to the problem is to use the canvas to draw your indicator. I looked and couldn't find anyone who had built one. So I wrote one. Turns out, if you keep things simple, its relatively easy to write your basic spinner.

function getLoading(context, bars, center, innerRadius, size, color) {
var animating = true,
    currentOffset = 0;

function makeRGBA(){
    return "rgba(" + [].slice.call(arguments, 0).join(",") + ")";
}
function drawBlock(ctx, barNo){
    ctx.fillStyle = makeRGBA(color.red, color.green, color.blue, (bars+1-barNo)/(bars+1));
    ctx.fillRect(-size.width/2, 0, size.width, size.height);
}
function calculateAngle(barNo){
    return 2 * barNo * Math.PI / bars;
}
function calculatePosition(barNo){
    angle = calculateAngle(barNo);
    return {
        y: (innerRadius * Math.cos(-angle)),
        x: (innerRadius * Math.sin(-angle)),
        angle: angle
    };
}
function draw(ctx, offset) {
    clearFrame(ctx);
    ctx.save();
    ctx.translate(center.x, center.y);
    for(var i = 0; i<bars; i++){
        var curbar = (offset+i) % bars,
            pos = calculatePosition(curbar);
        ctx.save();
        ctx.translate(pos.x, pos.y);
        ctx.rotate(pos.angle);
        drawBlock(context, i);
        ctx.restore();
    }
    ctx.restore();
}
function clearFrame(ctx) {
    ctx.clearRect(0, 0, ctx.canvas.clientWidth, ctx.canvas.clientHeight);
}
function nextAnimation(){
    if (!animating) {
        return;
    };
    currentOffset = (currentOffset + 1) % bars;
    draw(context, currentOffset);
    setTimeout(nextAnimation, 50);
}
nextAnimation(0);
return {
    stop: function (){
        animating = false;
        clearFrame(context);
    },
    start: function (){
        animating = true;
        nextAnimation(0);
    }
};
}

This is a fair chunk of code but its not entirely clear how you use it. Its actually quite simple. Assuming you have a canvas to work with, All you do is call

var controller = getLoading(canvas.getContext("2d"), 9, {x:100, y:100}, 10, {width: 2, height:10}, {red: 0, green: 17, blue: 58});

The getLoading function takes a 2d context, the number of bars that you want your indicator to use, the X and Y coordinates of the center of the indicator, the radius of the inner portion of the indicator the height and width of each of the bars of the indicator and, finally, the color you want to use for the indicator. We ask for the components of the color separately because each spoke of the indicator gets a progressively smaller alpha value. The function returns a simple object with two methods, start() and stop(). The indicator is created spinning. If you wanted a more fancy set of spokes, all you would really have to do is rewrite the drawBlock function. Blogger seems to strip out script tags so there's no real way to provide a test (expect maybe a honkin huge bookmarklet) so if you want to try it out, copy the script and he following:

function (){
  var canvas = document.createElement("canvas");
  canvas.width= 200;
  canvas.height = 200;
  canvas.style.cssText="position:absolute; top:100px; left:100px; background:#transparent; border: 3px solid red";
  document.body.appendChild(canvas);
  var controller = getLoading(canvas.getContext("2d"), 9, {x:100, y:100}, 10, {width: 2, height:10}, {red: 0, green: 17, blue: 58});

  var button1 = document.createElement("input");
  button1.value="stop";
  button1.type="button";
  button1.onclick = function (){
      controller.stop();
  };
  document.body.appendChild(button1);

  var button2 = document.createElement("input");
  button2.value="start";
  button2.type = "button";
  button2.onclick = function (){
      controller.start();
  };
  document.body.appendChild(button2);
})();

in firebug and it should be fine.

Curried What Now?

With the release of Prototype 1.6.0 release candidate, we now have a new way to do things. We can now write curried functions.

Show of hands, how many of you know what a curried function is? I thought so. I first encountered currying about a year ago via Doug Crockfords site which brought me to Svend Tofte’s article on it. Perhaps its because I’m stark raving mad, but I immediately fell in love with the technique and tried to find ways to use it. 

I've shown the article to others but few have really seen the potential as I have. I think its hard to get your head around it without examples. 

The canonical example

It seems that every description of curried functions starts with something like the following, which Sam Stephenson used to announce the release of Prototype 1.6.0 RC1:

function sum(a, b) {
return a + b;
}
sum(10, 5) // 15
var addTen = sum.curry(10);
addTen(5)  // 15

The explanation is that you can write some function, and call it with some of the arguments defined, and then call it later with the rest of the arguments. That is all well and good but how on earth would you want to use it in a real situation?

Real uses for curry

Curried functions are all about not repeating yourself more than you have to. Don’t think of it as calling some function many times with some of the same arguments. Think of it as creating a specific function out of a more general function. That is probably not any more enlightening so a less trivial example might be illustrative.

Alerting buttons

This is probably not a much more complex example, but I think it starts providing an explanation. What we are going to do is have a page with 5 buttons. When the button is clicked, we are going to alert a number which is randomly assigned to that button (each button will have 1 random number assigned to it). We might right this as follows:

function alertMessageEventHandler(evt){
alert(this._message);
}

$$("#ButtonList button").each(function (button){
button._message = "Your random number is:\n" + Math.round(100 * Math.random());
button.observe("click", alertMessageEventHandler);
});

This is going to work just fine but lets see how we can rewrite it using curry:

function alertMessageEventHandler(msg, evt){
alert("curried" + msg);
}

$$("#ButtonList button").each(function (button){
var message = "Your random number is:\n" + Math.round(100 * Math.random());
button.observe("click", alertMessageEventHandler.curry(message));
});

This is obviously almost as trivial as our sum function above but it gets the point across. 

Validation

While its all well and good to provide trivial example, sometimes its more illustrative if we have a robust example. So lets see what we can come up with. 

Imagine that we want to build a validation system for forms. We know that we want to define, for any given field, a series of tests to which a value is subjected and we want to get back a list of the tests that failed so that we can display error messages. 

Lets start with a simple validation function. It will take the value to be tested and an array of testing objects. It will return an array of failed testing objects (which would be further used to produce the error message). 

function isValid (value, tests) {
return tests.reject(function (aTest){
return aTest.test(value);
});
}

This is a really small function but I think we can all see what it does. We loop through each element in tests and call its "test" function passing in the value to test in. If the test function returns true, then we know that the value passed that test and we boot it out of our array. We can use our new function by creating an array of tests:

function isLessThanSeven(val) {
return val <> 2;
}
function isMultiplOfThree(val) {
return (val % 3) == 0;
}
var test1 = [
{test: isLessThanSeven, errorMessage: "Must be less than 7"},
{test: isGreaterThanTwo, errorMessage: "Must be greater than 2"},
{test: isMultiplOfThree, errorMessage: "Must be a multiple of 3"}
];
console.debug("3:" + isValid(3, test1).length)
console.debug("10:" + isValid(10, test1).length)

This is a rather simple test. In the first test we should pass all the tests (thus have a length of 0) and in the second we should fail 2 test. 

The problem with this sort of thing is that we end up repeating ourselves with all our little test functions. This is where curry comes in. If we get a little more generic we can reuse some code easily:

function isLessThan(comparison, val){
return (val <> comparison );
}
function isMultiple(divisor, val){
return (val % divisor) == 0;
}
var test2 = [
{test: isLessThan.curry(7), errorMessage: "Must be less than 7"},
{test: isGreaterThan.curry(2), errorMessage: "Must be greater than 2"},
{test: isMultiple.curry(3), errorMessage: "Must be a multiple of 3"}
];

console.debug("3:" + isValid(3, test2).length)
console.debug("10:" + isValid(10, test2).length)

We can now reuse our isLessThan and other functions in many places and never repeat ourselves.

Coding

The question that remains is where and when to use curried functions. There are a few things to look for. The first is a function that you are calling frequently with the similar collections of arguments. In this case, set things up so that you curry the function with your common arguments and save the result and call it later (like in the canonical example). The second is somewhat simpler. Look for situations where you are storing some value on an element so that you can use it in an event handler (like in the alerting buttons example). Finally (and this is the hard one) you need to find combinations of functions that are fundamentally similar and consolidate them into functions that you curry as needed. 

The easiest way to write a function for currying is to reverse the order of your arguments. Its counter intuitive but if you think about, we typically put the most important arguments first. Often those are also the ones that are most likely to change between calls. So if we put it at the end then we can more easily use curry. 

In fact if we reorder the arguments of our isValid function, we can curry it and simplify things. 

function isValidCurry (tests, value) {
//we assume that tests is an array of objects which look like:
//{
//    test: function (value){return true;},
//    error: 120
//}
return tests.reject(function (aTest){
   return aTest.test(value);
});
}

var test3 = [
{test: isLessThan.curry(7), errorMessage: "Must be less than 7"},
{test: isGreaterThan.curry(2), errorMessage: "Must be greater than 2"},
{test: isMultiple.curry(3), errorMessage: "Must be a multiple of 3"}
];

var validateTestThree = isValidCurry.curry(test3);
console.debug("3:" + validateTestThree(3).length);
console.debug("10:" + validateTestThree(10).length);

So how do you like your curry?