Yariv’s Blog

Check out PicLike, the new app I made using the Flickr API, Google App Engine and the Facebook Like button: http://piclike.appspot.com.

In my last post I described how to use LFE to overcome some of the weaknesses of parameterized modules. Unfortunately, all is not rosy yet in the land of LFE types. Parameterized modules allow you to create only static types. The compiler doesn’t do static type checking, but you have to define the properties of your types at compile time. This works in many cases, but sometimes you want totally dynamic containers that map keys to values. In Erlang, this is typically done with dicts. We could still use them with LFE, but I don’t like having different methods of accessing the properties of objects depending on whether their types were defined at run time or compile time.

Let’s use macros to solve the problem.

In my last post, I relied on the build-in ‘call’ function to access the properties of static objects. Let’s create a wrapper to ‘call’ that lets us access the properties of dicts in exactly the same manner as we do properties of other objects:


(defmacro dot
  ((obj prop . tl)
   `(let ((obj ,obj))
      (case (element 1 obj)
        ('dict
         ,(case tl
            (()
             `(case (: dict find ,prop obj)
                ('error 'undefined)
                ((tuple 'ok valx) valx)))
            (_ `(: dict store ,prop ,(hd tl) obj))))
        (_ (call obj ,prop ,@tl))))))


We can use ‘dot’ to get and set the properties of both dicts and static objects:


(defun test ()
  (let* ((dog1 (: dog new))
     (dog2 (: dict new))
           (dog1_2 (dot dog1 'name 'lola))
           (dog2_2 (dot dog2 'name 'rocky)))
    (list (dot dog1_2 'name)
           (dot dog2_2 'name))))



> (: dog test)                                
(lola rocky)


I think this is kinda of cool, though to be honest I’m not entirely sure it’s a great idea to obfuscate in the code whether we’re dealing with dicts or static objects.

One of the features I dislike the most about Erlang is records. They’re ugly and they require too much typing. Erlang makes me write


Dog = #dog{name = "Lolo", parent = #dog{name = "Max"}},
Name = Dog#dog.name,
ParentName = (Dog#dog.parent)#dog.name,
Dog1 = Dog#dog{
   name = "Lola",
   parent = (Dog#dog.parent)#dog{name = "Rocky"}}


When I want to write


Dog = #Dog{name = "Lolo", parent = #dog{name = "Max"}},
Name = Dog.name,
Size = Dog.size,
Dog1 = Dog{
  name = "Lola",
  parent = Dog.parent{name = "Rocky"}}


In the defense of records, they’re just syntactic sugar over tuples and as such they enable fast access into a tuple’s properties despite Erlang’s dynamic nature. In compile time, they’re converted into fast element() and setelement() calls that don’t require looking up the property’s index in the tuple. Still, I dislike them because 1) in many cases, I’d rather optimize for more concise code than faster execution and 2) if the Erlang compiler were smarter could allow us to write the more concise code above by inferring the types of variables in your program when possible. (I wrote a rant about this a long time ago, with a proposed solution to it in the form of a yet unfinished parse transform called Recless.)

Parameterized modules provide a somewhat more elegant and dynamic dealing with types, but they still require you to do too much work. You can define a parameterized module like this:

-module(dog, [Name, Size]).

This creates a single function called ‘new/2′ in the module ‘dog’. You can call it as follows:


Dog = dog:new("Lola", "big").


It returns a tuple of the form


{dog, "Lola", "Big"}.


You can’t set only a subset of the record’s properties in the ‘new’ function. This doesn’t work


Dog = dog:new("Lola").


To access the record’s properties you need to define your own getter functions, e.g.


name() ->
  Name.


which you can call it as follows:


Name = Dog:name().


(This involves a runtime lookup of the ‘name’ function in the ‘dog’ module, which is slower than a record accessor).

There’s no way to set a record’s properties after it’s created short of altering the tuple directly with setelement(), which is lame and also quite brittle. To create a setter, you need do the following:


name(Val) ->
    setelement(2, THIS, Val).


Then you can call


Dog1 = Dog:name("Lola").


to change the object’s name.

When LFE came out I was hoping it would provide a better way of dealing with the problem. Unfortunately, records in LFE are quite similar to Erlang records, though they are a bit nicer. Instead of adding syntactic sugar, LFE creates a bunch of macros you can use to create a record and access its properties.


(record dog (name))
(let* ((dog (make-dog (name "Lola")))
     (name (dog-name dog))
     (dog1 (set-dog-name dog "Lolo")))
  ;; do something)


LFE still requires us to do too much typing when dealing with records to my taste, but LFE does give us a powerful tool to come up with our own solution to the problem: macros. We can use macros generate all those repetitive brittle parameterized module getters and setters that in vanilla Erlang we have to write by hand. This can help us avoid much the tedium involved in working with parameterized modules.

(ErlyWeb performs similar code generation on database modules, but it does direct manipulation of Erlang ASTs, which are gnarly.)

Let’s start with the ‘new’ functions. We want to generate a bunch of ‘new’ functions allow us to set only a subset of the record’s properties, implicitly setting the rest to ‘undefined’.


(defun make_constructors  (name props)
  (let* (((props_acc1 constructors_acc1)
          (: lists foldl
             (match-lambda
              ((prop (props_acc constructors_acc))
               (let* ((params (: lists reverse props_acc))
                      (constructor
                       `(defun new ,params
                          (new ,@params 'undefined))))
                 (list
                  (cons prop props_acc)
                  (cons constructor constructors_acc)))))
             (list () ())
             props))
         (main_constructor
           `(defun new ,props (tuple (quote ,name) ,@props))))
    (: lists reverse (cons main_constructor constructors_acc1))))


This function take the module name and a list of properties. It returns a list of sexps of the form


(defun new (prop1 prop2 ...prop_n-m)
  (new prop1 prop2 ... prop_n-m 'undefined))


as well as one sexp of the form


(defun (prop1 prop2 ... prop_n)
  (tuple module_name prop1 prop2... prop_n)


The first set of ‘new’ functions successively call the next ‘new’ function in the chain, passing into it their list of parameters together with ‘undefined’ as the last parameter. The last ‘new’ function takes all the parameters needed to instantiate an object and returns a tuple whose first element is the module, and the rest are the object’s property values. (We need to store the module name in the first element so we can use the parameterized modules calling convention, as you’ll see later.)

Now let’s create a function that generates the getters and setters


(defun make_accessors (props)
   (let*
       (((accessors idx1)
        (: lists foldl
           (match-lambda
            ((prop (acc idx))
             (let* ((getter `(defun ,prop (obj) (element ,idx obj)))
                     (setter `(defun ,prop (val obj)
                        (setelement ,idx obj val))))
                (list (: lists append
                         (list getter setter)
                         acc)
                       (+ idx 1)))))
           (list () 2)
           props)))
     accessors)))


This function takes a list of properties and returns a list of sexps that implement the getters and setters for the module. Each getter takes the object and returns its (n + 1)th element, where n is the position of its property (the first element is reserved for the module name). Each setter takes the new value and the object and returns the tuple after setting its (n + 1)th element to the new value.

Now, let’s tie it this up with the module declaration. We need to create a macro that generates the module declaration, constructors, getters, and setters, all in one swoop. But first, we need to expose make_constructors and make_accessors to the macro by nesting them inside a (eval-when-compile) sexp.


(eval-when-compile
  (defun make_constructors ...)
  (defun make_accessors ...))
 
(defmacro defclass
  (((name . props) . modifiers)
   (let* ((constructors (make_constructors name props))
          (accessors (make_accessors props)))
     `(progn
        (defmodule ,name ,@modifiers)
        ,@constructors
        ,@accessors))))


(defclass returns a `(progn) sexp with a list of new macros that it defines. This is a trick Robert Virding taught me for creating a macro that in itself creates multiple macros.)

Now, let’s see how this thing works. Create the file dog.lfe with the following code


;; import the macros here
 
(defclass (dog name size)
  (export all))


Compile it from the shell:


(c 'dog)


Create a dog with no properties


> (: dog new)
#(dog undefined undefined)


Create a new dog with just a name


> (: dog new '"Lola")
#(dog "Lola" undefined)


Create a new dog with a name and a size


> (: dog new '"Lola" '"medium")
#(dog "Lola" "medium")


Get and set the dog’s properties


> (let* ((dog (: dog new))
         (dog1 (call dog 'name '"Lola"))
         (dog2 (call dog1 'size '"medium")))
    (list (call dog2 'name) (call dog2 'size)))
("Lola" "medium")


This code uses the same parameterized module function calling mechanism that allows us to pass a tuple instead of a function as the first parameter to the ‘call’ function. Erlang infers the name of the module that contains the function from the first element of the tuple.

As you can see, LFE is pretty powerful. I won’t deny that sometimes I get the feeling of being lost in a sea of parenthesis, but over time the parentheses have grown on me. The benefits of macros speak for themselves. They give you great flexibility to change the language as you see fit.

I attended the Bay Area Erlang Factory last week. It was a great event. I met many Erlang hackers, attended interesting talks, learned about cool projects (CouchDB, QuickCheck, Nitrogen, Facebook Chat), gave a talk about ErlyWeb, and drank beer (without beer, it wouldn’t be a true Erlang meetup).

My favorite talk was by Damien Katz. He told the story of how he had decided to take a risk, quit his job, and work on his then amorphous project. He wanted to work on cool stuff, and that was the only way he could do it. Even if nothing else came out of it, he knew it would have been a great learning exercise. Something great did eventually come out of it, as he created CouchDB (which looks awesome btw) and IBM eventually hired him to work on it full time.

Damiens’ story reminded me of the time I started working ErlyWeb a few years ago. After I left the company I was working for at the time, I decided to take a few months and work on something cool. I didn’t know what exactly it would be or how long it would take, but I knew that I wanted to build a product that would help people communicate in new ways, and I wanted to build it with my favorite tools. I knew the chance of failure was high, but I figured the learning alone would be worth it. I also viewed open source as an insurance policy of sorts. Even if I couldn’t get a product off the ground, my code could live on and continue to provide value to people.

Doing it paid off. My savings dwindled, but I learned Erlang, created ErlyWeb and Vimagi, met many like minded people, and it opened new doors. Now I work on cool stuff at Facebook, ErlyWeb lives on, and every day people are using Vimagi to create amazing art and share it with their friends.

The moral of the story: if you’re not working on cool stuff, take a risk and try to make it happen. Don’t worry about building the next Google or making lots of money, because you’ll probably fail. But the lessons you learn and the connections you make will be worth it.

I’ve been thinking lately about the problem of scaling a service like Twitter or the Facebook news feed. When a user visits the site, you want to show her a list of all the recent updates from her friends, sorted by date. It’s easy when the user doesn’t have too many friends and all the updates are on a single database (as in Twoorl’s case :P). You use this query:


"select * from update where uid in ([fid1], [fid2], ...) order by creation_date desc limit 20"


(After making sure you created an index on uid and creation_date, of course :) )

However, what do you when the user has many thousands of friends, and each friend’s updates are stored on a different database? Clearly, you should fetch those updates in parallel. In Erlang, it’s easy. You use pmap():


fetch_updates(Uids) ->
  pmap(
    fun(Uid) ->
      Db = get_db_for_user(Uid),
      query(Db, [<<"select * from update where uid =">>,
         Uid, <<" order by creation_date desc limit 20">>])
    end, Uids).
 
%% Applies the function Fun to each element of the list in parallel
pmap(Fun, List) ->
  Parent = self(),
  %% spawn the processes
  Refs =
      lists:map(
        fun(Elem) ->
                Ref = make_ref(),
                spawn(
                  fun() ->
                          Parent ! {Ref, Fun(Elem)}
                  end),
                Ref
        end, List),
 
  %% collect the results
  lists:map(
    fun(Ref) ->
            receive
                {Ref, Elem} ->
                    Elem
            end
    end, Refs).


Getting the updates is straightforward. However, what do you do once you’ve got them? Merging thousands of lists can take a long time, especially if you do it in a single process. The last thing you want is that your site’s performance would grind to a halt when users add lots of friends.

Fortunately, merging a list of lists isn’t too hard to do in parallel. Once you’ve implemented your nifty parallel merge algorithm, you can theoretically speed up response time by adding more cores to your web servers. This should help you maintain low latency even for very dense social graphs.

So, how do you merge a list of sorted lists in parallel in Erlang? There is probably more than one way of doing it, but this is what I came up with: you create a list of single element lists. You scan through the main list, and for each pair of lists you spawn a process that merges the two lists and sends the result to the parent process. The parent process collects all the results, and repeats as longs as there is more than one result. When only one result is left, the parent returns it.

Let’s start with the base case of how to merge two lists:


%% Merges two sorted lists
merge(L1, L2) -> merge(L1, L2, []).
 
merge(L1, [], Acc) ->  lists:reverse(Acc) ++ L1;
merge([], L2, Acc) -> lists:reverse(Acc) ++ L2;
merge(L1 = [Hd1 | Tl1], L2 = [Hd2 | Tl2], Acc) ->
  {Hd, L11, L21} =
      if Hd1 < Hd2 ->
              {Hd1, Tl1, L2};
         true ->
              {Hd2, L1, Tl2}
      end,
  merge(L11, L21, [Hd | Acc]).


Now, to the more interesting part: how to merge a list of sorted lists in parallel.


%% Merges all the lists in parallel
merge_all(Lists) ->
    merge_all(Lists, 0).
 
%% When there are no lists to collect or to merge, return an
%% empty list.
merge_all([], 0) ->
     [];
 
%% When no lists are left to merge, we collect the results of
%% all the merges that were done in spawned processes
%% and recursively merge them.
merge_all([], N) ->
  Lists = collect(N, []),
  merge_all(Lists, 0);
 
%% If only one list remains, merge it with the result
%% of all the pair-wise merges
merge_all([L], N) ->
  merge(L, merge_all([], N));
 
%% If two or more lists remains, spawn a process to merge
%% the first two lists and move on to the remaining lists
%% without blocking. Also, increment the number
%% of spawned processes so we know how many results
%% to collect later.
merge_all([L1, L2 | Tl], N) ->
    Parent = self(),
    spawn(
      fun() ->
              Res = merge(L1, L2),
              Parent ! Res
      end),
    merge_all(Tl, N + 1).
 
%% Collects the results of N merges (the order
%% doesn't matter).
collect(0, Acc) -> Acc;
collect(N, Acc) ->
  L = receive
            Res -> Res
        end,
  collect(N - 1, [L | Acc]).


So, how well does this perform? I ran a benchmark on my 2.5 GHz Core 2 Duo Macbook Pro. First, I created a list of a million random numbers, each between 1 and a million:


> L = [random:uniform(1000000) || N <- lists:seq(1, 1000000)].


Then, I timed how long it takes to sort the list, first with lists:sort() and then with my shiny new parallel merge function.


> timer:tc(lists, sort, [L]).                                
{688149,
[1,1,1,1,2,6,7,8,10,11,11,11,13,13,14,15,15,16,17,17,20,21,
  21,22,23,24,29|...]}


Less than a second. lists:sort() is pretty fast!

Before we can pass the list of numbers into merge_all(), we have to break it up into multiple lists with a single element in each list:


> Lists = [[E] || E <- L].


Now for the moment of truth:


> timer:tc(psort, merge_all, [Lists]).
{8187563,
[1,1,1,1,2,6,7,8,10,11,11,11,13,13,14,15,15,16,17,17,20,21,
  21,22,23,24,29|...]}


About 8.2 seconds :(

It’s not exactly an improvement, but at least we learned something. In this test case, the overhead of process spawning and inter-process communications outweighed the benefits of parallelism. It would be interesting to run the same test it on machines that have more than two cores but I don’t have any at my disposal right now.

Another factor to consider is that lists:sort() is AFAIK implemented in C and therefore it has an unfair advantage over a function implemented in pure Erlang. Indeed, I tried sorting the list with the following pure Erlang quicksort function:


qsort([]) -> [];
qsort([H]) -> [H];
qsort([H | T]) ->
    qsort([E || E <- T, E =< H]) ++
        [H] ++
        qsort([E || E <- T, E > H]).



> timer:tc(psort, qsort, [L]).    
{2066387,
[1,2,3,3,4,5,6,6,7,7,8,8,10,10,10,12,13,14,14,15,17,18,19,
  22,24,26,26|...]}


It took about ~2 seconds to sort the million numbers.

The performance of merge_all() doesn’t seem great, but consider that we spawned ~1,000,000 processes during this test. It had ~19 levels of recursion (log2 500,000). At each level, we spawned half the number of processes as the previous level. The sum of all levels is 500,000*(1 + 1/2 + 1/4 + 1/8 … + 1/19) ~= 1,000,000 (http://en.wikipedia.org/wiki/Geometric_series). 8 seconds / 500,000 processes = 0.000016 seconds / process. It’s actually quite impressive!

Let’s go back to the original problem. It wasn’t to sort one big list, but to merge a list of sorted lists with 20 items in each list. In this scenario, we still benefit from parallelism but we don’t pay for the overhead of spawning hundreds of thousands of processes to merge tiny lists in the first few levels of recursion. Let’s see how long it takes merge_all() to merge a million random numbers split between 50,000 sorted lists.


> Lists = [lists:sort([random:uniform(1000000) || N <- lists:seq(1, 20)])
       || N1 <-  lists:seq(1, 50000)].
> timer:tc(psort, merge_all, [lists]).
{2259870,
[1,1,4,5,8,10,10,12,12,13,14,14,14,16,16,17,18,18,18,18,18,
  19,19,19,21,22,25|...]}


This function call took just over 2 seconds to run, roughly the same time as qsort(), yet it involved spawning 25,000*(1 – 0.5^15)/(1 – 0.5) ~= 50,000 processes! Now the benefits of concurrency start being more obvious.

Can you think of ways to improve performance further? Let me know!

Check out the announcement on the developer blog about a project I’ve been working on at Facebook . It’s a feature that lets you create your own FBML tags for Platform apps.

Francesco Cesarini and I will be giving a talk about Erlang at the San Francisco Bay Area ACM chapter on Nov 19th. The full description is at sfbayacm.org. Please come by if you’re interested!

Last week, I attended the Numenta workshop. I didn’t know much about Numenta before I went. My friend’s excitement about the technology Numenta is building piqued my curiosity, so I decided to check it out. It seemed that almost everyone else in the conference had read Jeff Hawkins’s On Intelligence and at least experimented with Numeta’s tools, so I felt like a real n00b. I’m happy I went, though, because I learned about some interesting ideas and technologies.

Jeff Hawkins, Numenta’s founder, has been fascinated with the workings of the brain throughout his career, but only two decades into it, after he founded Palm and Handspring, was he able to devote his efforts to artificial intelligence. In On Intelligence, Hawkins discusses his theories on the brain’s functions in detail. Numenta, a company he founded with Dileep George and Donna Dubinsky, aims to put these ideas to work in commercial and research applications.

Numenta is a platform company. The platform they develop is NuPIC (Numenta Platform for Intelligent Computing), a software toolkit essentially for building pattern classifiers. The fundamental concept behind NuPIC is called HTM (Hierarchical Temporal Memory). It postulates that the cortex learns to recognize patterns using a combination of two basic algorithms: hierarchical belief propagation, and the detection of invariants in a sequence of transformations in time. I won’t get into what this all means because there’s plenty of documentation on the Numenta website. I recommend browsing it if you find this interesting.

HTM is not just theory. Although NuPIC is in a very early stage, companies are applying NuPIC to a wide range of problems, including vision, voice recognition, finance, motion recognition (recognizing motion capture data to detect if a person is walking, running, sitting, etc) and games. This is just a small subset of its potential uses.

Using Nupic in its current state isn’t trivial. It provides the building blocks for HTM pattern classifiers, but application developers still have to do a good deal of work to tune the parameters of their HTM (How many nodes? How many levels in the hierarchy? How much training data to use? How many categories? What transformations to apply to the input over time to train the system?) to their problem domain. Also, some important features haven’t been implemented yet. For example, although NuPIC can be pretty effective at classifying images that contain a single object against a plain background (with enough training), it isn’t designed to recognize objects in images with noisy backgrounds or with multiple objects. (The problem of how to identify interesting objects in a scene is called the “attention” problem. To solve it you need to have a mechanism by which the top nodes could send feedback down to the bottom nodes. Hawkins said Numenta will tackle it in a future release.)

One reason I find Numenta so interesting is that I believe that NuPIC, or something like it, will play a role in the evolution of the Web. The current generation of web applications is effective at aggregating massive amounts of data in different verticals (pictures, videos, bookmarks, status messages, paintings), slicing and dicing it in different ways, searching it, and displaying it in an organized fashion. Mashups provide additional context for the data gathered in the different silos of the web (Kosmix is a good example), but they don’t add any real “intelligence” to the mix, i.e. they don’t extract new knowledge from the data they aggregate. Numenta’s technology could be used to implement a new layer of intelligence on top of existing services by training it to recognize spacial and temporal pattens in the data they’ve collected. For example, imagine a Flickr API that let you submit an image and Flickr would tell you what the objects in the image are and where the picture was taken. Or a Facebook API for identifying the people in a picture. Or a Skype API for recognizing the speaker from a voice sample (creepy, I know). Or a HotOrNot API for automatically classifying the hotness of a person (ok, bad example :) ). Or a YouTube API for identifying the objects and events in a video clip. Or a icanhascheezburger API for automatically classifying the LOLness of a cat (well… maybe not :) ).

If this happens, maybe some day a mashup of these web services will be used to build something that resembles real AI. If (when?) someone manages to build a real-life WALL*E (great movie!), I think there’s a good chance its HTMs will be trained on the vast amounts of data gathered on the web.

Since its launch, Twoorl users have helped translate it to Spanish, German, French, Korean, Polish, Portuguese (Brazilian) and Russian. This is an awesome contribution from the Twoorl community. Big thanks to everyone who contributed a translation!

If you’re fluent in a language that Twoorl hasn’t been translated into and you’d like to contribute a translation for it, obtain the file twoorl_eng.erl, translate the english strings, and email me the modified file. (Please make sure the file is encoded in UTF-8.) If you’re familiar with Git, you can also clone the repository, make the changes, and send me a message through GitHub to pull your updates. Thanks in advance!

With the recent brouhaha over Twitter’s scalability problems, I thought, wouldn’t it be fun to write a Twitter clone in Erlang?

Last weekend was cold and rainy here in Palo Alto, so I sat down and hacked one, and thus Twoorl was born. It took me one full day plus a couple of evenings. The codebase is about 1700 lines (including comments). You can get it at http://code.google.com/p/twoorl

Note: you need the trunk version of ErlyWeb to make it work (when released, it will be the 0.7.1 version).

Many people written about Twitter’s scalability problems and how to solve them. Some have blamed Rails (TechCrunch is among them), whereas others, including Blaine Cook, Twitter’s Architect, have convincingly argued that you can scale a webapp written in any language/framework if you’ve figured out how to Just Add More Servers to handle the growing traffic. Eran Hammer-Lahav wrote some of the most insightful articles on the subject, On Scaling a Microblogging Service.

I have no idea why Twitter is having a hard time scaling. Well, I have some suspicions, but since I haven’t been in the Twitter trenches, such speculation isn’t worth wasting many pixels on.

I didn’t write a Twitter clone in Erlang because I thought my implementation would be inherently more scalable than a Rails one (although it may be cheaper to scale because Erlang has very good performance) . In fact, Twoorl right now wouldn’t scale well at all since I prioritized simplicity above all else.

The reasons I wrote Twoorl are:

- ErlyWeb needs more open source apps showing how to use the framework. It’s hard to pick how to use the framework just from the API docs.
- Twitter is awesome. Once you start using it, it becomes addictive. I thought it would be fun to write my own.
- Twitter is very popular, but I don’t know of any open source clones. I figured somebody may actually want one!
- Some people think Erlang isn’t a good language for building webapps. I like to prove them wrong :)
- Although you can scale pretty much anything, your choice of language can make a difference in of performance and stability, both of which lead to happy users.
- I think Erlang is a great language for writing a Twitter clone because Twitter’s functionality offers interesting opportunities benefit from concurrency. Here are a couple of ideas I thought of:

1) If you use sharding, the Tweets for different users would be stored in separated databases. When you render the page for someone’s timeline, wouldn’t it be advantageous to fetch the tweets for all the users she follows in parallel? In Ruby, you would probably do something like this:


def get_tweets(users)
  var alltweets = Array.new()
  users.each { | user |
    alltweets.add(user.fetch_tweets())
  }
  alltweets.sort()
  return alltweets
end


(Please forgive any language errors — my Ruby is very rusty. Treat the above as Pseudo code.).

This code would work well enough for a small number of tweet streams, but as the number gets large, it would take a very long time to execute.

In ErlyWeb, you could instead do the following:


get_tweets(Users) ->
  sort(flatten(pmap(fun(Usr) -> Usr:tweets() end, Users)))


This would spawn a process for each user the user follows, fetch the tweets for that user, then reassemble them in sorted order in the original process before rendering the page. (Think of it as map/reduce implemented directly in the application controller.) If a user follows hundreds of other users, querying their tweets in parallel can significantly reduce page rendering time.

2) Background tasks. When a user sends a tweet, the first thing you want to do is store it in the database. Then, depending on the features, you have to do a bunch of other stuff: send IM/SMS notifications, update RSS feeds, expire caches, etc. Why not do those tasks in different background processes? After to write to the DB, you can return an immediate reply to the user, giving him or her the perception of speed, and then let the background processes do all the extra work for processing the tweet.

(Such technique works very well for Facebook apps, by the way. In Vimagi, when the user submits a painting, the app first saves the painting data, and then it spawns a new process to update the news feed and profile box, send notifications, etc.)

Anyway, I hope you enjoy Twoorl. It’s still in very early alpha. It doesn’t have many features and it probably has bugs. Please take Twoorl for a spin and give me your feedback! I’ll also appreciate useful contributions :)