Yariv’s Blog

In my last post I described how to use LFE to overcome some of the weaknesses of parameterized modules. Unfortunately, all is not rosy yet in the land of LFE types. Parameterized modules allow you to create only static types. The compiler doesn’t do static type checking, but you have to define the properties of your types at compile time. This works in many cases, but sometimes you want totally dynamic containers that map keys to values. In Erlang, this is typically done with dicts. We could still use them with LFE, but I don’t like having different methods of accessing the properties of objects depending on whether their types were defined at run time or compile time.

Let’s use macros to solve the problem.

In my last post, I relied on the build-in ‘call’ function to access the properties of static objects. Let’s create a wrapper to ‘call’ that lets us access the properties of dicts in exactly the same manner as we do properties of other objects:


(defmacro dot
  ((obj prop . tl)
   `(let ((obj ,obj))
      (case (element 1 obj)
        ('dict
         ,(case tl
            (()
             `(case (: dict find ,prop obj)
                ('error 'undefined)
                ((tuple 'ok valx) valx)))
            (_ `(: dict store ,prop ,(hd tl) obj))))
        (_ (call obj ,prop ,@tl))))))


We can use ‘dot’ to get and set the properties of both dicts and static objects:


(defun test ()
  (let* ((dog1 (: dog new))
     (dog2 (: dict new))
           (dog1_2 (dot dog1 'name 'lola))
           (dog2_2 (dot dog2 'name 'rocky)))
    (list (dot dog1_2 'name)
           (dot dog2_2 'name))))



> (: dog test)                                
(lola rocky)


I think this is kinda of cool, though to be honest I’m not entirely sure it’s a great idea to obfuscate in the code whether we’re dealing with dicts or static objects.

One of the features I dislike the most about Erlang is records. They’re ugly and they require too much typing. Erlang makes me write


Dog = #dog{name = "Lolo", parent = #dog{name = "Max"}},
Name = Dog#dog.name,
ParentName = (Dog#dog.parent)#dog.name,
Dog1 = Dog#dog{
   name = "Lola",
   parent = (Dog#dog.parent)#dog{name = "Rocky"}}


When I want to write


Dog = #Dog{name = "Lolo", parent = #dog{name = "Max"}},
Name = Dog.name,
Size = Dog.size,
Dog1 = Dog{
  name = "Lola",
  parent = Dog.parent{name = "Rocky"}}


In the defense of records, they’re just syntactic sugar over tuples and as such they enable fast access into a tuple’s properties despite Erlang’s dynamic nature. In compile time, they’re converted into fast element() and setelement() calls that don’t require looking up the property’s index in the tuple. Still, I dislike them because 1) in many cases, I’d rather optimize for more concise code than faster execution and 2) if the Erlang compiler were smarter could allow us to write the more concise code above by inferring the types of variables in your program when possible. (I wrote a rant about this a long time ago, with a proposed solution to it in the form of a yet unfinished parse transform called Recless.)

Parameterized modules provide a somewhat more elegant and dynamic dealing with types, but they still require you to do too much work. You can define a parameterized module like this:

-module(dog, [Name, Size]).

This creates a single function called ‘new/2′ in the module ‘dog’. You can call it as follows:


Dog = dog:new("Lola", "big").


It returns a tuple of the form


{dog, "Lola", "Big"}.


You can’t set only a subset of the record’s properties in the ‘new’ function. This doesn’t work


Dog = dog:new("Lola").


To access the record’s properties you need to define your own getter functions, e.g.


name() ->
  Name.


which you can call it as follows:


Name = Dog:name().


(This involves a runtime lookup of the ‘name’ function in the ‘dog’ module, which is slower than a record accessor).

There’s no way to set a record’s properties after it’s created short of altering the tuple directly with setelement(), which is lame and also quite brittle. To create a setter, you need do the following:


name(Val) ->
    setelement(2, THIS, Val).


Then you can call


Dog1 = Dog:name("Lola").


to change the object’s name.

When LFE came out I was hoping it would provide a better way of dealing with the problem. Unfortunately, records in LFE are quite similar to Erlang records, though they are a bit nicer. Instead of adding syntactic sugar, LFE creates a bunch of macros you can use to create a record and access its properties.


(record dog (name))
(let* ((dog (make-dog (name "Lola")))
     (name (dog-name dog))
     (dog1 (set-dog-name dog "Lolo")))
  ;; do something)


LFE still requires us to do too much typing when dealing with records to my taste, but LFE does give us a powerful tool to come up with our own solution to the problem: macros. We can use macros generate all those repetitive brittle parameterized module getters and setters that in vanilla Erlang we have to write by hand. This can help us avoid much the tedium involved in working with parameterized modules.

(ErlyWeb performs similar code generation on database modules, but it does direct manipulation of Erlang ASTs, which are gnarly.)

Let’s start with the ‘new’ functions. We want to generate a bunch of ‘new’ functions allow us to set only a subset of the record’s properties, implicitly setting the rest to ‘undefined’.


(defun make_constructors  (name props)
  (let* (((props_acc1 constructors_acc1)
          (: lists foldl
             (match-lambda
              ((prop (props_acc constructors_acc))
               (let* ((params (: lists reverse props_acc))
                      (constructor
                       `(defun new ,params
                          (new ,@params 'undefined))))
                 (list
                  (cons prop props_acc)
                  (cons constructor constructors_acc)))))
             (list () ())
             props))
         (main_constructor
           `(defun new ,props (tuple (quote ,name) ,@props))))
    (: lists reverse (cons main_constructor constructors_acc1))))


This function take the module name and a list of properties. It returns a list of sexps of the form


(defun new (prop1 prop2 ...prop_n-m)
  (new prop1 prop2 ... prop_n-m 'undefined))


as well as one sexp of the form


(defun (prop1 prop2 ... prop_n)
  (tuple module_name prop1 prop2... prop_n)


The first set of ‘new’ functions successively call the next ‘new’ function in the chain, passing into it their list of parameters together with ‘undefined’ as the last parameter. The last ‘new’ function takes all the parameters needed to instantiate an object and returns a tuple whose first element is the module, and the rest are the object’s property values. (We need to store the module name in the first element so we can use the parameterized modules calling convention, as you’ll see later.)

Now let’s create a function that generates the getters and setters


(defun make_accessors (props)
   (let*
       (((accessors idx1)
        (: lists foldl
           (match-lambda
            ((prop (acc idx))
             (let* ((getter `(defun ,prop (obj) (element ,idx obj)))
                     (setter `(defun ,prop (val obj)
                        (setelement ,idx obj val))))
                (list (: lists append
                         (list getter setter)
                         acc)
                       (+ idx 1)))))
           (list () 2)
           props)))
     accessors)))


This function takes a list of properties and returns a list of sexps that implement the getters and setters for the module. Each getter takes the object and returns its (n + 1)th element, where n is the position of its property (the first element is reserved for the module name). Each setter takes the new value and the object and returns the tuple after setting its (n + 1)th element to the new value.

Now, let’s tie it this up with the module declaration. We need to create a macro that generates the module declaration, constructors, getters, and setters, all in one swoop. But first, we need to expose make_constructors and make_accessors to the macro by nesting them inside a (eval-when-compile) sexp.


(eval-when-compile
  (defun make_constructors ...)
  (defun make_accessors ...))
 
(defmacro defclass
  (((name . props) . modifiers)
   (let* ((constructors (make_constructors name props))
          (accessors (make_accessors props)))
     `(progn
        (defmodule ,name ,@modifiers)
        ,@constructors
        ,@accessors))))


(defclass returns a `(progn) sexp with a list of new macros that it defines. This is a trick Robert Virding taught me for creating a macro that in itself creates multiple macros.)

Now, let’s see how this thing works. Create the file dog.lfe with the following code


;; import the macros here
 
(defclass (dog name size)
  (export all))


Compile it from the shell:


(c 'dog)


Create a dog with no properties


> (: dog new)
#(dog undefined undefined)


Create a new dog with just a name


> (: dog new '"Lola")
#(dog "Lola" undefined)


Create a new dog with a name and a size


> (: dog new '"Lola" '"medium")
#(dog "Lola" "medium")


Get and set the dog’s properties


> (let* ((dog (: dog new))
         (dog1 (call dog 'name '"Lola"))
         (dog2 (call dog1 'size '"medium")))
    (list (call dog2 'name) (call dog2 'size)))
("Lola" "medium")


This code uses the same parameterized module function calling mechanism that allows us to pass a tuple instead of a function as the first parameter to the ‘call’ function. Erlang infers the name of the module that contains the function from the first element of the tuple.

As you can see, LFE is pretty powerful. I won’t deny that sometimes I get the feeling of being lost in a sea of parenthesis, but over time the parentheses have grown on me. The benefits of macros speak for themselves. They give you great flexibility to change the language as you see fit.

I attended the Bay Area Erlang Factory last week. It was a great event. I met many Erlang hackers, attended interesting talks, learned about cool projects (CouchDB, QuickCheck, Nitrogen, Facebook Chat), gave a talk about ErlyWeb, and drank beer (without beer, it wouldn’t be a true Erlang meetup).

My favorite talk was by Damien Katz. He told the story of how he had decided to take a risk, quit his job, and work on his then amorphous project. He wanted to work on cool stuff, and that was the only way he could do it. Even if nothing else came out of it, he knew it would have been a great learning exercise. Something great did eventually come out of it, as he created CouchDB (which looks awesome btw) and IBM eventually hired him to work on it full time.

Damiens’ story reminded me of the time I started working ErlyWeb a few years ago. After I left the company I was working for at the time, I decided to take a few months and work on something cool. I didn’t know what exactly it would be or how long it would take, but I knew that I wanted to build a product that would help people communicate in new ways, and I wanted to build it with my favorite tools. I knew the chance of failure was high, but I figured the learning alone would be worth it. I also viewed open source as an insurance policy of sorts. Even if I couldn’t get a product off the ground, my code could live on and continue to provide value to people.

Doing it paid off. My savings dwindled, but I learned Erlang, created ErlyWeb and Vimagi, met many like minded people, and it opened new doors. Now I work on cool stuff at Facebook, ErlyWeb lives on, and every day people are using Vimagi to create amazing art and share it with their friends.

The moral of the story: if you’re not working on cool stuff, take a risk and try to make it happen. Don’t worry about building the next Google or making lots of money, because you’ll probably fail. But the lessons you learn and the connections you make will be worth it.

Check out the announcement on the developer blog about a project I’ve been working on at Facebook . It’s a feature that lets you create your own FBML tags for Platform apps.

Francesco Cesarini and I will be giving a talk about Erlang at the San Francisco Bay Area ACM chapter on Nov 19th. The full description is at sfbayacm.org. Please come by if you’re interested!

I attended startup school on Saturday. It was a great experience and a rare opportunity to hear to a such impressive speakers share their wisdom about technology and entrepreneurship. Some of my favorite talks were by David Heinemeier Hansson, Greg McAdoo, Marc Andreesen, Paul Buchheit and Michael Arrington. I met a bunch of programmers and entrepreneurs and I also chatted briefly with DHH about 37signals and Peter Norvig about new search startups and their chances of competing with Google. Many thanks to YCombinator for organizing such a great event!

If you haven’t attended, you can see all the videos here. Highly recommended.

I just caught Amazon’s announcement of the new persistent storage engine for EC2. This is great stuff. It lets you create persistent block level storage devices ranging from 1GB to 1TB in size and attach them to EC2 instances in predetermined availability zones. This service complements Amazon’s other storage services — EC2 and SimpleDB — in providing raw block-level storage devices that are persistent, fast and local (so you don’t have to worry about SimpleDB’s eventual consistency issues). You can use these volumes for anything — running a traditional DBMS (MySQL, Postgres) is the first thing that comes to mind.

This announcement is a departure from Amazon’s tradition of announcing services only once they become available. It looks like Amazon is feeling the heat of competition from Google App Engine and is becoming more open to win over the hearts and minds of developers who are drawn to GAE for its auto-magical scalability. The ability to attach multiple terabyte-sized volumes on demand alleviates some of those concerns when deploying on Amazon’s infrastructure. I’m sure it won’t be long before someone creates an open source BigTable-like solution for applications that need massive scalability and redundancy on top of multiple persistent storage volumes (I think this would be a great application to write in Erlang, but I don’t know how well Erlang performs in applications that require heavy disc IO).

I like what Amazon is doing. By providing the basic building blocks for scalable applications, its enables startups to create their own GAE competitors (Heroku is the first one that comes to mind) on top of Amazon’s infrastructure. Smart move.

Google has the advantage of being able to provide APIs for tight integration with other Google services such as authentication and search (the latter is hypothetical as of now). We’ll see how strongly this plays in Google’s favor in the coming months.

Of course, price is still a question mark. Neither Amazon’s persistent storage service nor GAE have had their prices announced.

Another missing detail is the Amazon store service’s reliability. If a disc fails, do you lose your data? What’s the failure probability? Etc.

All this is great for developers. Competition between Amazon and Google means developers will enjoy more services and for lower prices in the coming years.

Nick Gerakines wrote a good tutorial on how to make tag clouds in ErlyWeb. Check it out at http://blog.socklabs.com/2008/04/tag_clouds_in_erlang_with_erly/.

Erlang is *almost* a tipping point. Thanks to reddit, many people are interested in it. However, it’s not there yet. Despite being the only language that got concurrency right, and all its other standout features, many developers still use other languages. The only explanation I can think of is that Erlang hasn’t had much PR over the years (the Erlang movie nonwithstanding). I’m confident that a good PR boost will help push Erlang over the hill. Unfortunately, Ericsson doesn’t seem interested in heavily promoting Erlang the way Microsoft promotes .NET and Sun promotes Java (this may be because many Ericsson employees have never heard of Erlang). So, I decided to take things into my own hands. I don’t have a budget, so I need to get creative. The best way to succeed if you’re small is to ride a big wave — and what’s a better wave to ride than Ruby on Rails?

Ruby on Rails is very popular — much more than ErlyWeb. I believe this popularity is due to the “on Rails” meme, which is just bursting with positive connotations. It sounds young, fresh, happy. It’s the anti-enterprisy software. It emancipates you from burdensome type systems, explicit getters and setters, and (ugh) XML. Its metaprogramming wizardry is made of bliss. Its evokes images of riding in environmentally-friendly transportation looking out the window at grassy meadows, rolling hills and sunny skies.

I think that renaming ErlyWeb to “Erlang on Rails” will help win over the hearts and minds of many programmers who are currently on the fence. They may be curious about Erlang but are turned off by its telcom image. “Erlang on Rails” conveys a more balanced feeling of industrial strength applications from the telcom world mixed with the social Web 2.0 era of interconnectedness that celebrates the rise of individualism over grey corporate culture.

2008 will be the year of Erlang on Rails. I know it.

Update: This was an April Fool’s joke, in case it’s not obvious anymore :)

I just hacked together an (alpha) implementation of the FriendFeed API for Erlang. You can get the code at http://code.google.com/p/erlang-friendfeed/. Enjoy!

By the way, my shiny new FriendFeed is at http://friendfeed.com/yariv.