Yariv’s Blog

The title says it all :)

Not only that, I also added more powerful SELECT query capabilities to many-to-one relations.

Here’s how it works:

Add these tables to erlydb.sql


create table developer(
        id integer auto_increment primary key,
        name varchar(30),
        country varchar(20));
 
create table developer_project(
        developer_id integer,
        project_id integer,
        primary key(developer_id, project_id));


This is our modified project.erl file:


-module(project).
-compile(export_all).
 
relations() ->
    [{many_to_one, [language]},
     {many_to_many, [developer]}].


This is our new developer.erl files:


-module(developer).
-export([relations/0]).
 
relations() ->
    [{many_to_many, [project]}].


Following is some code demonstrating the new capabilities. The full module is in the distribution, in test/erlydb_test.erl.


    %% fancier project queries
    {ok, [Yaws4]} = language:projects_where(
          Erlang, "name='Yaws'"),
    true = Yaws4 == Yaws1,
 
    {ok, [Yaws5]} = language:projects_with(
          Erlang, "limit 1"),
    true = Yaws4 == Yaws5,
 
    {ok, [Ejabberd4]} = language:projects(
        Erlang, "name like '%e%'", "limit 1"),
    true = Ejabberd4 == Ejabberd3,
 
    %% Let's show off some many-to-many features
 
    %% First, add some more projects
    [OTP, Mnesia] =
  lists:map(
    fun(Proj) ->
      {ok, P1} = project:save(Proj),
      P1
    end,
    [project:new("OTP", "The Open Telephony Platform", Erlang),
     project:new("Mnesia", "A distributed database " ++
           "engine written in Erlang", Erlang)]),
 
    %% Next, add some developers
    [Joe, Ulf, Klacke] =
  lists:map(
    fun(Developer) ->
      {ok, D1} =
          developer:save(Developer),
      D1
    end,
    [developer:new("Joe Armstrong", "Sweden"),
     developer:new("Ulf Wiger", "Sweden"),
     developer:new("Claes (Klacke) Wikstrom", "Sweden")]),
 
    %% Add some developers to our projects
    ok = project:add_developer(OTP, Joe),
    ok = project:add_developer(OTP, Klacke),
    ok = project:add_developer(OTP, Ulf),
    
    %% Add some projects to our developers
    ok = developer:add_project(Klacke, Yaws1),
    ok = developer:add_project(Klacke, Mnesia),
    ok = developer:add_project(Ulf, Mnesia),
 
    
    %% basic SELECT query
    {ok, [Joe, Ulf, Klacke]} =
  project:developers(OTP),
 
    %% fancier SELECT queries
    {ok, [Ulf, Klacke]} =
  project:developers(Mnesia),
    {ok, [Ulf]} = project:developers_with(Mnesia,
            "limit 1"),
    {ok, [Klacke]} =
  project:developers_where(Mnesia,
         "name like 'Claes%'"),
    
    %% SELECT query, from the other direction
    {ok, [Yaws1, OTP, Mnesia]} =
  developer:projects(Klacke),
 
    %% Klacke, nothing personal here :)
    ok = developer:remove_project(Klacke, Yaws1),
    {ok, [OTP, Mnesia]} = developer:projects(Klacke),
    ok = developer:remove_project(Klacke, OTP),
    {ok, [Mnesia]} = developer:projects(Klacke),
    ok = developer:remove_project(Klacke, Mnesia),
    {ok, []} = developer:projects(Klacke),
 
    %% I forgot to show you can delete records, too :)
    ok = language:delete(Java).


ErlyDB is still in alpha version, so you may have problems. Please let me know if you find any bugs :)

(FYI, I’m moving to Boston tomorrow, so I will probably not be of much help in the next few days.)

Enjoy!

Every web developer craves a database abstraction layer. Ruby, PHP, Java and Python all have them. Many developers have adopted web frameworks just for their database abstraction capabilities. The lack of such a framework has even caused some developers to hesitate before using the best language of them all for building world-class backends: Erlang.

That’s why I created ErlyDB: The Erlang Twist on Database Abstraction :)

ErlyDB isn’t a database abstraction layer, but a database abstraction layer generator. ErlyDB taps into Erlang’s runtime metaprogramming powers to generate an abstraction layer for your database on the fly. This layer is as flexible as it is paper-thin.

Unlike ORM frameworks, ErlyDB works directly with Erlang tuples, which are simple immutable arrays that map very easily to database records. This incurs less overhead than object-relational mapping solutions, which rely on more heavyheight objects to represent database rows.

With ErlyDB, productivity doesn’t come at the expense of performance :)

ErlyDB is in an alpha stage right now. It only supports MySQL; it hasn’t been thoroughly tested and optimized; and it’s not feature-complete by any means. However, I’ll be adding many more features and improvements over time. If you strongly need a feature before I implement it, I encourage you to implement it yourself and share it with the rest of us. I will be accepting any useful, high quality contributions from other developers.

Let’s do a short tutorial.

• Get ErlyDB from the subversion repository and install it in your code path (on my Mac, it’s /usr/local/lib/erlang/lib).
• Get the MySQL driver from process-one.net and install it in your code path.
• Create a MySQL database called ‘test’
• Run the following SQL script:


create table language (
        id integer auto_increment primary key,
        name varchar(50),
        paradigm varchar(30),
        creation_year integer);
 
create table project (
        id integer auto_increment primary key,
        name varchar(50),
        description varchar(50),
        language_id integer,
        index(language_id));


• Create the file “language.erl” with the follwing code and compile it in your code path:


-module(language).
-export([relations/0]).
 
relations() ->
    [{one_to_many, [project]}].


• Create the file “project.erl” with the following code and compile it in your code path:


-module(project).
-export([relations/0]).
 
relations() ->
    [{many_to_one, [language]}].


• Implement the following module, which illustrates some of the common idioms of ErlyDB.


-module(erlydb_test).
-export(
   [init/0,
    test/0]).
 
init() ->
    %% connect to the database
    erlydb:connect(mysql, "localhost", "username", "password",
      "test"),
 
    %% generate the abstraction layer modules
    erlydb:code_gen([language, project]).
 
test() ->
    init(),
    
 
    %% clean up old records
    erlydb:q("delete from language;"),
    erlydb:q("delete from project;"),
    
    %% Create some new records
    Languages =
  [language:new("Erlang",
            "A fun and productive functional language with " ++
            "top-notch concurrency features designed for "++
            "building scalable, fault-tolerant systems. " ++
            "Erlang is an excellent choice for building "++
            "anything from websites like hamsterster.com " ++
            "to commercial phone switches.",
            "functional/dynamic/concurrent", 1981),
   language:new("Java",
            "An OO language designed to make the lives of " ++
            "C++ programmers less painful.",
            "OO/static", 1992),
   language:new("Ruby",
            "An OO language designed to make the lives of " ++
            "Java and Perl programmers less painful.",
            "OO/script/dynamic", 1995)],
 
    %% Save the records in the database and collect the updated
    %% tuples.
    [Erlang, Java, Ruby] =
  lists:map(
    fun(Language) ->
      %% executes an INSERT statement
      {ok, Lang} = language:save(Language),
      Lang
    end, Languages),
 
    %% demonstrate getters
    "Erlang" = language:name(Erlang),
    "functional/dynamic/concurrent" =
        language:paradigm(Erlang),
    1981 = language:creation_year(Erlang),
 
    %% demonstrate setter
    J1 = language:creation_year(Java, 1993),
 
    %% executes an UPDATE statement
    {ok, J2} = language:save(J1),
 
    1993 = language:creation_year(J2),
 
    %% Lets run some queries
    {ok, E1} = language:find_id(language:id(Erlang)),
    true = E1 == Erlang,
    
    {ok, [E2]} = language:find_where("name = 'Erlang'"),
    true = E2 == Erlang,
 
    {ok, [E3]} = language:find(
      "WHERE paradigm = 'functional/dynamic/concurrent' " ++
       "LIMIT 1"),
    true = E3 == Erlang,
    {ok, [E4, J4, R4]} = language:find_all(),
    true =
  E4 == Erlang andalso
  J4 == J1 andalso
  R4 == Ruby,
    
    %% Let's make some projects
    
    Yaws = project:new(
   "Yaws", "A beautiful web server written in Erlang",
          Erlang),
    Ejabberd = project:new("ejabberd",
         "The best Jabber server, hands down", Ruby),
    OpenPoker =
  project:new("OpenPoker",
            "A scalable, fault-tolerant poker server", Erlang),
 
    %% We call language:id just to demonstrate that constructors
    %% accept both related tuples or, alternatively, their id's.
    %% This example would behave identically if we used the
    %% Java variable directly.
    Tomact =
  project:new("Tomcat",
            "A Java Server with XML config files",
             language:id(Java)),
 
    JBoss =
  project:new("JBoss",
            "A Java Application Server with more XML files",
            Java),
    Spring =
  project:new("Spring Framework",
            "A Java IoC framework oozing XML love", Java),
 
    
    Mongrel =
  project:new("Mongerl",
            "A web server with a funny name", Ruby),
    Rails =
  project:new("Ruby on Rails",
               "A nice integrated web framework for building " ++
               "CRUD websites.", Ruby),
    Ferret = project:new("Ferret",
           "A Ruby port of Apache Lucene. It would be nice " ++
           "if someone ported it to Erlang", Ruby),
    Gruff = project:new("Gruff",
        "A Ruby library for easy graph generation. " ++
        "An Erlang implementation would be nice as well",
         Ruby),
 
    Projects =  [Yaws, Ejabberd, OpenPoker, Tomact, JBoss,
       Spring, Mongrel, Rails, Ferret, Gruff],
    
    %% Insert our projects into the database
    [Yaws1, Ejabberd1, OpenPoker1 | _Rest] =
  lists:map(
    fun(Project) ->
      {ok, P1} = project:save(Project),
      P1
    end, Projects),
    
    %% let's get the language associated with Yaws
    {ok, Erlang2} = project:language(Yaws1),
    true = (Erlang2 == Erlang),
    
    %% now let's correct a grave error
    {ok, Ejabberd2} = project:save(
         project:language(Ejabberd1, Erlang)
        ),
    true = language:id(Erlang) ==
       project:language_id(Ejabberd2),
 
    %% let's get all the projects for a language
    {ok, [Yaws3, Ejabberd3, OpenPoker3]} =
          language:projects(Erlang),
    true =
  Yaws3 == Yaws1
  andalso Ejabberd3 == Ejabberd2
  andalso OpenPoker3 == OpenPoker1,
 
    ok.


(This code is included in the ‘test’ directory of the ErlyDB distribution.)

• Read the code and experiment.

As you’ve probably guessed, ErlyDB uses Smerl to do its magic. (Confession: when I created Smerl, ErlyDB was just what I had in mind :) ) ErlyDB takes advantage of all of Smerl’s advanced features, such as metacurrying, parameter embedding and module extension. You may find it interesting to look at the code and see how ErlyDB works under the hood.

Adhering to the Erlang philosophy of zero downtime, the ErlyDB abstraction layer can change in runtime without taking the system offline. Simply call erlydb:code_gen with the module names to regenerate and everything will work as expected.

Please get your hands on ErlyDB and give it a test drive. Let me know if you find any bugs or if you have any suggestions.

Coming very soon:

• Many-to-many relations
• Transactions
• Event handlers (before_save, after_save, etc.)

Coming soon:

• Prepared statements
• Support for additional database engines (probably when edbc comes out, unless other people make temporary drivers beforehand.)
• Better connection pooling (also probably tied to edbc)
• Real documentation :)
• Performance optimizations
• More customizations
• Fine-grained control on field visibility
• Multiple models per table
• More versatile SQL query generation
• Support for complex queries
• and more

Enjoy!

Appendix

Erlang users have actually had a nice alternative to database abstraction: Mnesia. Mnesia is an industrial-strength, distributed database engine written in pure Erlang. Although Mnesia is quite beautiful, it isn’t the best fit for all applications. Mnesia’s biggest drawback (at least as I see it) is that Mnesia isn’t designed to scale to very large (many gigs) data volumes. I’m hoping the OTP team will improve this aspect of Mnesia, but until it does, we have to live with this limitation.

Even if Mnesia didn’t have this shortcoming, many developers would still prefer to use other database engines for their various capabilities. MySQL, Postgres and Oracle aren’t going away any time soon.

Having said that, Mnesia can very useful even if you’re using another database engine. For instance, a killer application for Mnesia is a distributed session store shared between instances of a Yaws cluster. Mnesia could also be used as a distributed cache for recently-accessed database records.

Update: I just realized that there’s no reason ErlyDB shouldn’t have a Mnesia driver. Any takers? :)

2 Quick updates:

• I made source discovery in Smerl more robust by using the module_info function.
• I changed the Smerl license from the BSD to the MIT license.

Dear readers,

I’d like to take a short break from my Erlang postings and share a few pictures I took on my trip earlier today to the Museum of Natural History in New York.

Note: I did some retouching with GIMP (actually, GimpShop for OS X) to enhance the dynamic range of these images, as the originals suffered from faded colors due to the poor lighting conditions.

A Velociraptor

A green lizard

This dino must have been coding in Java. Its neck looks like this stack trace :)

A chameleon

You can view the rest of the set here

Hi,

I’m curious as to who’s planning on going to the Fifth ACM SIGPLAN Erlang Workshop in Portlang on the 16th (the link is on the front page of http://erlang.org).

I haven’t made up my mind yet if I should go. It would be fun to meet some Erlangers face to face but I have no idea how popular this workshop is.

I’d appreciate it if somebody could clue me in.

Thanks in advance!

The picture says it all.

This is weird.

When I started blogging about Erlang, I enjoyed being in a small niche of bloggers that write about “exotic” languages. I’ve never thought my blog would get all this attention. Sometimes, I don’t like all this attention :) However, you don’t always get in life what you’ve expected, so I’ll do my best to enjoy the ride.

Erlang is just getting started. Erlang will make huge strides into the web development world. We will build frameworks that make Erlang developer’s lives easy, if not easier, than those of Ruby on Rails developers. Ruby on Rails didn’t exist two years ago, and Erlang (with Yaws) is years ahead of where Ruby was when Rails was born. In a matter of months, Erlang will be a serious contender in the web development world — not just for developers who want insane scalability with 3 milliseconds downtime per year — but also for developers who just want to get things done.

Real-time interactivity on the web is a relatively unexploited land full of opportunities for adventerous developers. These days, building real-time interactive applications with Comet technology in any language but Erlang just makes no sense. Erlang has 20 years of development behind it with the specific design goal of building scalable servers. No other language comes close. It also makes development fun.

As Joe Armstrong has said, Erlangers have a head start.

Libraries are coming.

As you may have noticed, Ericsson hasn’t done the best PR for Erlang. The closest things resembling sales pitches I see from that part of the world are either academic-looking papers or PowerPoint presentations that require downloading heavy PDF files. There are also the mailing list postings that are often intelligent and hilarious but they are ultimately preaching to the choir.

When people talk about Erlang, they usually mention how Erlang is great for building concurrent, distributed, highly-available, fault-tolerant, systems; it powers phone switches with nine nines availability; the AXD 301 switch has 850K lines of Erlang code; Yaws scales with large numbers of connections; ejabberd is the most scalable XMPP server; and all sorts of such heavy-sounding accomplishments.

If I hadn’t read Joel Reymont’s wagerlabs.com blog, where he wrote about his experiences writing a poker server in Erlang, I probably would have never thought I would actually like Erlang as a language.

What nobody from Ericsson talks about is that Erlang makes coding FUN.

It may be possible to arrive at this realization through deduction: to build large-scale, stable systems you need a productive language. If the language got in your way and prevented you from doing things in a straightforward manner, your code would start growing in complexity. Complexity leads to bugs. Bugs lead to downtime. Systems written in Erlang must have close to zero downtime. Therefore, Erlang must make developer’s lives easy.

For developers, ease often equates with fun. Ease means that they can express their thoughts succinctly, test their solutions quickly, and then move on to the next problem.

Programmers love garbage collection, for instance, because it makes their lives easy. Similarly, some programmers are Ruby fanatics because they think Ruby is the epitome of ease (I used to think so too in the old days — about 3 months ago :) ). Other programmers believe ease comes from using functional languages, because they are the most powerful.

You can read Paul Graham’s essays on Lisp about this. You can also read Peter Norvig’s comparison between Python and Lisp. (As you’re reading this comparison, mentally add Erlang as well as 2 more categories: concurrency and fault tolerance, and picture Erlang as being the top dog :) )

So what exactly makes Erlang coding fun? To me, these are the main factors:

• Erlang is a functional language. It fits well in my brain.
• Erlang is dynamically typed, which means that I can write less code (read Haskell vs. Erlang, Reloaded) and also test my code easily in the Erlang shell. (Erlang’s dynamic nature also allowed me to create Smerl :) )
• Erlang has pattern matching, which lets me write a single line of code to express a thought that would in other languages translate into a horrendous structure of nested if-else statements.
• Erlang has single-assignment semantics, which means that functions have no side effects (on bound variables). This makes Erlang code very readable. It also makes debugging easy.
• Concurrency, concurrency, concurrency.
• With Erlang’s hot code swapping, I can deploy my changes unto a system while it’s running. When I made the haXe remoting adapter for Yaws, I didn’t have to take Yaws offline (try that with Apache or Lighttpd :) ).
• The compiler checks the validity of my code without locking me into a type system.
• Good runtime metaprogramming capabilities (with Smerl).
• I used to like Eclipse, but Erlang made me an Emacs fiend :)
• I’m learning a lot and interacting with very smart people.

Some people have asked me, “If all I want is to build a stateless, CRUD-style webapp, and I don’t need telcom-grade scalability, etc, why should I bother learning Erlang? Ruby on Rails handles it just fine. It also has more web development libraries.”

I hope I was able to give a reasonable answer to this question. However, as much as I write about Erlang, reading this blog won’t take you all the way there. You have to experiment with Erlang and see how it feels for yourself. I can’t guarantee you’ll like it — all I can do is tell you why I do.

There’s big news in the Erlang blogsphere.

Today, Joe Armstrong, one of Erlang’s original creators (”The” creator of Erlang?), started a new blog in which he’ll write about the events and people that have shaped Erlang’s history, as well as the stories that will shape its future.

This is very exciting. Until today, the most comprehesive source for Erlang history I’ve read has been Joe’s PhD thesis. However, as you might expect, although this paper is very rich in technical material, it doesn’t dwell too much on the human side. Now, we’ll be able to know more about the Erlang story from the perspective of a true insider.

Judging by the substance and humor of Joe’s mailing list postings, which I sometimes quote on this blog, I’d bet that Joe’s blog will be a great read.

I hope more people follow in Joe’s footsteps and start blogging about Erlang. Many people on the Erlang mailing list are far more knowledgable about Erlang — and other programming languages (well, maybe except for Ruby and haXe :) ) — than I am. Many of them are also great writers. If they started blogging, it would make the Erlang blogsphere much richer.

I get much of my knowledge about Erlang from reading mailing list postings, but this kind of knowledge isn’t very accessible to non-Erlangers. Blogging is a better tool for spreading the word to the outside world. Blogs also can be less technical and more open-ended than mailing list postings, in which one follows a stricter etiquette in trying to stick to the point and not waste too much of people’s time. A blog, on the other hand, is what you make of it. If a blog is good, people discover it and its importance grows. If a blog sucks, at least people have the option of ignoring it :)

The best part of Joe’s announcement, at least for me, is that he credited me with inspiring him to start blogging. Reading these words made me feel very honored.

Yariv’s blog http://yarivsblog.com/ with his frequent Erlang postings
has inspired me.

I think he needs a little competition :-)

So I’ve started writing …

In this blog I’ll try to tell you some of the things that are happening in the Erlang world. A lot of stuff is happening behind the scenes, I’ll try to tell you about some of this.

The first post is:

http://armstrongonsoftware.blogspot.com/2006/08/making-money-from-erlang.html

Cheers

/Joe

When I stared this blog, I really didn’t have very high expectations, so it’s thrilling to see this blog make a real impact on the Erlang world. (I hope that the world outside has taken notice as well :) )

Getting credit is great, but giving it is even better: it was Joel Reymont’s wagerlabs.com blog which has turned a mild curiousity about Erlang into a strong interest for me and also inspired me to start blogging about Erlang.

Oh, I almost forgot to mention — Joe will also blog about cats. I must not forget the cats! :)

Happy RSSing! :)

I added a few new features to Smerl earlier today.

• smerl:extend/2 now injects the parent/0 function, which returns the parent module’s name, into the child module. This allows explicitly calling parent functions from the child module. (Using smerl:for_module/1, the child can also mutate its own parent in runtime, which puts an interesting twist on family relationships :-) )
• smerl:to_src/1 returns the pretty-printed source code for the MetaMod object.
• smerl:to_src/2 writes the pretty-printed source code for the MetaMod object into a file.

Enjoy!

Have you ever wished you could easily add all exports from a “parent” module to a “child” module in the form of remote function calls to the parent, without overwriting the child module’s exported functions?

If you have, today is your lucky day! With Smerl, you can now do it in a single line of code by calling smerl:extend/2 :)

I know words like ‘extension’ and ‘inheritance’ are generatlly shunned in the functional programming world, and for a good reason: in OO languages, these concepts have caused endless complexity, crufty code and bug-ridden horrors.

However, I should stress that the Smerl extension mechanism doesn’t violate any functional programming tenets. Smerl’s extend/2 is basically a smart function generator. It does not allow you to extend data fields in any way, nor does it create any artifical bindings between data and code.

(I believe Haskell has a similar, statically typed approach to module inheritance, but I only have rudimentary knowledge of Haskell so I may be wrong on that one.)

Here’s a quick example:


P1 = smerl:new(parent),
{ok, P2} = smerl:add_func(P1, "add(A, B) -> A+B."),
 
%% 'false' indicates to not export the function
{ok, P3} = smerl:add_func(P2, "subtract(A, B) -> A-B.", false),
 
{ok, P4} = smerl:add_func(P3, "multiply(A,B) -> A*B."),
smerl:compile(P4),
 
C1 = smerl:new(child),
{ok, C2} = smerl:add_func(C1,
   "multiply(A,B) -> throw(not_allowed)."),
 
C3 = smerl:extend(P4, C2),
smerl:compile(C3),
 
8 = parent:add(3,5),
{'EXIT', {undef, _}} = begin catch parent:subtract(4,3) end,
6 = parent:multiply(3, 2),
11 = child:add(2,9),
not_allowed = begin catch child:multiply(5,3) end,
{'EXIT', {undef, _}} = begin catch child:subtract(4,2) end.


Seems exotic? Maybe. However, I think this feature is very powerful. I have actually seriously needed this feature in my own project. If you have needed it too, I hope I just made your life a bit easier :)

Please let me know if you have any problems or suggestions.

Enjoy.

Update (8/23/06): The extend/2 function now also embeds the parent/0 function in the child module. parent/0 returns an atom with the name of the parent’s module. This allows child functions to explicitly invoke parent functions (among other things).