Comments on: The Erlang Challenge

By: The (Unofficial) Erlang Blog » Blog Archive » Erlang Challenge Announced

Sun, 17 Feb 2008 22:59:12 +0000

[...] last week’s Arc release and subsequent Arc challenge, Yariv Sadan (ErlyWeb) has announced the Erlang Challenge. The goal is to design & program an advanced “parallel map” function that can [...]

By: Martial

Martial — Mon, 11 Feb 2008 10:19:48 +0000

I do like Erlang programming but the Smalltalk continuation-based solution with Seaside looks like the simpler way:
http://www.lukas-renggli.ch/blog/take-the-arc-challenge
Erlang has the advantage of low memory consumption and ‘infinite’ scalability.

By: Yariv’s Blog » Blog Archive » More Erlang Fun: Distributed, Fault Tolerant MapReduce

Mon, 11 Feb 2008 02:13:05 +0000

[...] the Erlang Challenge posting, I showed how to implement a simple distributed parallel map function. You can pass it a [...]

By: Yariv

Yariv — Sat, 09 Feb 2008 23:42:29 +0000

@Bob There is a slight difference between the use of SpawnFun in this example and calling pmap(Fun, List, [node() | nodes()]) — in this example, if the list of nodes is not empty, the current node won’t be included in the call to spawn(). Obviously it’s not a very important difference — when I designed the example, I was just trying to illustrate how in Erlang, spawning processes on remote VM’s is as simple as spawning them on the local VM.

@wingedsubmariner I know about plists. It’s great.

By: wingedsubmariner

wingedsubmariner — Fri, 08 Feb 2008 23:31:39 +0000

I wrote a library to do this a while ago:

http://code.google.com/p/plists/

It will also parallelize many other operations. It supports proper error handling as well, trying to make itself as invisible as possible, while still not sending large lists across processes except when necessary (separate heaps == potential inefficiency). It’s capable of autodetecting the number of of threads Erlang is configured to use on each node and to spawn that many processes — effectively maxing out a machine.

It’s over one thousand lines of code though, so it fails succinctness.

By: Xichekolas

Xichekolas — Fri, 08 Feb 2008 22:55:50 +0000

@sjs: Ahhh, yeah. That makes total sense.

By: Bob Ippolito

Bob Ippolito — Fri, 08 Feb 2008 21:57:04 +0000

It seems most people are missing the fact that this will distribute the work over all nodes (usually nodes are run on different machines). The data parallel languages cited in the comments will do a great job of this too, almost certainly better than Erlang, but I believe that most of them will only do computation on the local machine.

The equivalent single-node pmap is for Erlang is short too, but less interesting (remove all but the last 3 statements, change SpawnFun to spawn, and remove Nodes from the argument list).

There isn’t really a different method for process creation on the local node versus any other node. It looks like SpawnFun is just handling a case that it really doesn’t need to (bad input). spawn(node(), F) would work just fine (node() is the local node). spawn(F) is just shorter to write, but it’s not necessary.

Realistically you’d probably call pmap(Fun, List, [node() | nodes()]) which would distribute the computation over all nodes that the local node can see, including itself node. He could’ve just as well checked for an empty Nodes and used [node()] instead.

As far as mailbox size goes, it depends on what you mean by “properly”. The answer is probably not — this isn’t an optimal pmap for very large lists due to that issue.

By: horia314

horia314 — Fri, 08 Feb 2008 20:25:59 +0000

Touche, one might say. On the other hand, just like Arc is trying to prove it’s the best Arc around, so is this post proving that Erlang is the best Erlang around.

Except for it, there isn’t any production ready programming language that can do the sort of massive parallelism that Erlang can provide.

Kudos to it for being able to do that, but monopolies are bad things in programming languages as they are in business.

One thing in special bothers me about Erlang : why is there a different method for node creation on the local machine than it is on another node? I mean, it’s clearly something the runtime can handle, and it’ll spare you some fair amount of work on the programming side.

By: David Mathers

David Mathers — Fri, 08 Feb 2008 20:11:58 +0000

update T1 set L2 = F2(L1)

The clustering and scaling is handled by the SQL implementation, not mixed in with the application logic.

By: Garrett

Garrett — Fri, 08 Feb 2008 19:55:02 +0000

pmap f [] = []
pmap f (x:xs) = y `par` (ys `pseq` (y : ys))
where y = f x
ys = pmap f xs