Comments on: Parallel merge sort in Erlang

By: Jim Dukhovny

Jim Dukhovny — Mon, 19 Oct 2009 05:07:18 +0000

Can you just take a “penalty” on store rather then on retrieve?
If you need to keep top 100 posts of friends sorted by date then…
For every user keep “Recent top 100″ posts and every time someone makes a posts you update all their friend’s “Recent top 100″ by putting it at #1 and kicking last one out.
I am a Front End guy:) so user’s perception of Facebook working fast will be much higher if retrieve will be fast rather then save…
Just a subjective thought :)

By: Chris Hagan

Chris Hagan — Sun, 26 Apr 2009 04:06:34 +0000

Would you mind going into a little more detail about timing spawned processes in Erlang? I’m trying at the moment to benchmark a heavily parallel server, but what I’m seeing is that timer:tc/4 seems to time the first process until it terminates, which isn’t necessarily all the work that needed to be executed – the spawned processes continue until they also terminate, but the stopwatch has stopped already.

In your map/reduce case, of course, you’ve got your parent receiving so it’s certain that the spawned processes are finished before the stopwatch stops, but that is a side effect of those spawned processes notifying their parent.

I don’t want to have to end every process throughout my system by signalling a termination just so that I can benchmark. What are your thoughts on a general practice for accurate timing of the interval until all work initiated by a method call is complete?

By: tmd

tmd — Thu, 19 Mar 2009 16:36:34 +0000

“When a user visits the site, you want to show her a list of all the recent updates from her friends, sorted by date.”

Is Facebook just for women?

By: Witek BARYLUK

Witek BARYLUK — Thu, 19 Mar 2009 12:05:14 +0000

You actually don’t need fully sorted list in this case, just N first elemenets, so you mergers can stop if they ciollect this first N elements. Actually this can be also optimalised in qsort (after selection do only right recursion if needed).

I’m also interested about more complex key used for comparission so it will be dominating (something like now() for date can be).

By: Ulf Wiger

Ulf Wiger — Sun, 15 Mar 2009 21:32:37 +0000

lists:sort/1 is not a BIF, actually. It’s a merge sort algorithm implemented completely in Erlang. The one “dirty trick” it uses is calling lists:reverse(L1, L2), which is the same as doing lists:reverse(L1) ++ L2, but much faster (this is not esp dirty, since the function is documented…)

Your quicksort most likely takes a beating since it uses append.

By: Michael S.

Michael S. — Sun, 15 Mar 2009 19:01:19 +0000

Switching sort technique once the lists get small isn’t really “cheating”–glibc’s qsort switches to insertion sort when the sublists get small. See

http://sourceware.org/cgi-bin/cvsweb.cgi/libc/stdlib/qsort.c?rev=1.12.2.1&content-type=text/x-cvsweb-markup&cvsroot=glibc

By: Alex Arnon

Alex Arnon — Thu, 12 Mar 2009 12:34:18 +0000

Shifting around all these lists must also incur a penalty. What would happen when we’re close to completion, and start sending lists of 100,000’s of elements?

Maybe each worker should actually keep its sorted list, and then directly forward it to another (arbitrarily chosen by the master) once its work queue is truly empty. It could then wait for either more work, or a signal to terminate.

By: Jebu Ittiachen

Jebu Ittiachen — Thu, 12 Mar 2009 05:48:41 +0000

Nothing significant but combining the collect with the merge shaves a second off the merge and does better than the quick sort now :) in tune with what Jake mentioned about losing parallelism in between merge and collect

% mmerge process takes two elements off the list spawns merge
% merge merges them and send it back to the master, this adds it back
% to the tail end of the list continue till list has one element and
% no mergers are left
mmerge_all([List|[]], 0) ->
List;
mmerge_all([L1, L2 | Tl], N) ->
Parent = self(),
spawn(
fun() ->
Res = merge(L1, L2),
Parent ! {list, Res}
end),
mmerge_all(Tl, N + 1);
mmerge_all(List, N) ->
L =
receive
{list, L1} ->
lists:append(List, [L1])
end,
mmerge_all(L, N – 1).

By: Matt Williamson

Matt Williamson — Wed, 11 Mar 2009 11:22:57 +0000

It would be nice to see a number much bigger than 1,000,000 run across multiple machines.

By: Jake Donham

Jake Donham — Tue, 10 Mar 2009 17:53:36 +0000

Seems like master / worker parallelism could be effective here. Instead of spawning a process for every pair of lists, keep a pool of processes, and have them request pairs of lists from the master then return the merged list. You really only need as many processes in the pool as you have processors.

Furthermore, in your code you proceed in phases (issue a bunch of merges, collect them all, repeat), so you lose some parallelism at the end of the phase when there are fewer active processes than processors. Instead, the master could return merged lists to the pool of lists needing merging (until there is just one list in the pool when you’re done) so it’s only at the end of the whole run that you have fewer tasks than processors.

As you discovered, the granularity of the tasks is important to balance parallelism against communication cost. There is a great but little-known book that you might be interested in on these topics: Carriero and Gelernter’s How to Write Parallel Programs (http://www.amazon.com/dp/026203171X).