Multithreading for performance in shell scripts

Now that everyone and their grandmother have at least two cores, you can double the efficiency by distributing the workload. However, multithreading support in pure shell scripts is terrible, even though you often do things that can take a while, like encoding a bunch of chip tunes to ogg vorbis:

mkdir ogg
for file in *.mod
	xmp -d wav -o - "$file" | oggenc -q 3 -o "ogg/$file.ogg"

This is exactly the kind of operation that is conceptually trivial to parallelize, but not obvious to implement in a shell script. Sure, you could run them all in the background and wait for them, but that will give you a load average equal to the number of files. Not fun when there are hundreds of files.

You can run two (or however many) in the background, wait and then start two more, but that’ll give terrible performance when the jobs aren’t of roughly equal length, since at the end, the longest running job will be blocking the other eager cores.

Instead of listing ways that won’t work, I’ll get to the point: GNU (and FreeBSD) xargs has a -P for specifying the number of jobs to run in parallel!

Let’s rewrite that conversion loop to parallelize

mod2ogg() { 
	for arg; do xmp -d wav -o - "$arg" | oggenc -q 3 -o "ogg/$arg.ogg" -; done
export -f mod2ogg
find . -name '*.mod' -print0 | xargs -0 -n 1 -P 2 bash -c 'mod2ogg "$@"' -- 

And if we already had a mod2ogg script, similar to the function just defined, it would have been simpler:

find . -name '*.mod' -print0 | xargs -0 -n 1 -P 2 mod2ogg

Voila. Twice as fast, and you can just increase the -P with fancier hardware.

I also added -n 1 to xargs here, to ensure an even distribution of work. If the work units are so small that executing the command starts becoming a sizable portion of it, you can increase it to make xargs run mod2ogg with more files at a time (which is why it’s a loop in the example).

7 thoughts on “Multithreading for performance in shell scripts”

  1. This blog entry is intriguing to me. I don’t understand some of your script though, for example the xmp command. I get a command not found when I try to run it on a linux box. Can you explain?

  2. @ John
    xmp is a .mod music player. It’s not related to the parallelization, it’s just the task being parallelized.

  3. I agree with John. I also face the same problem, when i try to run this i get a error report. So please try to rectify this. I am waiting for your reply.
    Just checked out this Link to gets some idea of Linux Administrators Guide.

  4. @ Ricky
    xmp can be installed through your distro’s package manager (for Debian, apt-get install xmp). oggenc comes from the package vorbis-tools.

  5. For even better workload distribution, there is GNU Parallel. Parallel is like xargs, but geared specifically towards parallelization across cores and even hosts.

  6. For those who only want to convert from FLAC to Ogg, you can remove the xmp part from the script.

    flac2ogg() {
    for arg; do oggenc -q 9 “$arg” -o “$arg.ogg”; done
    export -f flac2ogg
    find . -name ‘*.flac’ -print0 | xargs -0 -n 1 -P 4 bash -c ‘flac2ogg “$@”‘ –[/code]

    In my tests with the -n parameter, “-n 1” is the fastest at 20 to 22 seconds compared to 1 minute and 11 seconds when executing one encoder at a time. I specified 4 for -P parameter as I have a quad core processor.

Leave a Reply