Why Bash is like that: Order of expansion

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

# Why can't you use variables in {}?
for i in {0..$n}; do ..

Short answer: {..} is expanded before $n

Shell execution is based on successive expansions. {..} is evaluated early, before variable expansion, and thus you can’t use variables in them.

This also implies that you can use {..} in variable names, like a1=foo; a2=bar; echo $a{1,2}.

Instead, use for ((i=0; i<n; i++)); do ....

# Why aren't any of the Linux rename tools like Windows 'ren'?
touch foo0001.jpg foo0002.jpg
ren foo*.jpg bar*.jpg        # Windows
rename foo bar foo*.jpg      # Coreutils rename
rename 's/foo/bar/' foo*.jpg # Perl (debian) rename

Short answer: globs are expanded before the command sees them

Bash expands globs before running the command. This means that running rename foo*.jpg bar*.jpg is exactly the same as running rename foo0000.jpg foo0001.jpg .... Since rename can’t know what pattern was originally used, it has to use an alternative syntax.

Of course, you could write a rename where you quote the globs, like rename "foo*.jpg" "bar*.jpg", but that’s not simpler than the coreutils version. It just adds complexity, edge cases and general confusion.

There have been proposals for environment variables to set so that commands can see the shell arguments with globs intact, but that has its own problems so they weren’t widely used.

Why Bash is like that: Command expansion

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

# Why does esc contain "x1B" instead of the escape char?
esc=`printf \\x1B`

Short answer: “ requires another level of backslash escaping

To embed a backtick inside “, you escape it with a backslash. To embed a backslash, you escape it with another backslash. So `printf \\x1B` actually runs printf \x1B, and the shell interprets \x as a literal x with a superfluous escape. In other words, printf just sees “x1B”, and that’s what you get.

The problem grows exponentially as you try to nest ` `.

$(..) has distinct start and stop characters, so they can be used and nested without adding layers of backslashes. In this particular case you’d use esc=$'\x1B', and in general you could use esc=$(printf \\x1B).

# Why is newline empty instead of containing the line feed character?
newline=$(printf "\n")
echo "hello${newline}world"

Short answer: “ and $(..) strips trailing line feeds

$(..) and “ always strip trailing line feeds from command output. This is that special kind of magic that works so well you never think about it. echo “Hello $(whoami), how are you?” comes out as one line even though “whoami” (and basically all other commands) writes the username followed by a line feed.

This causes problems here because the output is only a single \n, i.e. the empty string followed by a trailing line feed. In this case, you’d again use newline=$'\n', but you could also have done newline=$(printf '\n'; printf x); newline=${newline#x} (append a x and then remove it), so that the line feeds in the output aren’t trailing.

# Bonus: Exercise for the reader
# Try adding the appropriate quotes and
# escapes to this quote normalization example
normalized=`echo ``Go on'', he said | sed -e s/`/'/g`

The variable should contain ''Go on'', he said. Can you get it right on the first try?

Why Bash is like that: Pseudo-syntax

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

 # Why doesn't || work in [ ] ?
 if [ -f /etc/inetd.conf || -d /etc/xinetd.d ]; then .. 

# And why does it work in [[ ]] ?
 if [[ -f /etc/inetd.conf || -d /etc/xinetd.d ]]; then ..

Short answer: [ is a regular command, and can’t override ||. [[ is shell syntax.

[ is a pseudo-syntactical command. That is, it’s a regular command much like cp or grep, the name just happens to be a single opening square bracket (try ls -l /usr/bin/[).

Seeing as how it’s a regular command, it can’t affect shell grammar. In grep -q kittens file || echo kittens >> file, grep can’t know or do anything about the fact that it’s being used with “||”. The same goes for [ in the example.

In Bash (but not necessarily other shells), [ is now a builtin command emulating /usr/bin/[ for efficiency. There’s no reason why Bash couldn’t make [ a || b ] work, but this would break compatibility.

Instead, we have [[ which is not bound by legacy, and does in fact alter shell syntax. [[ interprets ||, &&, globs and unquoted variable expansions in ways that an external [-command couldn’t, and an internal [ therefore can’t either.

To get around it without using [[, we’d do [ a ] || [ b ] or [ a -o b ].

# Why is this always true (should be false when foo is empty)?
if [ -n $foo ]

Short answer: $foo disappears and [ -n ] is shorthand for [ -n “-n” ], which is true

This comes back to the fact that [ is a regular command. If $foo is empty, the shell runs [ -n ], and there’s no way for [ to know that there was a variable that was expanded out.

[ -n x ] checks that x is not empty. [ x ] is shorthand for the same. [ -n ] therefore checks if dash-n is empty, which it isn’t, and thus the expression is always true.

Why doesn’t [ complain if you have -n without a parameter? Same thing again. It wouldn’t be able to tell that you said [ $1 ] rather than [ -n ], and then scripts would break whenever you work with your parameter list or any other strings starting with a dash.

Instead, you’d use [ -n "$foo" ], which will prevent bash from removing foo when it’s empty, and instead send in a zero-length argument. Or you could use [[ -n $foo ]], since [[ is built into the shell grammar and can tell that there is a variable there.

# Why doesn't this work?
if [ grep -q text myfile ]

Short answer: [ is a command, grep is a command. Choose one.

if [ -n "$foo" ] is the same as if test -n "$foo", which makes it more obvious how to use other commands (if grep -q text myfile).

Pseudo-syntactical elements makes it harder to tell language from implementation for better and for worse. It makes code look neat, but can be confusing. Especially when basically all if-statements include [.

A similar effect is seen in here-documents (cmd << EOF), where many people only ever see the delimiter “EOF” and think it’s some kind of keyword. Next time, use cmd << KITTENS to tip off anyone reading your script.

Why PulseAudio?

PulseAudio has been on the receiving end of quite a bit of flaming. Some deserved, a lot not. Most critics have been appeased as PulseAudio has matured and distros make setup smoother, but there are still a number of troll^Wusers who trash it at every opportunity.

Common objections to Pulseaudio includes:

“We don’t need another sound system!”

First of all, that’s what we said when ALSA was introduced. It played out exactly the same way. I have IRC logs from 2004 where the conversation goes “My sound is gone. God damn ALSA!”. “Yeah, I’m starting to think ALSA is like the emperors new clothes. It never works, but people say it’s because you’re a noob.” Then two people immediately chime in saying “works fine here”. Replace ALSA with PulseAudio, and you could have seen it in any forum today.

Second, we definitely do need another sound system. ALSA works ok as an output mechanism, but it’s not a sound layer worthy of a modern operating system. Even when it came out, it was way behind Windows (but far ahead of OSS). In terms of features, ALSA is to PulseAudio what SVGALib is to X11.

“I just want to play sound, I don’t need all these crap features”

Maybe. This is why OSS worked for so long. 90% of use cases are covered by allowing a single app to play audio at any one time. However, there is a lot more to a smooth audio experience. The following things are a non-exhaustive list of convenient features that a layer like PulseAudio can provide:

  • On-the-fly output device switching: In “the old days”, you could plug in a headset and the hardware would disconnect the speakers and play through the headset only. Today, the headset might be Bluetooth or USB, so no such hacks are possible. PulseAudio lets you handle this in software!
  • Per-application volume control: The ability to turn up the music without also turning up the IM notification sounds. If you use MPlayer with ALSA for example, 9/0 adjusts the system-wide volume, not just MPlayer’s.
  • Automatic muting of other audio on incoming phone calls, such as through Skype
  • Audio forwarding: Like X11 forwarding? Me too. With PulseAudio, you can also get application audio over the network. pax11publish lets you set PulseAudio settings like network forwarding per-display!

 

And here are some features that are less day-to-day useful, but very cool:

  • Networked audio: You can transmit audio from one (or more) apps to another box. Load Spotify on your laptop and play it via the media center’s speakers!
  • Combine two stereo cards into one 4 channel device. The cards don’t even have to be on the same computer!
  • Use the monitor device to record application audio, or connect it to a visualizer (demonstrated in a previous post)!

 

“PulseAudio uses a lot more CPU time than ALSA”

PulseAudio uses a high quality resampling algorithm by default. ALSA supports only one, a simple linear resampler. If, like most people, you can’t hear a difference anyways, you can configure PulseAudio to use a linear algorithm too by adding “resample-method = src-linear” in /etc/pulse/daemon.conf. CPU usage will drop from 10% to 1%!

“Thanks, I love PulseAudio now”

No problem

What’s up with directory hard link counts?

Ever considered the hard link count from ls on directories?

 
vidar@kelvin ~/src $ ls -l
total 108
drwxr-xr-x  4 vidar vidar  4096 2009-11-22 12:52 aml-lsb
drwxr-xr-x 13 vidar vidar  4096 2009-12-13 16:00 delta3d_REL-2.4.0
drwxr-xr-x 23 vidar vidar  4096 2010-02-02 18:22 linux-2.6.32.7
...

For files, this is the number of hard links. You can use find / -samefile filename to find all files that point to the same file inode.

So what does this number mean for directories? Exactly the same thing.

Users, including root, are blocked from creating directory hard links out of the kernel’s mortal fear of cyclical directory trees (or should I say directory graphs?). The kernel still creates them though, specifically in the form of the “.” entry in the directory itself, and “..” in each subdirectory.

An empty directory /foo/bar will have two links, /foo/bar itself, and /foo/bar/.. When creating a subdirectory /foo/bar/baz, you will get the additional hard link /foo/bar/baz/... In other words, the hard link count is the number of subdirectories plus two.

Here’s a party trick for listing directory hard links in bash:

vidar@kelvin ~/src $ ls -ld aml-lsb/{,.,*/..}
drwxr-xr-x 4 vidar vidar 4096 2009-11-22 12:52 aml-lsb/
drwxr-xr-x 4 vidar vidar 4096 2009-11-22 12:52 aml-lsb/.
drwxr-xr-x 4 vidar vidar 4096 2009-11-22 12:52 aml-lsb/bin/..
drwxr-xr-x 4 vidar vidar 4096 2009-11-22 12:52 aml-lsb/lib/..
vidar@kelvin ~/src $ 

Clearly, each of them refers to the same thing, and the numbers add up (if they don’t, shopt -s dotglob)

As a side note, you can use mount --rbind to fake a directory hard link. This will remount a directory and all submounts on some other directory, but will prevent cycles.

You can also use mount --bind to remount without submounts. This can be useful for when you want to copy the contents of a directory that has another file system mounted over it. This is most commonly /dev, which is over-mounted with udev early in the boot process. Many people don’t realize that they have an entire /dev they’ve never seen!

Simple ad-hoc file sharing

There is a distinct lack of simple, ad-hoc file sharing mechanisms for one-off file transfers between platforms. Maintaining an ftp or http server securely and grant users access to files is cumbersome. An ssh guest account opens more than you’d like, and still requires you to somehow grant access to a certain file to a user and then close it. IRC requires that the file is on the box you run the client on (which is often not your local box), and MSN requires that you add people to your contact list, assuming you don’t use it through bitlbee anyways.

Here is a little script I have lying around, I call it wwwshare:

#!/bin/bash

die() { echo "$*" >&2; exit 1; }

[[ $# != 0 ]] || die "Usage: $0 filename"
[[ -f $1 ]]   || die "No such file: $1"

file="$1"
ip=$(curl -s 'http://checkip.dyndns.com/' | sed 's/.* \([0-9.]*\).*/\1/')
port=$((8000 + RANDOM%1000))

echo "http://$ip:$port/$file"

cat - "$file" << EOF | nc -l -p $port -q 0
HTTP/1.1 200 Ok
Content-Type: application/octet-stream
Content-Length: $(stat -c %s "$file")

EOF

Just run wwwshare filename, and it’ll print an URL and start a wannabe http server on a random port (8000-9000) for a single session. When the file is downloaded, it exits. No setup or cleanup required.