cat > $'/bin/bash\r' << "EOF"
#!/usr/bin/env bash
script=$1
shift
exec bash <(tr -d '\r' < "$script") "$@"
EOF


This allows you to execute scripts with DOS/Windows \r\n line endings with ./yourscript (but it will fail if the script specifies parameters on the shebang, or if you run it with bash yourscript). It works because from a UNIX point of view, DOS/Windows files specify the interpretter as "bash^M", and we override that to clean the script and run bash on the result.

Of course, you can also replace the helpful exec bash part with echo "Run dos2unix on your file!" >&2 if you'd rather give your users a helpful reminder rather than compatibility or a crazy error.

• find -exec fails on commands that are perfectly valid
• 0==1 is apparently true
• Comparisons are always false, and write files while failing
• Variable values are available inside loops, but reset afterwards
• Looping over filenames with spaces fails, and quoting doesn’t help

ShellCheck is my latest project. It will check shell scripts for all of the above, and also tries to give helpful tips and suggestions for otherwise working ones. You can paste your script and have it checked it online, or you can downloaded it and run it locally.

Other things it checks for includes reading from and redirecting to a file in the same pipeline, useless uses of cat, apparent variable use that won’t expand, too much or too little quoting in [[ ]], not quoting globs passed to find, and instead of just saying “syntax error near unexpected token `fi’”, it points to the relevant if statement and suggests that you might be missing a ‘then’.

It’s still in the early stages, but has now reached the point where it can be useful. The online version has a feedback button (in the top right of your annotated script), so feel free to try it out and submit suggestions!

If you have other suggestions for scenarios, tools or approaches, leave a commment. Otherwise, I’ll wish you a speedy recovery!

# I run this script, but afterwards my PATH and current dir hasn't changed!

#!/bin/bash
export PATH=$PATH:/opt/local/bin
cd /opt/games/

or more interestingly

# Why does this always say 0?
n=0
cat file | while read line; do (( n++ )); done
echo $n

In the first case, you can add a echo "Path is now $PATH", and see the expected path. In the latter case, you can put a echo $n in the loop, and it will count up as you’d expect, but at the end you’ll still be left with 0.

To make things even more interesting, here are the effects of running these two examples (or equivalents) in different shells:

What we’re experiencing are subshells, and different shells have different policies on what runs in subshells.

Environment variables, as well as the current directory, is only inherited parent-to-child. Changes to a child’s environment are not reflect in the parent. Any time a shell forks, changes done in the forked process are confined to that process and its children.

In Unix, all normal shells will fork to execute other shell scripts, so setting PATH or cd’ing in a script will never have an effect after the command is done (instead, use "source file" aka ". file" to read and execute the commands without forking).

However, shells can differ in when subshells are invoked. In Bash, all elements in a pipeline will run in a subshell. In Ksh and Zsh, all except the last will run in a subshell. POSIX leaves it undefined.

This means that echo "2 + 3" | bc | read sum will work in Ksh and Zsh, but fail to set the variable sum in Bash.

To work around this in Bash, you can usually use redirection and process substition instead:

read sum < <(echo "2 + 3" | bc)

So, where do we find subshells? Here are a list of commands that in some way fails to set foo=bar for subsequent commands (note that all the examples set it in some subshell, and can use it until the subshell ends):

# Executing other programs or scripts
./setmyfoo
foo=bar ./something

# Anywhere in a pipeline in Bash
true | foo=bar | true

# In any command that executes new shells
awk '{ system("foo=bar") }'h
find . -exec bash -c 'foo=bar' \;

# In backgrounded commands and coprocs:
foo=bar &
coproc foo=bar

# In command expansion
true "$(foo=bar)"

# In process substitution
true < <(foo=bar)

# In commands explicitly subshelled with ()
( foo=bar )

and probably some more that I'm forgetting.

Trying to set a variable, option or working dir in any of these contexts will result in the changes not being visible for following commands.

Knowing this, we can use it to our advantage:

# cd to each dir and run make
for dir in */; do ( cd "$dir" && make ); done

# Compare to the more fragile
for dir in */; do cd "$dir"; make; cd ..; done

# mess with important variables
fields=(a b c); ( IFS=':'; echo ${fields[*]})

# Compare to the cumbersome
fields=(a b c); oldIFS=$IFS; IFS=':'; echo ${fields[*]}; IFS=$oldIFS; 

# Limit scope of options
( set -e; foo; bar; baz; )

There, I said it.

I have five Linux devices in my home, I write Python in Vim on Ubuntu professionally, and I have rolled my eyes at every major shift in Windows since 3.11.

But I’m excited about Windows 8.

Before I ruin your day with my fanboy ramblings, let me clear up some reasons for why I don’t hate it (if you don’t either, feel free to skip towards the end).

“I don’t want a nerfed touch interface on my desktop!”

My actual Windows 8 desktop

Agreed. Fortunately, as you can see from this screenshot, a Windows 8 desktop session looks just like a Windows 7 one, with efficient, dense, touch hostile apps.

Believe me, I’ll be the first to scream bloody murder if I have to develop Windows software in a touch UI with only two apps on screen.

On that note, ARM based Windows 8 devices won’t support normal desktop sessions, so x86 is implied for the rest of the article. The reason for this should become apparent later.

“I want a start menu!”

Oh, that wonderful start menu

The start menu was awesome in 1998, when having 15 apps installed made you a power user. But now my entertainment PC (pictured) has 55 top level entries, and my development box has 92! In a tiny list with a huge scroll bar!

“But”, I hear you say, “that’s the full list! No uses that!”.

Quite right. The apps you love, you pin on the taskbar. This works exactly the same in Windows 8.

With your hands on the keyboard, you find by keyboard search. This works exactly the same in Windows 8.

This leaves the apps you use fairly often, which are either pinned to the start menu, or have bubbled their way up through frequent use.

Default 1080p start menu area

On a full HD display, the default is 20 items, with a maximum of 24. Shown in a linear list. The default uses literally less than a tenth of the available screen real estate (pictured).

On a smaller screen, you get 12-14 items, which is a very snug fit, and definitely not enough if you want shortcuts to web sites like I have on my Android and iOS devices.

So what do you do? Instead of having a short, narrow list on the left, we could lay them out in a grid (like desktop icons) to fit more of them.

And that pretty much describes the Windows 8 start screen!

Of course, you have tiles instead of icons, but I’ll get to those in a bit.

“Secure boot will block Linux and other OS!”

Maybe. I hope it doesn’t. I’m not saying it’s all cream and peaches, but hopefully it’ll continue to be optional on all devices.

Slimmer and faster!

There’s a running joke that both processing power and Windows boot time doubles every 18 months.

Windows 8, however, cold boots in 10 seconds on my low end, portable test hardware. It also uses about 30% less RAM than Windows 7.

No one is more surprised than me.

Tiles — Icons++

I no longer “check my email”, I get a notification when new mail is received. I no longer visit web sites regularly, I use RSS to tell me when there’s something new. Why should I still open my apps to see what’s happening?

Sure, my RSS reader could be an icon saying “RSS Reader”, but it would be more convenient if it said “Bitcoin market crashes, and 14 more”. Windows 8 does this.

My ebook reader could say “eBook reader”. Or it could say “Reading Dune, page 134 of 245″, with cover art instead of a generic icon. Windows 8 does this.

My video player could say “VLC”, or it could show a screenshot of where I stopped a video last night. Wait, *cough*, that’s a terrible idea. Windows 8 lets you disable this tile-by-tile!

A People app tile, plus two tiles representing two specific friends

One app can have multiple tiles. Just like how you can have two books on your desktop that open in the same reader, you can have two tiles. They’ll show the book title and page number separately.

For non-file based links, Win7 users can create shortcuts to invoke a program with different command line arguments (but no one does). With Windows 8, you can pin application specific contents, e.g. a Facebook contact, and have a tile representing them.

This tile will then show photos of that particular person (pictured), plus their recent activity and messages they send you. Great for close friends and people you’re dating (or stalking). When opened, you’ll go immediately to their page.

Likewise, you can have an icon for a specific Wordfeud game, showing current score and status. An often used playlist? A news source you follow closely? A specific alt in a mmorpg? Separate work and private email accounts? All available at a glance.

Is it worth an upgrade?

Honestly? While I think Windows 8 is an improvement in some ways, I probably won’t be upgrading my Windows 7 desktop/laptop systems any time soon.

However, (and here’s the twist), I will camp out a cold winter’s night to get a Microsoft Surface or similar Windows 8 tablet PC. Because you see, I consider Windows 8:

The hands down, best tablet PC OS to date

I don’t even care about Windows 8 vs GNU+Linux, I see Windows 8 tablets vs Windows desktops and iOS/Android tablets.

Why?

I hope I’ve convincingly argued that Windows 8 is not nerfed, but a complete desktop OS (“as far as Windows goes”, hurr durr).

Now, imagine that on a tablet:

Fully usable desktop — Bring it to the office, dock it with keyboard, mouse and external monitor. You can now use it as a regular PC running Visual Studio, Excel, and all that jazz!

Fully usable tablet — Bring it with you on the bus, and use it as a small, touch friendly browser and time waster, just like an iPad or Android tablet!

…in one device — Inspiration strikes? Switch back to your Visual Studio project and fix, test and commit while still on the bus.

Useful, new features — I’m sure tiles can come in handy on a desktop system, but on a tablet I think they’ll set a new standard for convenience.

No software limitations — Encounter something stupid, like not being able to run an app in the background, or not being able to search on a web page (looking at you, iPad!!)? You have a full desktop OS to fall back on!

Both toy and tool — It’s no longer just a fun but useless toy. Bittorrent a Ubuntu iso, checksum it, and write it out to a thumb drive? Sure thing!

Want to demo something? Want to script something? You got it.

I would ditch my iPad 2 in a second.

Seriously?

Seriously.

And yes, I think it will work in practice. You see, it already has:

I loved Maemo, the Debian based cell phone OS featuring xterm and a GNU toolset in the factory default. Like Windows 8, Maemo had a touch UI for normal operation, but also allowed you to go behind the scenes and run “proper” software.

While I obviously spent most of my time in the touch UI, Maemo was there for me when I needed more. Granted, not everyone feels they have to strip and encode the audio track from a youtube video on their phones, but the core point remains:

When someone with a modern, online, multi-gigabyte, multi-core tablet says “Oh darn, I can’t. I don’t have a computer with me”, something is tragically wrong.

And that’s why I can’t wait for Windows 8.

How do I write a regex that matches lines that do not contain hi?

That’s a frequently asked question if I ever saw one.

Of course, the proper answer is: you don’t. You write a regex that does match hi and then invert the matching logic, ostensibly with grep -v. But where’s the fun in that?

One interesting theorem that pops up in any book or class on formal grammars is that regular languages are closed under complement: the inverse of a regular expression is also a regular expression. This means that writing inverted regular expressions is theoretically possible, though it turns out to be quite tricky

Just try writing a regex that matches strings that does not contain “hi”, and test it against “hi”, “hhi” and “ih”, “iih” and such variations. Some solutions are coming up.

A way to cheat is using PCRE negative lookahead: ^(?!.*foo) matches all strings not containing the substring “foo”. However, lookahead assertions require a stack, and thus can’t be modelled as a finite state machine. In other words, they don’t fit the mathematical definition of a regular expression, and therefore disqualify.

There are simple, well-known algorithms for turning regular expressions into non-deterministic finite automata, and from there to deterministic FA. Less commonly used and known are algorithms for inverting a DFA and for generating familiar textual regex from it.

You can find these described in various lecture notes and slides, so I won’t recite them.

What I had a harder time finding was software that actually did this. So here is a Haskell program. It’s highly suboptimal but it does the job. When executed, it will ask for a regex and will then output a grep command that matches everything the regex does not (without -v, obviously).

The expressions it produces are quite horrific; it’s computer generated code, after all.

A regular expression for matching strings that do not match .*hi.* could be grep -E '^([^h]|h+$|h+[^hi])*$'.

This app, however, suggests grep -E '^([^h]([^h]|)*||([^h][^h]*h|h)|([^h][^h]*(h(hh*[^hi]|[^hi]))|(h(hh*[^hi]|[^hi])))((hh*[^hi]|[^h])|)*|([^h][^h]*(h([^hi][^h]*h|h))|(h([^hi][^h]*h|h)))(([^hi][^h]*h|h)|)*)$'

It still works exactly as stated though!

The app just supports a small subset of regex, just enough to convince someone that it works, and as a party trick lets you answer the original question exactly as stated.

Technically correct is the best kind of correct.

Since the implementation of sha512 is really less interesting than the comparison with md5-crypt, I’ll describe it by striking out the relevant parts of the md5-crypt description and writing in what sha512-crypt does instead.

Like md5-crypt, it can be divided into three phases. Initialization, loop, and finalization.

1. Generate a simple md5 sha512 hash based on the salt and password
2. Loop 1000 5000 times, calculating a new sha512 hash based on the previous hash concatenated with alternatingly the hash of the password and the salt. Additionally, sha512-crypt allows you to specify a custom number of rounds, from 1000 to 999999999
3. Use a special base64 encoding on the final hash to create the password hash string

The main differences are the higher number of rounds, which can be user selected for better (or worse) security, the use of the hashed password and salt in each round, rather than the unhashed ones, and a few tweaks of the initialization step.

Here’s the real sha512-crypt initialization.

1. Let “password” be the user’s ascii password, “salt” the ascii salt (truncated to 8 16 chars) , and “magic” the string “$1$”
2. Start by computing the Alternate sum, sha512(password + salt + password)
3. Compute the Intermediate₀ sum by hashing the concatenation of the following strings:
   1. Password
   2. Magic
   3. Salt
   4. length(password) bytes of the Alternate sum, repeated as necessary
   5. For each bit in length(password), from low to high and stopping after the most significant set bit
      • If the bit is set, append a NUL byte the Alternate sum
      • If it’s unset, append the first byte of the password
4. New: Let S_factor be 16 + the first byte of Intermediate₀
5. New: Compute the S bytes, length(salt) bytes of sha512(salt, concatenated S_factor times).
6. New: Compute the P bytes, length(password) bytes of sha512(password), repeated as necessary

Step 3.5 — which was very strange in md5-crypt — now makes a little more sense. We also calculated the S bytes and P bytes, which from here on will be used just like salt and password was in md5-crypt.

From this point on, the calculations will only involve the password P bytes, salt S bytes, and the Intermediate₀ sum. Now we loop 5000 times (by default), to stretch the algorithm.

• For i = 0 to 4999 (inclusive), compute Intermediate_i+1 by concatenating and hashing the following:
  1. If i is even, Intermediate_i
  2. If i is odd, password P bytes
  3. If i is not divisible by 3, salt S bytes
  4. If i is not divisible by 7, password P bytes
  5. If i is even, password P bytes
  6. If i is odd, Intermediate_i

  At this point you don’t need Intermediate_i anymore.

You will now have ended up with Intermediate₅₀₀₀. Let’s call this the Final sum. Since sha512 is 512bit, this is 64 bytes long.

The bytes will be rearranged, and then encoded as 86 ascii characters using the same base64 encoding as md5-crypt.

1. Output the magic, “$6$”
2. New: If using a custom number of rounds, output “rounds=12345$”
3. Output the salt
4. Output a “$” to separate the salt from the encrypted section
5. Pick out the 64 bytes in this order: 63 62 20 41 40 61 19 18 39 60 59 17 38 37 58 16 15 36 57 56 14 35 34 55 13 12 33 54 53 11 32 31 52 10 9 30 51 50 8 29 28 49 7 6 27 48 47 5 26 25 46 4 3 24 45 44 2 23 22 43 1 0 21 42
   • For each group of 6 bits (there’s 86 groups), starting with the least significant
     • Output the corresponding base64 character with this index

And yes, I do have a shell script for this as well: sha512crypt. This one takes about a minute to generate a hash, due to the higher number of rounds. However, it doesn’t support custom rounds.

I hope these two posts have provided an interesting look at two exceedingly common, but often overlooked, algorithms!

Here’s an example:

/etc/shadow:
$1$J7iYSKio$aEY4anysz.gtXxg7XlL6v1

md5sum:
7c6483ddcd99eb112c060ecbe0543e86

What’s the difference in generating these hashes? Why are they different at all?

Just running md5sum on a password and storing that is just marginally more secure than storing the plaintext password.

Thanks to GPGPUs, a modern gaming rig can easily try 5 billion such passwords per second, or go over the entire 8-character alphanumeric space in a day. With rainbow tables, a beautiful time–space tradeoff, you can do pretty much the same in 15 minutes.

MD5-crypt employs salting to make precomputational attacks exponentially more difficult. Additionally, it uses stretching to make brute force attacks harder (but just linearly so).

As an aside, these techniques were used in the original crypt from 1979, so there’s really no excuse to do naive password hashing anymore. However, at that time the salt was 12 bits and the number of rounds 25 — quite adorable in comparison with today’s absolute minimum of 64 bits and 1000 rounds.

The original crypt was DES based, but used a modified algorithm to prevent people from using existing DES cracking hardware. MD5-crypt doesn’t do any such tricks, and can be implemented in terms of any MD5 library, or even the md5sum util.

As regular reads might suspect, I’ve written a shell script to demonstrate this: md5crypt. There are a lot of workarounds for Bash’s inability to handle NUL bytes in strings. It takes 10 seconds to generate a hash, and is generally awful..ly funny!

Let’s first disect a crypt hash. man 3 crypt has some details.

If salt is a character string starting with the characters
"$id$" followed by a string terminated by "$":

       $id$salt$encrypted

then instead of using the DES machine, id  identifies  the
encryption  method  used  and this then determines how the
rest of the password string is interpreted.  The following
values of id are supported:

       ID  | Method
       -------------------------------------------------
       1   | MD5
       2a  | Blowfish (on some Linux distributions)
       5   | SHA-256 (since glibc 2.7)
       6   | SHA-512 (since glibc 2.7)

Simple and easy. Split by $, and then your fields are Algorithm, Salt and Hash.

md5-crypt is a function that takes a plaintext password and a salt, and generate such a hash.

To set a password, you’d generate a random salt, input the user’s password, and write the hash to /etc/shadow. To check a password, you’d read the hash from /etc/shadow, extract the salt, run the algorithm on this salt and the candidate password, and then see if the resulting hash matches what you have.

md5-crypt can be divided into three phases. Initialization, loop, and finalization. Here’s a very high level description of what we’ll go through in detail:

1. Generate a simple md5 hash based on the salt and password
2. Loop 1000 times, calculating a new md5 hash based on the previous hash concatenated with alternatingly the password and the salt.
3. Use a special base64 encoding on the final hash to create the password hash string

Put like this, it relatively elegant. However, there are a lot of details that turn this from elegant to eyerolling.

Here’s the real initialization.

1. Let “password” be the user’s ascii password, “salt” the ascii salt (truncated to 8 chars), and “magic” the string “$1$”
2. Start by computing the Alternate sum, md5(password + salt + password)
3. Compute the Intermediate₀ sum by hashing the concatenation of the following strings:
   1. Password
   2. Magic
   3. Salt
   4. length(password) bytes of the Alternate sum, repeated as necessary
   5. For each bit in length(password), from low to high and stopping after the most significant set bit
      • If the bit is set, append a NUL byte
      • If it’s unset, append the first byte of the password

I know what you’re thinking, and yes, it’s very arbitrary. The latter part was most likely a bug in the original implementation, carried along as UNIX issues often are. Remember to stay tuned next week, when we’ll compare this to SHA512-crypt as used on new installations!

From this point on, the calculations will only involve the password, salt, and Intermediate₀ sum. Now we loop 1000 times, to stretch the algorithm.

• For i = 0 to 999 (inclusive), compute Intermediate_i+1 by concatenating and hashing the following:
  1. If i is even, Intermediate_i
  2. If i is odd, password
  3. If i is not divisible by 3, salt
  4. If i is not divisible by 7, password
  5. If i is even, password
  6. If i is odd, Intermediate_i

  At this point you don’t need Intermediate_i anymore.

You will now have ended up with Intermediate₁₀₀₀. Let’s call this the Final sum. Since MD5 is 128bit, this is 16 bytes long.

The bytes will be rearranged, and then encoded as 22 ascii characters with a special base64-type encoding. This is not the same as regular base64:

Normal base64 set:
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

Crypt base64 set:
./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

Additionally, there is no padding. The leftover byte will be encoded into 2 base64 ascii characters.

1. Output the magic
2. Output the salt
3. Output a “$” to separate the salt from the encrypted section
4. Pick out the 16 bytes in this order: 11 4 10 5 3 9 15 2 8 14 1 7 13 0 6 12.
   • For each group of 6 bits (there are 22 groups), starting with the least significant
     • Output the corresponding base64 character with this index

Congratulations, you now have a compatible md5-crypt hash!

As you can see, it’s quite far removed from a naive md5(password) attempt.

Fortunately, one will only ever need this algorithm for compatibility. New applications can use the standard PBKDF2 algorithm, implemented by most cryptography libraries, which does the same thing only in a standardized and parameterized way.

As if this wasn’t bad enough, the next post next week will be more of the same, but with SHA512-crypt!

How do I simulate pressing Ctrl-C when running this:
while true; do echo sleeping; sleep 30; done

Are you thinking “SIGINT, duh!”? Hold your horses!

I tried kill -INT pid, but it doesn't work the same:

Ctrl-C    kills the sleep and the loop
SIGINTing the shell does nothing
SIGINTing sleep makes the loop continue with the next iteration

HOWEVER, if I run the script in the background and kill -INT %1
instead of kill -INT pid, THEN it works :O

Why does Ctrl-C terminate the loop, while SIGINT doesn’t?

Additionally, if I run the same loop with ping or top instead of sleep,
Ctrl-C doesn't terminate the loop either!

Yeah. Well… Yeah…

This behaviour is due to an often overlooked feature in UNIX: process groups. These are important for getting terminals and shells to work the way they do.

A process group is exactly what it sounds like: a group of processes. They have a leader, which is the process that created it using setsid(2). The leader’s pid is also the process group id. Child processes are in the same group as their parent by default.

Terminals keep track of the foreground process group (set by the shell using tcsetpgrp(3)). When receiving a Ctrl-C, they send the SIGINT to the entire foreground group. This means that all members of the group will receive SIGINT, not just the immediate process.

kill -INT %1 sends the signal to the job’s process group, not the backgrounded pid! This explains why it works like Ctrl-C.

You can do the same thing with kill -INT -pgrpid. Since the process group id is the same as the process group leader, you can kill the group by killing the pid with a minus in front.

But why do you have to kill both?

When the shell is interrupted, it will wait for the running command to exit. If this child’s status indicates it exited abnormally due to that signal, the shell cleans up, removes its signal handler, and kills itself again to trigger the OS default action (abnormal exit). Alternatively, it runs the script’s signal handler as set with trap, and continues.

If the shell is interrupted and the child’s status says it exited normally, then Bash assumes the child handled the signal and did something useful, so it continues executing. Ping and top both trap SIGINT and exit normally, which is why Ctrl-C doesn’t kill the loop when calling them.

This also explains why interrupting just the shell does nothing: the child exits normally, so the shell thinks the child handled the signal, though in reality it was never received.

Finally, if the shell isn’t interrupted and a child exits, Bash just carries on regardless of whether the signal died abnormally or not. This is why interrupting the sleep just continues with the loop.

In case one would like to handle such cases, Bash sets the exit code to 128+signal when the process exits abnormally, so interrupting sleep with SIGINT would give the exit code 130 (kill -l lists the signal values).

Bonus problem:

I have this C app, testpg:
int main() {
    setsid();
    return sleep(10);
}

I run bash -c './testpg' and press Ctrl-C. The app is killed.
Shouldn't testpg be excluded from SIGINT, since it used setsid?

A quick strace unravels this mystery: with a single command to execute, bash execve’s it directly — a little optimization trick. Since the pid is the same and already had its own process group, creating a new one doesn’t have any effect.

Using a compound command prevents this: bash -c 'true; ./testpg' can’t be killed with Ctrl-C.

Why can't bash scripts be SUID?

Bash scripts can’t run with the suid bit set. First of all, Linux doesn’t allow any scripts to be setuid, though some other OS do. Second, bash will detect being run as setuid, and immediately drop the privileges.

This is because shell script security is extremely dependent on the environment, much more so than regular C apps.

Take this script, for example, addmaildomain:

#!/bin/sh
[[ $1 ]] || { man -P cat $0; exit 1; } 

if grep -q "^$(whoami)\$" /etc/accesslist
then
    echo "$1" > /etc/mail/local-host-names
else
    echo "You don't have permissions to add hostnames"
fi

The intention is to allow users in /etc/accesslist to run addmaildomain example.com to write new names to local-host-names, the file which defines which domains sendmail should accept mail for.

Let’s imagine it runs as suid root. What can we do to abuse it?

We can start by setting the path:

echo "rm -rf /" > ~/hax/grep && chmod a+x ~/hax/grep
PATH=~/hax addmaildomain

Now the script will run our grep instead of the system grep, and we have full root access.

Let’s assume the author was aware of that, had set PATH=/bin:/usr/bin as the first line in the script. What can we do now?

We can override a library used by grep

gcc -shared -o libc.so.6 myEvilLib.c
LD_LIBRARY_PATH=. addmaildomain

When grep is invoked, it’ll link with our library and run our evil code.

Ok, so let’s say LD_LIBRARY_PATH is closed up.

If the shell is statically linked, we can set LD_TRACE_LOADED_OBJECTS=true. This will cause dynamically linked executables to print out a list of library dependencies and return true. This would cause our grep to always return true, subverting the test. The rest is builtin and wouldn’t be affected.

Even if the shell is statically compiled, all variables starting with LD_* will typically be stripped by the kernel for suid executables anyways.

There is a delay between the kernel starting the interpretter, and the interpretter opening the file. We can try to race it:

while true
do
    ln /usr/bin/addmaildomain foo
    nice -n 20 ./foo &
    echo 'rm -rf /' > foo
done

But let’s assume the OS uses a /dev/fd/* mechanism for passing a fd, instead of passing the file name.

We can rename the script to confuse the interpretter:

ln /usr/bin/addmaildomain ./-i
PATH=.
-i

Now we’ve created a link, which retains suid, and named it “-i”. When running it, the interpretter will run as “/bin/sh -i”, giving us an interactive shell.

So let’s assume we actually had “#!/bin/sh –” to prevent the script from being interpretted as an option.

If we don’t know how to use the command, it helpfully lists info from the man page for us. We can compile a C app that executes the script with “$0″ containing “-P /hax/evil ls”, and then man will execute our evil program instead of cat.

So let’s say “$0″ is quoted. We can still set MANOPT=-H/hax/evil.

Several of these attacks were based on the inclusion of ‘man’. Is this a particularly vulnerable command?

Perhaps a bit, but a whole lot of apps can be affected by the environment in more and less dramatic ways.

• POSIXLY_CORRECT can make some apps fail or change their output
• LANG/LC_ALL can thwart interpretation of output
• LC_CTYPE and LC_COLLATE can modify string comparisons
• Some apps rely on HOME and USER
• Various runtimes have their own paths, like RUBY_PATH, LUA_PATH and PYTHONPATH
• Many utilities have variables for adding default options, like RUBYOPT and MANOPT
• Tools invoke EDITOR, VISUAL and PAGER under various circumstances

So yes, it’s better not to write suid shell scripts. Sudo is better than you at running commands safely.

Do remember that a script can invoke itself with sudo, if need be, for a simulated suid feel.

So wait, can’t perl scripts be suid?

They can indeed, but there the interpretter will run as the normal user and detect that the file is suid. It will then run a suid interpretter to deal with it.