Vidar’s Blog

If you have a new installation, you’re probably using SHA512-based passwords instead of the older MD5-based passwords described in detail in the previous post, which I’ll assume you’ve read. sha512-crypt is very similar to md5-crypt, but with some interesting differences.

Since the implementation of sha512 is really less interesting than the comparison with md5-crypt, I’ll describe it by striking out the relevant parts of the md5-crypt description and writing in what sha512-crypt does instead.

Like md5-crypt, it can be divided into three phases. Initialization, loop, and finalization.

1. Generate a simple md5 sha512 hash based on the salt and password
2. Loop 1000 5000 times, calculating a new sha512 hash based on the previous hash concatenated with alternatingly the hash of the password and the salt. Additionally, sha512-crypt allows you to specify a custom number of rounds, from 1000 to 999999999
3. Use a special base64 encoding on the final hash to create the password hash string

The main differences are the higher number of rounds, which can be user selected for better (or worse) security, the use of the hashed password and salt in each round, rather than the unhashed ones, and a few tweaks of the initialization step.

Here’s the real sha512-crypt initialization.

1. Let “password” be the user’s ascii password, “salt” the ascii salt (truncated to 8 16 chars) , and “magic” the string “$1$”
2. Start by computing the Alternate sum, sha512(password + salt + password)
3. Compute the Intermediate₀ sum by hashing the concatenation of the following strings:
   1. Password
   2. Magic
   3. Salt
   4. length(password) bytes of the Alternate sum, repeated as necessary
   5. For each bit in length(password), from low to high and stopping after the most significant set bit
      • If the bit is set, append a NUL byte the Alternate sum
      • If it’s unset, append the first byte of the password
4. New: Let S_factor be 16 + the first byte of Intermediate₀
5. New: Compute the S bytes, length(salt) bytes of sha512(salt, concatenated S_factor times).
6. New: Compute the P bytes, length(password) bytes of sha512(password), repeated as necessary

Step 3.5 — which was very strange in md5-crypt — now makes a little more sense. We also calculated the S bytes and P bytes, which from here on will be used just like salt and password was in md5-crypt.

From this point on, the calculations will only involve the password P bytes, salt S bytes, and the Intermediate₀ sum. Now we loop 5000 times (by default), to stretch the algorithm.

• For i = 0 to 4999 (inclusive), compute Intermediate_i+1 by concatenating and hashing the following:
  1. If i is even, Intermediate_i
  2. If i is odd, password P bytes
  3. If i is not divisible by 3, salt S bytes
  4. If i is not divisible by 7, password P bytes
  5. If i is even, password P bytes
  6. If i is odd, Intermediate_i

  At this point you don’t need Intermediate_i anymore.

You will now have ended up with Intermediate₅₀₀₀. Let’s call this the Final sum. Since sha512 is 512bit, this is 64 bytes long.

The bytes will be rearranged, and then encoded as 86 ascii characters using the same base64 encoding as md5-crypt.

1. Output the magic, “$6$”
2. New: If using a custom number of rounds, output “rounds=12345$”
3. Output the salt
4. Output a “$” to separate the salt from the encrypted section
5. Pick out the 64 bytes in this order: 63 62 20 41 40 61 19 18 39 60 59 17 38 37 58 16 15 36 57 56 14 35 34 55 13 12 33 54 53 11 32 31 52 10 9 30 51 50 8 29 28 49 7 6 27 48 47 5 26 25 46 4 3 24 45 44 2 23 22 43 1 0 21 42
   • For each group of 6 bits (there’s 86 groups), starting with the least significant
     • Output the corresponding base64 character with this index

And yes, I do have a shell script for this as well: sha512crypt. This one takes about a minute to generate a hash, due to the higher number of rounds. However, it doesn’t support custom rounds.

I hope these two posts have provided an interesting look at two exceedingly common, but often overlooked, algorithms!

Advanced Linux-related things, Linux, Security hashing, md5, password, sha512

If you haven’t reinstalled recently, chances are you’re using MD5-based passwords. However, the password hashes you find in /etc/shadow look nothing like what md5sum returns.

Here’s an example:

/etc/shadow:
$1$J7iYSKio$aEY4anysz.gtXxg7XlL6v1

md5sum:
7c6483ddcd99eb112c060ecbe0543e86

What’s the difference in generating these hashes? Why are they different at all?

Just running md5sum on a password and storing that is just marginally more secure than storing the plaintext password.

Thanks to GPGPUs, a modern gaming rig can easily try 5 billion such passwords per second, or go over the entire 8-character alphanumeric space in a day. With rainbow tables, a beautiful time–space tradeoff, you can do pretty much the same in 15 minutes.

MD5-crypt employs salting to make precomputational attacks exponentially more difficult. Additionally, it uses stretching to make brute force attacks harder (but just linearly so).

As an aside, these techniques were used in the original crypt from 1979, so there’s really no excuse to do naive password hashing anymore. However, at that time the salt was 12 bits and the number of rounds 25 — quite adorable in comparison with today’s absolute minimum of 64 bits and 1000 rounds.

The original crypt was DES based, but used a modified algorithm to prevent people from using existing DES cracking hardware. MD5-crypt doesn’t do any such tricks, and can be implemented in terms of any MD5 library, or even the md5sum util.

As regular reads might suspect, I’ve written a shell script to demonstrate this: md5crypt. There are a lot of workarounds for Bash’s inability to handle NUL bytes in strings. It takes 10 seconds to generate a hash, and is generally awful..ly funny!

Let’s first disect a crypt hash. man 3 crypt has some details.

If salt is a character string starting with the characters
"$id$" followed by a string terminated by "$":

       $id$salt$encrypted

then instead of using the DES machine, id  identifies  the
encryption  method  used  and this then determines how the
rest of the password string is interpreted.  The following
values of id are supported:

       ID  | Method
       -------------------------------------------------
       1   | MD5
       2a  | Blowfish (on some Linux distributions)
       5   | SHA-256 (since glibc 2.7)
       6   | SHA-512 (since glibc 2.7)

Simple and easy. Split by $, and then your fields are Algorithm, Salt and Hash.

md5-crypt is a function that takes a plaintext password and a salt, and generate such a hash.

To set a password, you’d generate a random salt, input the user’s password, and write the hash to /etc/shadow. To check a password, you’d read the hash from /etc/shadow, extract the salt, run the algorithm on this salt and the candidate password, and then see if the resulting hash matches what you have.

md5-crypt can be divided into three phases. Initialization, loop, and finalization. Here’s a very high level description of what we’ll go through in detail:

1. Generate a simple md5 hash based on the salt and password
2. Loop 1000 times, calculating a new md5 hash based on the previous hash concatenated with alternatingly the password and the salt.
3. Use a special base64 encoding on the final hash to create the password hash string

Put like this, it relatively elegant. However, there are a lot of details that turn this from elegant to eyerolling.

Here’s the real initialization.

1. Let “password” be the user’s ascii password, “salt” the ascii salt (truncated to 8 chars), and “magic” the string “$1$”
2. Start by computing the Alternate sum, md5(password + salt + password)
3. Compute the Intermediate₀ sum by hashing the concatenation of the following strings:
   1. Password
   2. Magic
   3. Salt
   4. length(password) bytes of the Alternate sum, repeated as necessary
   5. For each bit in length(password), from low to high and stopping after the most significant set bit
      • If the bit is set, append a NUL byte
      • If it’s unset, append the first byte of the password

I know what you’re thinking, and yes, it’s very arbitrary. The latter part was most likely a bug in the original implementation, carried along as UNIX issues often are. Remember to stay tuned next week, when we’ll compare this to SHA512-crypt as used on new installations!

From this point on, the calculations will only involve the password, salt, and Intermediate₀ sum. Now we loop 1000 times, to stretch the algorithm.

• For i = 0 to 999 (inclusive), compute Intermediate_i+1 by concatenating and hashing the following:
  1. If i is even, Intermediate_i
  2. If i is odd, password
  3. If i is not divisible by 3, salt
  4. If i is not divisible by 7, password
  5. If i is even, password
  6. If i is odd, Intermediate_i

  At this point you don’t need Intermediate_i anymore.

You will now have ended up with Intermediate₁₀₀₀. Let’s call this the Final sum. Since MD5 is 128bit, this is 16 bytes long.

The bytes will be rearranged, and then encoded as 22 ascii characters with a special base64-type encoding. This is not the same as regular base64:

Normal base64 set:
ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

Crypt base64 set:
./0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz

Additionally, there is no padding. The leftover byte will be encoded into 2 base64 ascii characters.

1. Output the magic
2. Output the salt
3. Output a “$” to separate the salt from the encrypted section
4. Pick out the 16 bytes in this order: 11 4 10 5 3 9 15 2 8 14 1 7 13 0 6 12.
   • For each group of 6 bits (there are 22 groups), starting with the least significant
     • Output the corresponding base64 character with this index

Congratulations, you now have a compatible md5-crypt hash!

As you can see, it’s quite far removed from a naive md5(password) attempt.

Fortunately, one will only ever need this algorithm for compatibility. New applications can use the standard PBKDF2 algorithm, implemented by most cryptography libraries, which does the same thing only in a standardized and parameterized way.

As if this wasn’t bad enough, the next post next week will be more of the same, but with SHA512-crypt!

Advanced Linux-related things, Linux, Security hashing, md5, password

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

How do I simulate pressing Ctrl-C when running this:
while true; do echo sleeping; sleep 30; done

Are you thinking “SIGINT, duh!”? Hold your horses!

I tried kill -INT pid, but it doesn't work the same:

Ctrl-C    kills the sleep and the loop
SIGINTing the shell does nothing
SIGINTing sleep makes the loop continue with the next iteration

HOWEVER, if I run the script in the background and kill -INT %1
instead of kill -INT pid, THEN it works :O

Why does Ctrl-C terminate the loop, while SIGINT doesn’t?

Additionally, if I run the same loop with ping or top instead of sleep,
Ctrl-C doesn't terminate the loop either!

Yeah. Well… Yeah…

This behaviour is due to an often overlooked feature in UNIX: process groups. These are important for getting terminals and shells to work the way they do.

A process group is exactly what it sounds like: a group of processes. They have a leader, which is the process that created it using setsid(2). The leader’s pid is also the process group id. Child processes are in the same group as their parent by default.

Terminals keep track of the foreground process group (set by the shell using tcsetpgrp(3)). When receiving a Ctrl-C, they send the SIGINT to the entire foreground group. This means that all members of the group will receive SIGINT, not just the immediate process.

kill -INT %1 sends the signal to the job’s process group, not the backgrounded pid! This explains why it works like Ctrl-C.

You can do the same thing with kill -INT -pgrpid. Since the process group id is the same as the process group leader, you can kill the group by killing the pid with a minus in front.

But why do you have to kill both?

When the shell is interrupted, it will wait for the running command to exit. If this child’s status indicates it exited abnormally due to that signal, the shell cleans up, removes its signal handler, and kills itself again to trigger the OS default action (abnormal exit). Alternatively, it runs the script’s signal handler as set with trap, and continues.

If the shell is interrupted and the child’s status says it exited normally, then Bash assumes the child handled the signal and did something useful, so it continues executing. Ping and top both trap SIGINT and exit normally, which is why Ctrl-C doesn’t kill the loop when calling them.

This also explains why interrupting just the shell does nothing: the child exits normally, so the shell thinks the child handled the signal, though in reality it was never received.

Finally, if the shell isn’t interrupted and a child exits, Bash just carries on regardless of whether the signal died abnormally or not. This is why interrupting the sleep just continues with the loop.

In case one would like to handle such cases, Bash sets the exit code to 128+signal when the process exits abnormally, so interrupting sleep with SIGINT would give the exit code 130 (kill -l lists the signal values).

Bonus problem:

I have this C app, testpg:
int main() {
    setsid();
    return sleep(10);
}

I run bash -c './testpg' and press Ctrl-C. The app is killed.
Shouldn’t testpg be excluded from SIGINT, since it used setsid?

A quick strace unravels this mystery: with a single command to execute, bash execve’s it directly — a little optimization trick. Since the pid is the same and already had its own process group, creating a new one doesn’t have any effect.

Using a compound command prevents this: bash -c 'true; ./testpg' can’t be killed with Ctrl-C.

Advanced Linux-related things, Linux bash, shell script, signal, unix, why-bash-is-like-that

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

Why can't bash scripts be SUID?

Bash scripts can’t run with the suid bit set. First of all, Linux doesn’t allow any scripts to be setuid, though some other OS do. Second, bash will detect being run as setuid, and immediately drop the privileges.

This is because shell script security is extremely dependent on the environment, much more so than regular C apps.

Take this script, for example, addmaildomain:

#!/bin/sh
[[ $1 ]] || { man -P cat $0; exit 1; } 

if grep -q “^$(whoami)\$” /etc/accesslist
then
    echo “$1″ > /etc/mail/local-host-names
else
    echo “You don’t have permissions to add hostnames”
fi

The intention is to allow users in /etc/accesslist to run addmaildomain example.com to write new names to local-host-names, the file which defines which domains sendmail should accept mail for.

Let’s imagine it runs as suid root. What can we do to abuse it?

We can start by setting the path:

echo "rm -rf /" > ~/hax/grep && chmod a+x ~/hax/grep
PATH=~/hax addmaildomain

Now the script will run our grep instead of the system grep, and we have full root access.

Let’s assume the author was aware of that, had set PATH=/bin:/usr/bin as the first line in the script. What can we do now?

We can override a library used by grep

gcc -shared -o libc.so.6 myEvilLib.c
LD_LIBRARY_PATH=. addmaildomain

When grep is invoked, it’ll link with our library and run our evil code.

Ok, so let’s say LD_LIBRARY_PATH is closed up.

If the shell is statically linked, we can set LD_TRACE_LOADED_OBJECTS=true. This will cause dynamically linked executables to print out a list of library dependencies and return true. This would cause our grep to always return true, subverting the test. The rest is builtin and wouldn’t be affected.

Even if the shell is statically compiled, all variables starting with LD_* will typically be stripped by the kernel for suid executables anyways.

There is a delay between the kernel starting the interpretter, and the interpretter opening the file. We can try to race it:

while true
do
    ln /usr/bin/addmaildomain foo
    nice -n 20 ./foo &
    echo 'rm -rf /' > foo
done

But let’s assume the OS uses a /dev/fd/* mechanism for passing a fd, instead of passing the file name.

We can rename the script to confuse the interpretter:

ln /usr/bin/addmaildomain ./-i
PATH=.
-i

Now we’ve created a link, which retains suid, and named it “-i”. When running it, the interpretter will run as “/bin/sh -i”, giving us an interactive shell.

So let’s assume we actually had “#!/bin/sh –” to prevent the script from being interpretted as an option.

If we don’t know how to use the command, it helpfully lists info from the man page for us. We can compile a C app that executes the script with “$0″ containing “-P /hax/evil ls”, and then man will execute our evil program instead of cat.

So let’s say “$0″ is quoted. We can still set MANOPT=-H/hax/evil.

Several of these attacks were based on the inclusion of ‘man’. Is this a particularly vulnerable command?

Perhaps a bit, but a whole lot of apps can be affected by the environment in more and less dramatic ways.

• POSIXLY_CORRECT can make some apps fail or change their output
• LANG/LC_ALL can thwart interpretation of output
• LC_CTYPE and LC_COLLATE can modify string comparisons
• Some apps rely on HOME and USER
• Various runtimes have their own paths, like RUBY_PATH, LUA_PATH and PYTHONPATH
• Many utilities have variables for adding default options, like RUBYOPT and MANOPT
• Tools invoke EDITOR, VISUAL and PAGER under various circumstances

So yes, it’s better not to write suid shell scripts. Sudo is better than you at running commands safely.

Do remember that a script can invoke itself with sudo, if need be, for a simulated suid feel.

So wait, can’t perl scripts be suid?

They can indeed, but there the interpretter will run as the normal user and detect that the file is suid. It will then run a suid interpretter to deal with it.

Advanced Linux-related things, Security bash, shell script, why-bash-is-like-that

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

# Why don't the options in "man time" work?
time -f %w myapp

Short answer: ‘time’ runs the builtin version, ‘man time’ shows the external version

time is a builtin in the shell, as well as an external command (this also goes for kill, pwd, and test). The man time shows info about the external command, while help time shows the internal one.

To run the external version, one can use command time or /usr/bin/time or just \time.

The reason why time is built in is so timing pipelines will work properly. time true | sleep 10 would say 0 seconds with an external command (which can’t know what it’s being piped into), and while the internal version can say 10 seconds since it knows about the whole pipeline.

POSIX leaves the behaviour of time a | b undefined.

# This finds the full path to ls. Why isn't there a 'man type'?
type -P ls

type is a bash builtin, not an external command. This allows it to take shell functions and aliases into account, something whereis can’t.

Builtins are documented in man bash, or more conveniently, “help type” (help is also a builtin).

Basic Linux-related things, Uncategorized bash, shell script, why-bash-is-like-that

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

# Why can't you use variables in {}?
for i in {0..$n}; do ..

Short answer: {..} is expanded before $n

Shell execution is based on successive expansions. {..} is evaluated early, before variable expansion, and thus you can’t use variables in them.

This also implies that you can use {..} in variable names, like a1=foo; a2=bar; echo $a{1,2}.

Instead, use for ((i=0; i<n; i++)); do ....

# Why aren't any of the Linux rename tools like Windows 'ren'?
touch foo0001.jpg foo0002.jpg
ren foo*.jpg bar*.jpg        # Windows
rename foo bar foo*.jpg      # Coreutils rename
rename 's/foo/bar/' foo*.jpg # Perl (debian) rename

Short answer: globs are expanded before the command sees them

Bash expands globs before running the command. This means that running rename foo*.jpg bar*.jpg is exactly the same as running rename foo0000.jpg foo0001.jpg .... Since rename can’t know what pattern was originally used, it has to use an alternative syntax.

Of course, you could write a rename where you quote the globs, like rename "foo*.jpg" "bar*.jpg", but that’s not simpler than the coreutils version. It just adds complexity, edge cases and general confusion.

There have been proposals for environment variables to set so that commands can see the shell arguments with globs intact, but that has its own problems so they weren’t widely used.

Basic Linux-related things, Uncategorized bash, shell script, why-bash-is-like-that

Due to recent misuse of frozen goods, the RIAA has joined industry leaders in creating a new initiative: iScream!

Enjoy iScream wherever you are; at home, in your yard, and one other location of your choice (subject to restrictions, location must be preregistered, and can only be changed once per year).

It’s the perfect treat for a warm summer’s day, or at any celebration around the year, provided that no more than three people are present. iScream is only licensed for consumer use as a refreshing, frozen novelty food, and must therefore be enjoyed at an ambient temperature of at least 80F (26C).

iScream comes in three delicious flavours: vanilla, strawberry and spinach (vanilla and strawberry may be restricted in some areas, notably the Americas, Europe and Asia). Flavours may not be combined, and must be consumed within 24 hours of purchase.

The RIAA respects your individuality, and lets you make your iScream your own with the choice of one of two toppings, peanuts or walnuts. One of these approved toppings are required for consuming iScream, so individuals with nut allergies are adviced to select the one to which they respond the least.

In the past, RIAA has been critizised for not listening to consumers, and failing to keep up with modern trends in frozen treats. Therefore, you may now create an iScream Float by combining your iScream with a licensed and compatible bottle of natural spring water.

The RIAA is confident that iScream covers all reasonable consumer scenarios in a simple and convenient way. There is now a legal and respectable alternative to unlicensed frozen treat consumption, and the RIAA is working with law enforcement to allow violators to be shot on sight. We look forward to your continued cooperation!

Frequently asked questions about iScream:

- Can I use iScream as a comfort food after a relationship breakup?
- The iScream is not licensed for use as a comfort food. You can instead celebrate your new-found independence by checking “Celebration” in part 7B of your application form.

- Can I use iScream in my car?
- You sure can! Simply set your custom enjoyment location to a parking space at a rest stop you frequent. Whenever you want to enjoy iScream in your car, simply park in this spot and enjoy. Note that your custom enjoyment location must be a stationary position, otherwise people could select the car itself, and simply drive into other people’s homes to enjoy iScream there, in violation of the agreement (section 14, page 73).

- In my area, summers often don’t reach over 80F (26C). Can I still enjoy iScream?
- Of course! Simply set your indoor thermostat to 80F, and enjoy all the iScream you want in the comfort of your own home!

Random ideas

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

# Why does esc contain "x1B" instead of the escape char?
esc=`printf \\x1B`

Short answer: “ requires another level of backslash escaping

To embed a backtick inside “, you escape it with a backslash. To embed a backslash, you escape it with another backslash. So `printf \\x1B` actually runs printf \x1B, and the shell interprets \x as a literal x with a superfluous escape. In other words, printf just sees “x1B”, and that’s what you get.

The problem grows exponentially as you try to nest ` `.

$(..) has distinct start and stop characters, so they can be used and nested without adding layers of backslashes. In this particular case you’d use esc=$'\x1B', and in general you could use esc=$(printf \\x1B).

# Why is newline empty instead of containing the line feed character?
newline=$(printf "\n")
echo "hello${newline}world"

Short answer: “ and $(..) strips trailing line feeds

$(..) and “ always strip trailing line feeds from command output. This is that special kind of magic that works so well you never think about it. echo “Hello $(whoami), how are you?” comes out as one line even though “whoami” (and basically all other commands) writes the username followed by a line feed.

This causes problems here because the output is only a single \n, i.e. the empty string followed by a trailing line feed. In this case, you’d again use newline=$'\n', but you could also have done newline=$(printf '\n'; printf x); newline=${newline#x} (append a x and then remove it), so that the line feeds in the output aren’t trailing.

# Bonus: Exercise for the reader
# Try adding the appropriate quotes and
# escapes to this quote normalization example
normalized=`echo ``Go on'', he said | sed -e s/`/'/g`

The variable should contain ''Go on'', he said. Can you get it right on the first try?

Basic Linux-related things bash, shell script, why-bash-is-like-that

Bash can seem pretty random and weird at times, but most of what people see as quirks have very logical (if not very good) explanations behind them. This series of posts looks at some of them.

 # Why doesn't || work in [ ] ?
 if [ -f /etc/inetd.conf || -d /etc/xinetd.d ]; then .. 

# And why does it work in [[ ]] ?
 if [[ -f /etc/inetd.conf || -d /etc/xinetd.d ]]; then ..

Short answer: [ is a regular command, and can't override ||. [[ is shell syntax.

[ is a pseudo-syntactical command. That is, it's a regular command much like cp or grep, the name just happens to be a single opening square bracket (try ls -l /usr/bin/[).

Seeing as how it's a regular command, it can't affect shell grammar. In grep -q kittens file || echo kittens >> file, grep can't know or do anything about the fact that it's being used with "||". The same goes for [ in the example.

In Bash (but not necessarily other shells), [ is now a builtin command emulating /usr/bin/[ for efficiency. There's no reason why Bash couldn't make [ a || b ] work, but this would break compatibility.

Instead, we have [[ which is not bound by legacy, and does in fact alter shell syntax. [[ interprets ||, &&, globs and unquoted variable expansions in ways that an external [-command couldn't, and an internal [ therefore can't either.

To get around it without using [[, we'd do [ a ] || [ b ] or [ a -o b ].

# Why is this always true (should be false when foo is empty)?
if [ -n $foo ]

Short answer: $foo disappears and [ -n ] is shorthand for [ -n "-n" ], which is true

This comes back to the fact that [ is a regular command. If $foo is empty, the shell runs [ -n ], and there’s no way for [ to know that there was a variable that was expanded out.

[ -n x ] checks that x is not empty. [ x ] is shorthand for the same. [ -n ] therefore checks if dash-n is empty, which it isn’t, and thus the expression is always true.

Why doesn’t [ complain if you have -n without a parameter? Same thing again. It wouldn't be able to tell that you said [ $1 ] rather than [ -n ], and then scripts would break whenever you work with your parameter list or any other strings starting with a dash.

Instead, you’d use [ -n "$foo" ], which will prevent bash from removing foo when it’s empty, and instead send in a zero-length argument. Or you could use [[ -n $foo ]], since [[ is built into the shell grammar and can tell that there is a variable there.

# Why doesn't this work?
if [ grep -q text myfile ]

Short answer: [ is a command, grep is a command. Choose one.

if [ -n "$foo" ] is the same as if test -n "$foo", which makes it more obvious how to use other commands (if grep -q text myfile).

Pseudo-syntactical elements makes it harder to tell language from implementation for better and for worse. It makes code look neat, but can be confusing. Especially when basically all if-statements include [.

A similar effect is seen in here-documents (cmd << EOF), where many people only ever see the delimiter “EOF” and think it’s some kind of keyword. Next time, use cmd << KITTENS to tip off anyone reading your script.

Basic Linux-related things bash, shell script, why-bash-is-like-that

You probably have your own closely guarded ssh key pair. Chances are good that it’s based on RSA, the default choice in ssh-keygen.

RSA is a very simple and quite brilliant algorithm, and this article will show what a SSH RSA key pair contains, and how you can use those values to play around with and encrypt values using nothing but a calculator.

RSA is based on primes, and the difficulty of factoring large numbers. This post is not meant as an intro to RSA, but here’s a quick reminder. I’ll use mostly the same symbols as Wikipedia: you generate two large primes, p and q. Let φ = (p-1)(q-1). Pick a number e coprime to φ, and let d ≡ e^-1 mod φ.

The public key is then (e, n), while your private key is (d, n). To encrypt a number/message m, let the ciphertext c ≡ m^e mod n. Then m ≡ c^d mod n.

This is very simple modular arithmetic, but when you generate a key pair with ssh-keygen, you instead get a set of opaque and scary looking files, id_rsa and id_rsa.pub. Here’s a bit from the private key id_rsa (no passphrase):


-----BEGIN RSA PRIVATE KEY-----
MIIBygIBAAJhANj3rl3FhzmOloVCXXesVPs1Wa++fIBX7BCZ5t4lmMh36KGzkQmn
jDJcm+O9nYhoPx6Bf+a9yz0HfzbfA5OpqQAyC/vRTVDgHhGXY6HFP/lyWQ8DRzCh
tsuP6eq9RYHnxwIBIwJhAKdf+4oqqiUWOZn//vXrV3/19LrGJYeU

...
-----END RSA PRIVATE KEY-----


How can we get our nice RSA parameters from this mess?

The easy way is with openssl: (I apologize in advance for all the data spam in the rest of the article).


vidar@vidarholen ~/.ssh $ openssl rsa -text -noout < id_rsa
Private-Key: (768 bit)
modulus:
00:d8:f7:ae:5d:c5:87:39:8e:96:85:42:5d:77:ac:
54:fb:35:59:af:be:7c:80:57:ec:10:99:e6:de:25:
...
publicExponent: 35 (0x23)
privateExponent:
00:a7:5f:fb:8a:2a:aa:25:16:39:99:ff:fe:f5:eb:
57:7f:f5:f4:ba:c6:25:87:94:48:64:93:fb:3d:a7:
...
prime1:
...
prime2:
...
exponent1:
...
exponent2:
...
coefficient:
...


Here, modulus is n, publicExponent is e, privateExponent is d, prime1 is p, prime2 is q, exponent1 is d_P from the Wikipedia article, exponent2 is d_Q and coefficient is q_Inv.

Only the first three are strictly required to perform encryption and decryption. The latter three are for optimization and the primes are for verification.

It’s interesting to note that even though the private key from RSA’s point of view is (d,n), the OpenSSH private key file includes e, p, q and the rest as well. This is how it can generate public keys given the private ones. Otherwise, finding e given (d,n) is just as hard as finding d given (e,n), except e is conventionally chosen to be small and easy to guess for efficiency purposes.

If we have one of these hex strings on one line, without colons, and in uppercase, then bc can work on them and optionally convert to decimal.


# If you don't want to do this yourself, see end for a script
vidar@vidarholen ~/.ssh $ { echo 'ibase=16'; cat | tr -d ':\n ' | tr a-f A-F; echo; } | bc

00:d8:f7:ae:5d:c5:87:39:8e:96:85:42:5d:77:ac:
54:fb:35:59:af:be:7c:80:57:ec:10:99:e6:de:25:
98:c8:77:e8:a1:b3:91:09:a7:8c:32:5c:9b:e3:bd:
….
Ctrl-d to end input
13158045936463264355006370413708684112837853704660293756254884673628\
63292…


We also need a power-modulo function, since b^e % m is unfeasibly slow if you go by way of b^e. Luckily, bc is programmable.


vidar@vidarholen ~/.ssh $ bc
bc 1.06.94
Copyright 1991-1994, 1997, 1998, 2000, 2004, 2006 Free Software Foundation, Inc.
This is free software with ABSOLUTELY NO WARRANTY.
For details type `warranty'.

# Our powermod function:
define pmod(b,e,m) { if(e == 0 ) return 1; if(e == 1) return b%m; rest=pmod(b^2%m,e/2,m); if((e%2) == 1) return (b*rest)%m else return rest; }


#Define some variables (this time unabbreviated)
n=13158045936463264355006370413708684112837853704660293756254884673628\
63292777770859554071108633728590995985653161363101078779505801640963\
48597350763180843221886116453606059623113097963206649790257715468881\
4303031148479239044926138311

e=35

d=10150492579557375359576342890575270601332058572166512326253768176799\
23111571423234513140569517447770196903218153051479115016036905320557\
80231250287900874055062921398102953416891810163858645414303785372309\
5688315939617076008144563059

# Encrypt the number 12345
c=pmod(12345, e, n)

# Show the encrypted number
c
15928992191730477535088375321366468550579140816267293144554503305092\
03492035891240033089011563910196180080894311697511846432462334632873\
53515625


#Decrypt the number
pmod(c, d, n)

12345


Yay, we’ve successfully encrypted and decrypted a value using real life RSA parameters!

What’s in the public key file, then?

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAGEA2PeuXcWHOY6WhUJdd6xU+zVZr758gFfsEJnm3iWYyHfoobORCaeMMlyb472diGg/HoF/5r3LPQd/Nt8Dk6mpADIL+9FNUOAeEZdjocU/+XJZDwNHMKG2y4/p6r1FgefH vidar@vidarholen.spam

This is a very simple file format, but I don’t know of any tools that will decode it. Simply base64-decode the middle string, and then read 4 bytes of length, followed by that many bytes of data. Repeat three times. You will then have key type, e and n, respectively.

Mine is 00 00 00 07, followed by 7 bytes “ssh-rsa”. Then 00 00 00 01, followed by one byte of 0×23 (35, our e). Finally, 00 00 00 61 followed by 0×61 = 97 bytes of our modulus n.

If you want to decode the private key by hand, base64-decode the middle bit. This gives you an ASN.1 encoded sequence of integers.

This is an annotated hex dump of parts of a base64-decoded private key

30 82 01 ca   - Sequence, 0x01CA bytes
    02 01: Integer, 1 byte
        00
    02 61:    - Integer, 0x61 bytes (n).
        00 d8 f7 ae 5d c5 87 39 8e 96 ... Same as from openssl!
    02 01:  - Integer, 1 byte, 0x23=35 (e)
        23
    02 61  - Integer, 0x61 bytes (d)
        00 a7 5f fb 8a 2a aa 25 16 39 ...
    ...

Here’s a bash script that will decode a private key and output variable definitions and functions for bc, so that you can play around with it without having to do the copy-paste work yourself. It decodes ASN.1, and only requires OpenSSL if the key has a passphrase.

When run, and its output pasted into bc, you will have the variables n, e, d, p, q and a few more, functions encrypt(m) and decrypt(c), plus a verify() that will return 1 if the key is valid. These functions are very simple and transparent.

Enjoy!

Advanced Linux-related things, Linux, Security encryption, RSA, ssh