{"id":746,"date":"2019-03-27T03:07:33","date_gmt":"2019-03-27T03:07:33","guid":{"rendered":"http:\/\/www.vidarholen.net\/contents\/blog\/?p=746"},"modified":"2019-03-27T03:12:05","modified_gmt":"2019-03-27T03:12:05","slug":"a-shell-script-that-deleted-a-database-and-how-shellcheck-could-have-helped","status":"publish","type":"post","link":"https:\/\/www.vidarholen.net\/contents\/blog\/?p=746","title":{"rendered":"A shell script that deleted a database, and how ShellCheck could have helped"},"content":{"rendered":"

Summary: We examine a real world case of how an innocent shell scripting mistake caused the deletion of a production database, and how ShellCheck<\/a> (a GPLv3 shell script linting and analysis tool) would have pointed out the errors and prevented the disaster.<\/p>\n

Disclosure: I am the ShellCheck author.<\/p>\n

The event<\/h3>\n

Here is the sad case, taken from a recent StackOverflow post:<\/p>\n

My developer committed a huge mistake and we cannot find our mongo database anyone in the server. Rescue please!!!<\/p>\n

He logged into the server, and saved the following shell under ~\/crontab\/mongod_back.sh<\/code>:<\/p>\n

#!\/bin\/sh\nDUMP=mongodump\nOUT_DIR=\/data\/backup\/mongod\/tmp     \/\/ \u5907\u4efd\u6587\u4ef6\u4e34\u65f6\u76ee\u5f55\nTAR_DIR=\/data\/backup\/mongod         \/\/ \u5907\u4efd\u6587\u4ef6\u6b63\u5f0f\u76ee\u5f55\nDATE=`date +%Y_%m_%d_%H_%M_%S`      \/\/ \u5907\u4efd\u6587\u4ef6\u5c06\u4ee5\u5907\u4efd\u65f6\u95f4\u4fdd\u5b58\nDB_USER=Guitang                     \/\/ \u6570\u636e\u5e93\u64cd\u4f5c\u5458\nDB_PASS=qq____________              \/\/ \u6570\u636e\u5e93\u64cd\u4f5c\u5458\u5bc6\u7801\nDAYS=14                             \/\/ \u4fdd\u7559\u6700\u65b014\u592d\u7684\u5907\u4efd\nTAR_BAK=\"mongod_bak_$DATE.tar.gz\"   \/\/ \u5907\u4efd\u6587\u4ef6\u547d\u540d\u683c\u5f0f\ncd $OUT_DIR                         \/\/ \u521b\u5efa\u6587\u4ef6\u5939\nrm -rf $OUT_DIR\/*                   \/\/ \u6e05\u7a7a\u4e34\u65f6\u76ee\u5f55\nmkdir -p $OUT_DIR\/$DATE             \/\/ \u521b\u5efa\u672c\u6b21\u5907\u4efd\u6587\u4ef6\u5939\n$DUMP -d wecard -u $DB_USER -p $DB_PASS -o $OUT_DIR\/$DATE  \/\/ \u6267\u884c\u5907\u4efd\u547d\u4ee4\ntar -zcvf $TAR_DIR\/$TAR_BAK $OUT_DIR\/$DATE       \/\/ \u5c06\u5907\u4efd\u6587\u4ef6\u6253\u5305\u653e\u5165\u6b63\u5f0f\u76ee\nfind $TAR_DIR\/ -mtime +%DAYS -delete             \/\/ \u5220\u966414\u5929\u524d\u7684\u65e7\u5907\u6d32<\/code><\/pre>\n
And then he run .\/mongod_back.sh<\/code>, then there were lots of permission denied, then he did Ctrl+C. Then the server shut down automatically.<\/p>\n
He then contacted AliCloud, the engineer connected the disk to another working server, so that he could check the disk. Then, he realized that some folders have gone, including \/data\/<\/code> where the mongodb is!!!<\/p>\n
PS: he did not take snapshot of the disk before.<\/p><\/blockquote>\n
Essentially, it\u2019s every engineer\u2019s nightmare.<\/p>\n
The post-mortem of this issue is an interesting puzzle that requires only basic shell scripting knowledge. If you\u2019d like to give it a try, now\u2019s the time. If you\u2019d like some hints, here\u2019s shellcheck\u2019s output<\/a> for the script.<\/p>\n
The rest of this post details about what happened, and how ShellCheck could have averted the disaster.<\/p>\n
What went wrong?<\/h3>\n
The MCVE<\/a> for how to ruin your week is this:<\/p>\n
#!\/bin\/sh\nDIR=\/data\/tmp    \/\/ The directory to delete\nrm -rf $DIR\/*    \/\/ Now delete it<\/code><\/pre>\n
The fatal error here is that \/\/<\/code> is not a comment in shell scripts. It\u2019s a path to the root directory, equivalent to \/<\/code>.<\/p>\n
On some platforms, the rm<\/code> line would have been fatal by itself, because it\u2019d boil down to rm -rf \/<\/code> with a few other arguments. Implementation these days often don\u2019t allow this though. The disaster in question happened on Ubuntu, whose GNU rm<\/code> would have refused:<\/p>\n
$ rm -rf \/\/\nrm: it is dangerous to operate recursively on '\/\/' (same as '\/')\nrm: use --no-preserve-root to override this failsafe<\/code><\/pre>\n
This is where the assignment comes in.<\/p>\n
The shell treats variable assignments and commands as two sides of the same coin. Here\u2019s the description from POSIX<\/a>:<\/p>\n
A \u201csimple command\u201d is a sequence of optional variable assignments and redirections, in any sequence, optionally followed by words and redirections, terminated by a control operator.<\/p><\/blockquote>\n
(A \u201csimple command\u201d is in contrast to a \u201ccompound\u201d command, which are structures like if<\/code> statements and for<\/code> loops that contain one or more simple or compound commands.)<\/p>\n
This means that var=42<\/code> and echo \"Hello\"<\/code> are both simple commands. The former has one optional assignment and zero optional words. The latter has zero optional assignments and two optional words.<\/p>\n
It also implies that a single simple command can contain both: var=42 echo \"Hello\"<\/code><\/p>\n
To make a long spec short, assignments in a simple command will apply only to the invoked command name. If there is no command name, they apply to the current shell. This latter explains var=42<\/code> by itself, but when would you use the former?<\/p>\n
It\u2019s useful when you want to set a variable for a single command without affecting your the rest of your shell:<\/p>\n
$ echo \"$PAGER\"  # Show current pager\nless\n\n$ PAGER=\"head -n 5\" man ascii\nASCII(7)       Linux Programmer's Manual      ASCII(7)\n\nNAME\n       ascii  -  ASCII character set encoded in octal,\n       decimal, and hexadecimal\n\n$ echo \"$PAGER\"  # Current pager hasn't changed\nless<\/code><\/pre>\n
This is exactly what happened unintentionally in the fatal assignment. Just like how the previous example scoped PAGER<\/code> to man<\/code> only, this one scoped DIR<\/code> to \/\/<\/code>:<\/p>\n
$ DIR=\/data\/tmp    \/\/ The directory to delete\nbash: \/\/: Is a directory\n\n$ echo \"$DIR\"  # The variable is unset\n(no output)<\/code><\/pre>\n
This meant that rm -rf $DIR\/*<\/code> became rm -rf \/*<\/code>, and therefore bypassed the check that was is in place for rm -rf \/<\/code><\/p>\n
(Why can\u2019t or won\u2019t rm<\/code> simply refuse to delete \/*<\/code> too? Because it never sees \/*<\/code>: the shell expands it first, so rm<\/code> sees \/bin \/boot \/dev \/data ...<\/code>. While rm<\/code> could obviously refuse to remove first level directories as well, this starts getting in the way of legitimate usage \u2013 a big sin in the Unix philosophy)<\/p>\n
How ShellCheck could have helped<\/h3>\n
Here\u2019s the output from this minimized snippet (see online<\/a>):<\/p>\n
$ shellcheck myscript\n\nIn myscript line 2:\nDIR=\/data\/tmp    \/\/ The directory to delete\n                 ^-- SC1127: Was this intended as a comment? Use # in sh.\n\n\nIn myscript line 3:\nrm -rf $DIR\/*    \/\/ Now delete it\n       ^----^ SC2115: Use \"${var:?}\" to ensure this never expands to \/* .\n       ^--^ SC2086: Double quote to prevent globbing and word splitting.\n                 ^-- SC2114: Warning: deletes a system directory.<\/code><\/pre>\n
Two issues have already been discussed, and would have averted this disaster:<\/p>\n