Applications often behave differently in subtle ways when stdout is not a terminal. Most of the time, this is done so smoothly that the user isn’t even aware of it.<\/p>\n
When it works like magic<\/strong><\/p>\n Consider Now, say we want a list of projects in our ~\/src dir, ignoring version numbers:<\/p>\n Piece of cake, right?<\/p>\n But think about the magic that actually happened there: We started out with three lines of coloured text, ran it through sed to search&replace on each line, and ended up with nine lines of uncoloured text.<\/p>\n How did sed filter the colours? How did it put each filename a separate line, when the same does not happen for The answer, of course, is that it didn’t. When outputting to a terminal, you can be fairly sure that the user will be reading it directly, so you can make it as pretty and unparsable as you want. When output is not a terminal, it’s likely going to some program or file where pretty output will just complicate things.<\/p>\n Life without magic<\/strong><\/p>\n Try the previous example with The directories obviously exist, but the ANSI escape codes that give them that cute colour also prevents utilities from working with them. For additional fun, copy-pasting this error message from a terminal strips the colours, so anyone you reported it to would be quite stumped.<\/p>\n Magic efficiency tricks<\/strong><\/p>\n It’s not all about making output pretty or parsable depending on the situation. Read\/write syscalls are notoriously expensive; reading anything less than about 4k bytes at a time will make disk reads CPU bound.<\/p>\n glibc knows this, and will alter write buffering depending on the context. If the output is a terminal, a user is probably watching and waiting for it, so it will flush output immediately. If it’s a file, it’s better to buffer it up for efficiency:<\/p>\n In other words, grep wrote about god 3948 times (insert your own bible forum jokes).<\/p>\n This time, grep produced the exact same output, but wrote to a file instead. This resulted in 64 writes \u2013 about 1% of the more interactive mode!<\/p>\n Spells of confusion<\/strong><\/p>\n Sometimes magic can confuse and astound. What if output is kinda like a terminal, only not?<\/p>\n Here’s a much more confusing example (just go along with the simplified grep):<\/p>\n Basic stuff.<\/p>\n While the logic is the same as before, our realtime error log doesn’t show anything!<\/p>\n Why? Because grep’s output isn’t a terminal, so it will buffer up about 4k worth of data before writing it all in one go. In the mean time, the command will just seem to hang for no apparent reason!<\/p>\n (Observant readers might ask, “Isn’t tail buffering?”. And it might be or it might not. It depends on your version and distro patches.)<\/p>\n Mastering magic<\/strong><\/p>\n Ok, so what can we do to take charge of these useful peculiarities?<\/p>\n Many apps have flags for this, though none of them are POSIX.<\/p>\n GNU If your app doesn’t have such an option, there’s always Obviously, this depends on the app using standard C stdio, or that it checks for a terminal itself. Apps can unintentionally be written to avoid this, like when setting Java’s System.Out to a BufferedOutputStream.<\/p>\n And finally… how can you create such behaviour yourself?<\/p>\n Applications often behave differently in subtle ways when stdout is not a terminal. Most of the time, this is done so smoothly that the user isn’t even aware of it. When it works like magic Consider ls: vidar@vidarholen ~\/src $ ls PyYAML-3.09 bsd-games-2.17 nltk-2.0b9 alsa-lib-1.0.23 libsamplerate-0.1.7 pulseaudio-0.9.21 bash-4.0 linux tmp bitlbee-1.2.8 linux-2.6.32.8 vidar@vidarholen ~\/src $ … Continue reading “Is it terminal?”<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":"","jetpack_publicize_message":"","jetpack_is_tweetstorm":false,"jetpack_publicize_feature_enabled":true},"categories":[5,4],"tags":[29,53,21],"class_list":["post-22","post","type-post","status-publish","format-standard","hentry","category-advanced-linux","category-linux","tag-buffering","tag-linux","tag-shell-script"],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=\/wp\/v2\/posts\/22","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=22"}],"version-history":[{"count":0,"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=\/wp\/v2\/posts\/22\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=22"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=22"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.vidarholen.net\/contents\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=22"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}ls<\/code>:<\/p>\nvidar@vidarholen ~\/src $ ls\r\nPyYAML-3.09 bsd-games-2.17 nltk-2.0b9\r\nalsa-lib-1.0.23 libsamplerate-0.1.7 pulseaudio-0.9.21\r\nbash-4.0 <\/span>linux<\/span> tmp\r\nbitlbee-1.2.8 linux-2.6.32.8<\/span>\r\nvidar@vidarholen ~\/src $<\/pre>\n# For novelty purposes only; parsing ls is a bad idea\r\nvidar@vidarholen ~\/src $ ls | sed -n 's\/-[^-]*$\/\/p'\r\nPyYAML\r\nalsa-lib\r\nbash\r\nbitlbee\r\nbsd-games\r\nlibsamplerate\r\nlinux\r\nnltk\r\npulseaudio\r\nvidar@vidarholen ~\/src $<\/pre>\n
echo \"foo bar\" | sed ..<\/code>?<\/p>\nls<\/code> detected that output wasn’t a terminal and altered its output accordingly.<\/p>\nls -C --color=always<\/code> instead of just ls<\/code>, and see how different life would have been without this terminal detection. You can also try this with xargs, to see how colours could break things:<\/p>\nvidar@vidarholen ~\/src $ ls -C --color=always | xargs ls -ld\r\nls: cannot access PyYAML-3.09<\/span>: No such file or directory\r\nls: cannot access alsa-lib-1.0.23<\/span>: No such file or directory\r\n...<\/pre>\n
\nvidar@kelvin ~ $ strace -e write -o log grep God text\/bible12.txt
\n01:001:001 In the beginning God created the heaven and the earth.
\n...
\nvidar@kelvin ~ $ wc -l log
\n3948 log
\n<\/code><\/p>\n
\nvidar@kelvin ~ $ strace -e write -o log grep God text\/bible12.txt > tmp
\nvidar@kelvin ~ $ wc -l log
\n64 log
\n<\/code><\/p>\nls -l<\/code> gives the user pretty colours. ls -l | more<\/code> does not. The reason is not at all obvious for users who just consider ” | more” a way to scroll in output. But it works, even if it’s not as pretty as we’d like.<\/p>\n# Show apache traffic (works)\r\ncat access.log\r\n\r\n# Show 404 errors with line numbers (works)\r\ncat access.log | grep 404 | nl<\/pre>\n
# Show apache traffic in realtime (works)\r\ntail -f access.log\r\n\r\n# Show 404 errors with line numbers in realtime (FAILS<\/strong>)\r\ntail -f access.log | grep 404 | nl<\/pre>\n
ls<\/code> lets you specify -C<\/code> for columned mode, and --color=always<\/code> for colours, regardless of the nature of stdout.<\/p>\nsed<\/code> has -u<\/code>, grep<\/code> has a --line-buffered<\/code>. awk<\/code> has a fflush<\/code> function. tail<\/code>, if yours buffers at all, has a -u<\/code> since about 2008 which as of now isn’t in debian stable.<\/p>\nunbuffer<\/a><\/code> from Expect<\/a>, the interactive tool scripting package.<\/p>\nunbuffer<\/code> starts applications within its own pseudo-tty, much like how xterm and sshd does it. This usually tricks the application into not buffering (and maybe to prettify its output).<\/p>\nif [[ -t 1 ]] #if stdout is a terminal\r\nthen\r\n tput setaf 3 #Set foreground to yellow\r\nfi\r\necho \"Pure gold\"<\/pre>\n","protected":false},"excerpt":{"rendered":"