Shell pipes and redirection: stdin, stdout and stderr

The power of the command line lies not in individual programs but in the ability to connect them. Each command does one thing, and the shell lets you hand one result to the next as on an assembly line. The mechanism that makes it possible is simple and worth understanding thoroughly: three communication channels and a handful of operators.

The three standard streams

Every process starts with three open channels, identified by a number:

The fact that results and errors travel on separate channels is the key to everything: you can save the results to a file and still see the errors, or the other way round.

Redirecting output to a file

• command > file.txt writes the output to the file, replacing its content if it already exists.
• command >> file.txt appends the output to the end of the file.
• command 2> errors.txt saves only the error messages.
• command > all.txt 2>&1 sends both output and errors to the same file.

The last form confuses many people: 2>&1 means «send channel 2 wherever channel 1 is going». Order matters: it must come after the output redirection, otherwise errors still end up on the screen. Bash also offers the shortcut &> all.txt.

A classic mistake is sort data.txt > data.txt: the shell empties the destination file before sort starts reading it, and the result is an empty file. Write to a different file and then rename it.

Reading input from a file

The < operator makes the command read a file's content as if it were typed: wc -l < list.txt counts the lines. There is also the here document, handy in scripts to pass several lines of text:

cat <<END > greeting.txt
First line
Second line
END

/dev/null: the output bin

/dev/null accepts any data and discards it. It is useful when a command produces messages you do not care about:

find / -name "*.conf" 2>/dev/null

Here a search across the whole disk would produce dozens of «Permission denied» lines; sending them to /dev/null leaves only the useful results. Use it with judgement: hiding all of a script's errors means not noticing when something goes wrong.

The pipe: connecting two commands

The | operator connects one command's stdout to the next command's stdin. Data flows directly from one to the other, with no intermediate files.

ps aux | grep nginx

ps lists the processes, grep keeps only the lines containing «nginx». Pipes can be chained as far as you like, and this is where the command line shows its power.

A concrete example: who is knocking on the server

Suppose you want to know which IP addresses made the most requests to a web server, by reading its access log:

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -10

Step by step:

1. cut -d' ' -f1 extracts the first field of each line, that is the IP address;
2. sort orders the addresses, so identical ones become adjacent;
3. uniq -c merges identical consecutive lines and counts them;
4. sort -rn sorts numerically in descending order;
5. head -10 shows the top ten.

Five simple programs, each around for decades, produce in a second a statistic that would otherwise require a script. In my work this kind of chain is often the first tool for getting a sense of what is happening on a system.

Pipes and stderr

The pipe carries only stdout. The first command's errors keep appearing on the screen and are not filtered by the second. If you need them in the chain too, use 2>&1 | or, in Bash, |&.

tee: save and see at the same time

tee reads stdin, writes it to a file and passes it on. It is useful when you want to keep the output of a long operation without giving up watching it:

sudo apt upgrade | tee upgrade.log

With tee -a the file is extended instead of overwritten. A less obvious use concerns permissions: echo text > /etc/file with sudo in front fails, because the redirection is performed by the normal user's shell. The correct form is echo text | sudo tee /etc/file.

Commands in sequence

Besides the pipe there are operators that control the order of execution:

• a ; b runs b after a, whatever the outcome;
• a && b runs b only if a succeeded;
• a || b runs b only if a failed.

mkdir backup && cp data.db backup/ copies the file only if the folder was actually created. Getting used to && instead of ; stops an early error from spreading into later operations.

In short

• Every process has stdin, stdout and stderr: results and errors travel separately.
• > overwrites, >> appends, 2> redirects errors, 2>&1 merges them with the output.
• The pipe | passes one command's output to the next and enables quick analysis of logs and text.
• tee saves and shows at once; with sudo tee you can write to system files.
• Prefer && to ; when a command depends on the previous one succeeding.