stderr are three data streams created when you launch a Linux command. You can use them to tell if your scripts are being piped or redirected. We show you how.
Streams Join Two Points
As soon as you start to learn about Linux and Unix-like operating systems, you’ll come across the terms
stederr. These are three standard streams that are established when a Linux command is executed. In computing, a stream is something that can transfer data. In the case of these streams, that data is text.
Data streams, like water streams, have two ends. They have a source and an outflow. Whichever Linux command you’re using provides one end of each stream. The other end is determined by the shell that launched the command. That end will be connected to the terminal window, connected to a pipe, or redirected to a file or other command, according to the command line that launched the command.
The Linux Standard Streams
stdin is the standard input stream. This accepts text as its input. Text output from the command to the shell is delivered via the
stdout (standard out) stream. Error messages from the command are sent through the
stderr (standard error) stream.
So you can see that there are two output streams,
stderr, and one input stream,
stdin. Because error messages and normal output each have their own conduit to carry them to the terminal window, they can be handled independently of one another.
Streams Are Handled Like Files
Streams in Linux—like almost everything else—are treated as though they were files. You can read text from a file, and you can write text into a file. Both of these actions involve a stream of data. So the concept of handling a stream of data as a file isn’t that much of a stretch.
Each file associated with a process is allocated a unique number to identify it. This is known as the file descriptor. Whenever an action is required to be performed on a file, the file descriptor is used to identify the file.
These values are always used for
- 0: stdin
- 1: stdout
- 2: stderr
Reacting to Pipes and Redirects
To ease someone’s introduction to a subject, a common technique is to teach a simplified version of the topic. For example, with grammar, we are told that the rule is “I before E, except after C.” But actually, there are more exceptions to this rule than there are cases that obey it.
In a similar vein, when talking about
stderr it is convenient to trot out the accepted axiom that a process neither knows nor cares where its three standard streams are terminated. Should a process care whether its output is going to the terminal or being redirected into a file? Can it even tell if its input is coming from the keyboard or is being piped into it from another process?
Actually, a process does know—or at least it can find out, should it choose to check—and it can change its behavior accordingly if the software author decided to add that functionality.
We can see this change in behavior very easily. Try these two commands:
ls | cat
ls command behaves differently if its output (
stdout) is being piped into another command. It is
ls that switches to a single column output, it isn’t a conversion performed by
ls does the same thing if its output is being redirected:
ls > capture.txt
Redirecting stdout and stderr
There’s an advantage to having error messages delivered by a dedicated stream. It means we can redirect a command’s output (
stdout) to a file and still see any error messages (
stderr) in the terminal window. You can react to the errors if you need to, as they occur. It also stops the error messages from contaminating the file that
stdout has been redirected into.
Type the following text into an editor and save it to a file called error.sh.
#!/bin/bash echo "About to try to access a file that doesn't exist" cat bad-filename.txt
Make the script executable with this command:
chmod +x error.sh
The first line of the script echoes text to the terminal window, via the
stdout stream. The second line tries to access a file that doesn’t exist. This will generate an error message that is delivered via
Run the script with this command:
We can see that both streams of output,
stderr, have been displayed in the terminal windows.
Let’s try to redirect the output to a file:
./error.sh > capture.txt
The error message that is delivered via
stderr is still sent to the terminal window. We can check the contents of the file to see whether the
stdout output went to the file.
The output from
stdin was redirected to the file as expected.
> redirection symbol works with
stdout by default. You can use one of the numeric file descriptors to indicate which standard output stream you wish to redirect.
To explicitly redirect
stdout, use this redirection instruction:
To explicitly redirect
stderr, use this redirection instruction:
Let’s try to our test again, and this time we’ll use
./error.sh 2> capture.txt
The error message is redirected and the
echo message is sent to the terminal window:
Let’s see what is in the capture.txt file.
stderr message is in capture.txt as expected.
Redirecting Both stdout and stderr
Surely, if we can redirect either
stderr to a file independently of one another, we ought to be able to redirect them both at the same time, to two different files?
Yes, we can. This command will direct
stdout to a file called capture.txt and
stderr to a file called error.txt.
./error.sh 1> capture.txt 2> error.txt
Because both streams of output–standard output and standard error—are redirected to files, there is no visible output in the terminal window. We are returned to the command line prompt as though nothing has occurred.
Let’s check the contents of each file:
Redirecting stdout and stderr to the Same File
That’s neat, we’ve got each of the standard output streams going to its own dedicated file. The only other combination we can do is to send both
stderr to the same file.
We can achieve this with the following command:
./error.sh > capture.txt 2&>1
Let’s break that down.
- ./error.sh: Launches the error.sh script file.
- > capture.txt: Redirects the
stdoutstream to the capture.txt file.
>is shorthand for
- 2>&1: This uses the &> redirect instruction. This instruction allows you to tell the shell to make one stream got to the same destination as another stream. In this case, we’re saying “redirect stream 2,
stderr, to the same destination that stream 1,
stdout, is being redirected to.”
There is no visible output. That’s encouraging.
Let’s check the capture.txt file and see what’s in it.
stderr streams have been redirected to a single destination file.
To have the output of a stream redirected and silently thrown away, direct the output to
Detecting Redirection Within a Script
We discussed how a command can detect if any of the streams are being redirected, and can choose to alter its behavior accordingly. Can we accomplish this in our own scripts? Yes, we can. And it is a very easy technique to understand and employ.
Type the following text into an editor and save it as input.sh.
#!/bin/bash if [ -t 0 ]; then echo stdin coming from keyboard else echo stdin coming from a pipe or a file fi
Use the following command to make it executable:
chmod +x input.sh
The clever part is the test within the square brackets. The
-t (terminal) option returns true (0) if the file associated with the file descriptor terminates in the terminal window. We’ve used the file descriptor 0 as the argument to the test, which represents
stdin is connected to a terminal window the test will prove true. If
stdin is connected to a file or a pipe, the test will fail.
We can use any convenient text file to generate input to the script. Here we’re using one called dummy.txt.
./input.sh < dummy.txt
The output shows that the script recognizes that the input isn’t coming from a keyboard, it is coming from a file. If you chose to, you could vary your script’s behavior accordingly.
That was with a file redirection, let’s try it with a pipe.
cat dummy.txt | ./input.sh
The script recognizes that its input is being piped into it. Or more precisely, it recognizes once more that the
stdin stream is not connected to a terminal window.
Let’s run the script with neither pipes nor redirects.
stdin stream is connected to the terminal window, and the script reports this accordingly.
To check the same thing with the output stream, we need a new script. Type the following into an editor and save it as output.sh.
#!/bin/bash if [ -t 1 ]; then echo stdout is going to the terminal window else echo stdout is being redirected or piped fi
Use the following command to make it executable:
chmod +x input.sh
The only significant change to this script is in the test in the square brackets. We’re using the digit 1 to represent the file descriptor for
Let’s try it out. We’ll pipe the output through
./output | cat
The script recognizes that its output is no going directly to a terminal window.
We can also test the script by redirecting the output to a file.
./output.sh > capture.txt
There is no output to the terminal window, we are silently returned to the command prompt. As we’d expect.
We can look inside the capture.txt file to see what was captured. Use the following command to do so.
Again, the simple test in our script detects that the
stdout stream is not being sent directly to a terminal window.
If we run the script without any pipes or redirections, it should detect that
stdout is being delivered directly to the terminal window.
And that’s exactly what we see.
Streams Of Consciousness
Knowing how to tell if your scripts are connected to the terminal window, or a pipe, or are being redirected, allows you to adjust their behavior accordingly.
Logging and diagnostic output can be more or less detailed, depending on whether it is going to the screen or to a file. Error messages can be logged to a different file than the normal program output.
As is usually the case, more knowledge brings more options.