Linux terminal on a laptop
Fatmawati Achmad Zaenuri/Shutterstock.com

Use Linux pipes to choreograph how command-line utilities collaborate. Simplify complex processes and boost your productivity by harnessing a collection of standalone commands and turning them into a single-minded team. We show you how.

Pipes Are Everywhere

Pipes are one of the most useful command-line features that Linux and Unix-like operating systems have. Pipes are used in countless ways. Look at any Linux command line article—on any web site, not just ours—and you’ll see that pipes make an appearance more often than not. I reviewed some of How-To Geek’s Linux articles, and pipes are used in all of them, one way or another.

Linux pipes allow you to perform actions that are not supported out-of-the-box by the shell. But because the Linux design philosophy is to have many small utilities that perform their dedicated function very well, and without needless functionality—the “do one thing and do it well” mantra—you can plumb strings of commands together with pipes so that the output of one command becomes the input of another. Each command you pipe in brings its unique talent to the team, and soon you find you’ve assembled a winning squad.

A Simple Example

Suppose we have a directory full of many different types of file. We want to know how many files of a certain type are in that directory. There are other ways to do this, but the object of this exercise is to introduce pipes, so we’re going to do it with pipes.

We can get a listing of the files easily using ls:

ls

Collection if files in a directory, in a terminal window

To separate out the file type of interest, we’ll use grep. We want to find files that have the word “page” in their filename or file extension.

We will use the shell special character “|” to pipe the output from ls into grep.

ls | grep "page"

ls -l | grep "page" in a terminal window

grep prints lines that match its search pattern. So this gives us a listing containing only “.page” files.

listing of page files in a terminal window

Even this trivial example displays the functionality of pipes. The output from ls was not sent to the terminal window. It was sent to grep as data for the grep command to work with. The output we see comes from grep, which is the last command in this chain.

Extending Our Chain

Let’s start extending our chain of piped commands. We can count the “.page” files by adding the wc command. We will use the -l (line count) option with wc. Note we’ve also added the -l (long format) option to ls . We’ll be using this shortly.

ls - | grep "page" | wc -l

ls - | grep "page" | wc -l in a terminal window

grep is no longer the last command in the chain, so we don’t see its output. The output from grep is fed into the wc command. The output we see in the terminal window is from wc. wc reports that there are 69 “.page” files in the directory.

Let’s extend things again. We’ll take the wc command off the command line and replace it with awk. There are nine columns in the output from ls with the -l (long format) option. We’ll use awk to print columns five, three, and nine. These are the size, owner, and name of the file.

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}'

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}' in a terminal window

We get a listing of those columns, for each of the matching files.

Three column listing for each matching file in a terminal window

We’ll now pass that output through the sort command. We’ll use the -n (numeric) option to let sort know the first column should be treated as numbers.

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}' | sort -n

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}' | sort -n in a terminal window

The output is now sorted in file size order, with our customized selection of three columns.

Files sorted by size in a terminal window

Adding Another Command

We’ll finish off by adding in the tail command. We’ll tell it to list the last five lines of output only.

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}' | sort -n | tail -5

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}' | sort -n | tail -5 in a terminal window

This means our command translates to something like “show me the five largest “.page” files in this directory, ordered by size.” Of course, there is no command to accomplish that, but by using pipes, we’ve created our own. We could add this—or any other long command—as an alias or shell function to save all the typing.

Here is the output:

Five largest .page files listed by size order in a terminal window

We could reverse the size order by adding the -r (reverse) option to the sort command, and using head instead of tail  to pick the lines from the top of the output.

ls -l | grep "page" | awk '{print $5 " " $3 " " $9}' | sort -rn | head -5 in a terminal window

This time the five largest “.page” files are listed from largest to smallest:

Five largest .page files listed in reverse size order in a terminal window

Some Recent Examples

Here are two interesting examples from recent How-To geek articles.

Some commands, such as the xargscommand, are designed to have input piped to them. Here’s a way we can have wc count the words, characters, and lines in multiple files, by piping ls into xargs which then feeds the list of filenames to wc as though they had been passed to wc as command line parameters.

ls *.page | xargs wc

ls *.page | xargs wc in a terminal window

The total numbers of words, characters, and lines are listed at the bottom of the terminal window.

Count of words, charcaters, and lines in a terminal window

Here’s a way to get a sorted list of the unique file extensions in the current directory, with a count of each type.

ls | rev | cut -d'.' -f1 | rev | sort | uniq -c

ls | rev | cut -d'.' -f1 | rev | sort | uniq -c in a terminal window

There’s a lot going on here.

The output shows the list of file extensions, sorted alphabetically with a count of each unique type.

List of unique file extensions in a terminal window

Named Pipes

There’s another type of pipe available to us, called named pipes. The pipes in the previous examples are created on-the-fly by the shell when it processes the command line. The pipes are created, used, and then discarded. They are transient and leave no trace of themselves. They exist only for as long as the command using them is running.

Named pipes appear as persistent objects in the filesystem, so you can see them using ls. They’re persistent because they will survive a reboot of the computer—although any unread data in them at that time will be discarded.

Named pipes were used a lot at one time to allow different processes to send and receive data, but I haven’t seen them used that way for a long time. No doubt there are people out there still using them to great effect, but I’ve not encountered any recently. But for completeness’ sake, or just to satisfy your curiosity, here’s how you can use them.

Named pipes are created with the mkfifo command. This command will create a named pipe called “geek-pipe” in the current directory.

mkfifo geek-pipe

mkfifo geek-pipe in a terminal window

We can see the details of the named pipe if we use the ls command with the -l (long format) option:

ls -l geek-pipe

ls -l geek-pipe in a terminal window

The first character of the listing is a “p”, meaning it is a pipe. If it was a “d”, it would mean the file system object is a directory, and a dash “-” would mean it is a regular file.

Using the Named Pipe

Let’s use our pipe. The unnamed pipes we used in our previous examples passed the data immediately from the sending command to the receiving command. Data sent through a named pipe will stay in the pipe until it is read. The data is actually held in memory, so the size of the named pipe will not vary in ls listings whether there is data in it or not.

We’re going to use two terminal windows for this example. I’ll use the label:

# Terminal-1

in one terminal window and

# Terminal-2

in the other, so you can differentiate between them. The hash “#” tells the shell that what follows is a comment, and to ignore it.

Let’s take the entirety of our previous example and redirect that into the named pipe. So we’re using both unnamed and named pipes in one command:

ls | rev | cut -d'.' -f1 | rev | sort | uniq -c > geek-pipe

ls | rev | cut -d'.' -f1 | rev | sort | uniq -c > geek-pipe in a terminal window

Nothing much will appear to happen. You may notice that you don’t get returned to the command prompt though, so something is going on.

In the other terminal window, issue this command:

cat < geek-pipe

cat < geek-pipe in a terminal window

We’re redirecting the contents of the named pipe into cat, so that cat will display that content in the second terminal window. Here’s the output:

The contents of the named piped shown in a terminal window

And you’ll see that you have been returned to the command prompt in the first terminal window.

Completed task and the command prompt in in a terminal window

So, what just happened.

  • We redirected some output into the named pipe.
  • The first terminal window did not return to the command prompt.
  • The data remained in the pipe until it was read from the pipe in the second terminal.
  • We were returned to the command prompt in the first terminal window.

You may be thinking that you could run the command in the first terminal window as a background task by adding an & to the end of the command. And you’d be right. In that case, we would have been returned to the command prompt immediately.

The point of not using background processing was to highlight that a named pipe is a blocking process. Putting something into a named pipe only opens one end of the pipe. The other end isn’t opened until the reading program extracts the data. The kernel suspends the process in the first terminal window until the data is read from the other end of the pipe.

The Power of Pipes

Nowadays, named pipes are something of a novelty act.

Plain old Linux pipes, on the other hand, are one of the most useful tools you can have in your terminal window toolkit. The Linux command line starts to come alive for you, and you get a whole new power-up when you can orchestrate a collection of commands to produce one cohesive performance.

Parting hint: It is best to write your piped commands by adding one command at a time and getting that portion to work, then piping in the next command.

Dave McKay Dave McKay
Dave McKay first used computers when punched paper tape was in vogue, and he has been programming ever since. He is now a Data Protection Officer and has worked as a freelance programmer, manager of an international software development team, and an IT services project manager. Dave is a Linux evangelist and open source advocate.
Read Full Bio »