The Linux expect command lets you automate interactions with scripts and programs. You can send any kind of response to the script when it is waiting for some text input.

Automation Means Efficiency

When you administer a Linux computer or group of computers you'll run up against many repetitive tasks. The obvious answer is to write a script to perform the bulk of the work for you. The Bash shell, like all modern shells, provides a rich and flexible scripting language.

Scripting a long-winded or tedious job gives you the flexibility to run that task when the office is closed, or in the quietest periods of a 24/7 operation. Scripts also give you repeatability. They won't forget to perform a step, no matter how many times they are asked to perform the same boring chores.

That's fine, as far as it goes. But if your script requires user interaction, or if it calls a program that will need human input, what can you do? You don't want to have to be present when the script runs, or if you are present, to be watching the terminal window ready to jump in and hit a few keys.

Linux has the yes command, which sends a stream of "y" characters or any other user-specified string to the terminal window. If a program is waiting for a response to a yes or no question it'll be force-fed one of the "y" characters. It'll accept it as input and will be able to proceed. You need to pipe the output from yes into the script.

yes | script.sh

You might not want to send an affirmative response. Perhaps you need to say "no." You can do that too, using the yes command. You can send any phrase you like, you're not restricted to responses like "y", "Y", "Yes", "n", "N", or "No."

Perhaps you need to send an "r" for reinstall to an "install/upgrade/reinstall [i/u/r] " prompt.

yes r

The yes command can't cope if it needs to provide more than one type of response. For that situation, we need to use expect .

Related: How to Use the yes Command on Linux

Installing the expect Command

We tested expect on Ubuntu, Fedora, and Manjaro. The package wasn't bundled with these distributions, so it had to be installed. On Ubuntu type:

sudo apt install expect

On Fedora, the command you need is:

sudo dnf install expect

On Manjaro we use pacman:

sudo pacman -Sy expect

How expect Works

The expect command lets you manage the two ends of the conversation between your set of prepared answers and the program or script that you'll be sending them to. You do this by creating an "expect" script that watches for prompts in the main script and sends the appropriate response for each one.

Let's say you have a backup script that asks for a source directory name and a target directory name. It then makes a copy of the source directory files in the target directory. Without the bits that do the file copying, your script might look something like this:

#!/bin/bash
    
    
            echo "Directory to backup?"
    
    
            read source_directory
    
    
            echo "Backup location?"
    
    
            read target_directory
    
    
            echo
    
    
            echo "Target directory:" $target_directory

All this script does is ask for the paths to the two directories. It prints the target directory to the terminal window so that we can see that it received a response from expect and that it could read it correctly.

To provide the other half of the conversation we create an "expect script." By convention, these have a ".exp" extension. This example will work with our "backup.sh" script.

#!/usr/bin/expect -f
    
    
            set timeout -1
    
    
            spawn ./backup.sh
    
    
            expect "Directory to backup?r"
    
    
            send -- "/home/dave/Documents/r"
    
    
            expect "Backup location?r"
    
    
            send -- "/media/dave/external/backupr"
    
    
            expect eof

This shows the main three commands in expect scripts, the spawn, expect, and send commands.

The first thing to notice is the shebang is referring to expect, not to bash. The -f (file) flag tells expect the responses are coming from a file.

We effectively turn off timeouts by setting them to infinity with -1.

The spawn command launches the backup script.

We know the two questions the "backup.sh" script will ask us. For each question we create an "expect" line. These contain the text prompt that will be sent to the terminal window by the "backup.sh" script. This what the expect program will watch out for. When expect sees the prompt, the text in the send line is returned to the "backup.sh" script.

The expect eof lines tells expect to wait for the final end-of-file at the completion of processing in the automated script.

Make both scripts executable:

chmod +x backup.sh

chmod +x backup.exp

And use the "backup.exp" script to fire off the whole process.

./backup.exp

You can see the two prompts from "backup.sh" and the responses from "backup.exp."

That'll get Fiddly Fast

It's great to see two scripts interacting like this, with the automated responses being sent by our expect script and being accepted as genuine input by the automated script. This is just a simple example, of course. If you try to automate a long-winded process---and they're the ones you'll want to automate---you can soon find yourself bogged down.

The text in the expect lines must match exactly the prompts from the script you're automating. Misspellings and incorrect wording will cause a prompt to be skipped, and the automated process will stall. If you're still developing or tweaking the script you're trying to automate, the wording of prompts and their order within the script are likely to change.  Upgrades to scripts or programs might change, remove, or introduce prompts. Keeping track of the changes and replicating them in your expect script becomes tiresome and error-prone.

The most convenient way to create an expect script is to use autoexpect. This is installed along with expect. We can tell autoexpect to monitor our real-life interaction with a script or program, and to generate an expect file for us.

There might be some tidying up to do in the generated expect file, but it's much faster than writing one by hand. For example, if you mistype something in your session when autoexpect is monitoring you, the incorrect keystrokes and your editing and backspacing to correct them will all be captured. To tidy that up, simply delete that section and type the correct text.

Using autoexpect

Using autoexpect is easy. We use the -f (filename) flag to tell it the name of the expect file it should create, and the program or script that we want it to run. The prompts from the process and our responses are recorded and sent to the expect file. As a nice touch, autoexpect makes the expect file executable for us.

Let's say we frequently use rsync to send files from one computer to a directory on another computer over the network. We'll use autoexpect to create an expect file of that process.

Our command line looks like this. The first command is autoexpect  followed by the -f flag and the name of the expect file we want to create. In this case, it is "send-pics.exp."

The rest of the command is the regular rsync command. Because rsync uses the SSH protocol to remotely connect to the target computer, we'll need to provide the password for user account "dave" on the target computer.

autoexpect -f send-pics.exp rsync -rasv ~/Pictures/raw/ dave@nostromo.local:/home/dave/raw/

When we execute that command rsync is launched and we're prompted for the password for user dave's account on the remote computer. Once that's entered, some files are sent to the remote computer, rsync breaks the connection, and the process ends.

We're told that the file "send-pics.exp" has been created. Let's take a look:

ls -l send-pics.exp

It has indeed been created, and it is executable. We can run it by calling it by name:

./send-pics.exp

This time the process runs without any human interaction. Any new files since the last transfer are sent to the remote machine.

This is a very small example script. It hardly saves any effort compared to running the rsync command by hand. While that's true, it does illustrate the point that you can control programs as well as scripts, and you can apply the principles seen here to scripts or processes of any length.

Related: How to Back Up Your Linux System With rsync

Hang On, My Password!

You need to be aware that any passwords---or any other sensitive information---gathered during an autoexpect session are stored in plain text in the generated expect script. Passwords should never be written in plain text anywhere. For SSH connections a  better solution is to use SSH keys and have a passwordless authentication scheme.

Obviously, that's not going to help if you're not using SSH.

You can edit your expect script and change the line that deals with your password to:

interact ++ return

When the script reaches this line it will await your input, allowing you to enter your password. Control then returns to the expect script once you've provided your password.

./send-pics.exp

But that presents a different problem. It means your script is no longer able to run unattended. If you'll always be hand-launching your script that probably doesn't matter, especially if the password request happens right at the start of the script. You can launch the script, enter your password, and then let it look after itself.

Another way to address the issue is to:

• Change the file permissions on your expect script using chown to 740, -rwxr-x---.
• Use chmod to change the group owner of the expect script to a trusted group.
• Create a regular script that launches your expect script. Set this file to be owned by the trusted group too. Set the permissions on this to 2751, -rwxr-s--x.
• This creates a launcher script that anyone can read and execute, that is in the same trusted group as the expect script, and with its setgid bit set. Regardless of who runs this script, its effective group will be the trusted group and not the group of the person running the script. So this script will always be able to launch the expect script.
• Meanwhile, the expect script containing the password is inaccessible to anyone but you.

Related: How to Use SUID, SGID, and Sticky Bits on Linux

You Can expect Great Things

There's enough here to get you going, but there's a lot more to expect than the areas we've covered. The expect man page is over 1700 lines!

For tasks like automated installs, scheduled backups, or repetitive deployments, expect will transform your workflow.