How to Perform a Task When a New File is Added to a Directory in Linux

• Taylor Gibb

  @taybgibb



• March 25, 2019, 8:00am EDT

This tutorial will show you how to use Linux filesystem events (notify) to get notified every time a file appears in a directory. You could use these as triggers to automate common tasks on your system.

We’re going to write a script that watches a directory and acts on new files that are added. Each file is gzipped and moved to another directory, as soon as it is detected. The script uses the inotify subsystem, through a utility called inotify-tools. But first, let’s install the tool and experiment.

Installing inotify-tools and gzip

Use apt-get to install this package onto your system if you’re using Ubuntu or another Debian-based distribution. On other Linux distributions, use your Linux distribution’s package management tool instead.

sudo apt-get install inotify-tools gzip

Experimenting with inotify-tools

Let’s begin by watching a directory and seeing what events initiate when new files arrive. We will use a tool called  inotifywatch , which is part of inotify-tools. Create a new directory called “incoming”:

mkdir incoming

Start watching this directory by executing the following command:

inotifywatch -v incoming

This will instruct inotify to watch for all filesystem events in the “incoming” directory. The -v option makes the tool print out extra information about what it’s doing. We haven’t specified a timeout option (-t ), and the command will keep gathering events until we exit with CTRL+C. At this point, our terminal should look something like this:

Open a new terminal window (or tab) and change to the incoming directory. Use the touch command to create a new file named “newfile.”

cd incoming/
touch newfile

Now go back to the first terminal window and stop inotifywatch by hitting CTRL+C.

A table of events will be served to the console, indicating one instance of “create,” “open,” “attrib,” and “close_write.” These four events occurred when we used touch to create a new file, set its file access attributes, opened it to write a null terminating character, and then closed it afterward. These are just a few of the multitude of events that can be monitored on a filesystem with inotify-tools. You can see the full list on the main page for inotifywatch.

For our purposes we’re only interested in two events:

• “create” – when a file is created in the target directory.
• “moved_to” – when a file is moved from another location into the target directory.

Let’s try inotifywatch again, but this time instructing it to monitor these two events only. Run this command in the first terminal window:

inotifywatch -v -e create -e moved_to incoming

In the second terminal window or tab, let’s try creating a new file, changing its contents, and then moving the file from another location to the target directory. All these commands are run from the home directory.

touch incoming/created
echo Testing123 >> incoming/created
touch /tmp/created2
mv /tmp/created2 incoming/

Go back to the first terminal window and stop inotifywatch by hitting CTRL+C. We’ll see the following output:

Only two events were counted: creating a file called “created.txt” and moving an existing file called “created2.txt”. Everything else, such as modifying “created.txt,” was ignored.

Watching a Directory and Executing a Task

Now that we know what events to follow, we can use another tool called  inotifywait to block execution until a file is created in or moved to our target directory. We’ll use the same arguments as we did with inotifywatch and also specify how we want the filename to be formatted for use in our task.

Before we begin, we need a directory to hold files that have already been processed. Create a directory called “processed”:

mkdir processed

Next, create a new script called “watch-incoming.sh” and add the contents listed below:

#!/bin/bash

TARGET=~/incoming/
PROCESSED=~/processed/

inotifywait -m -e create -e moved_to --format "%f" $TARGET \
        | while read FILENAME
                do
                        echo Detected $FILENAME, moving and zipping
                        mv "$TARGET/$FILENAME" "$PROCESSED/$FILENAME"
                        gzip "$PROCESSED/$FILENAME"
                done

The script executes the  inotifywait command with the -m option. This makes the command monitor changes indefinitely. Each time a new event is detected, the filename is passed to the read command and injected into the “FILENAME” variable. The block under the while loop is executed, in which the file is first moved to the “processed” directory and then gzipped. The original file is replaced with the compressed file, and the filename will end in “.gz”.

Let’s grant execute permissions on this script and run it from our home directory.

chmod u+x watch-incoming.sh
./watch-incoming.sh

Open the second terminal window and create a new file in the “incoming” directory. List the contents of both the “incoming” and “processed” directory to see the results of the event being detected:

The raw text file that we copied into the “incoming” directory was detected by the script, copied into “processed” and then compressed using gzip.

We can do some interesting tasks now that we’re able to watch for new files arriving in a directory. For example, we could add a watermark to image files, compress raw videos into mp4 format and even upload every new file we see to an Amazon S3 bucket. This script is a good starting point for rolling your own workflows and automating common tasks on your system.