SEARCH

How-To Geek

The Beginner’s Guide to Shell Scripting: The Basics

banner-01

The term “shell scripting” gets mentioned often in Linux forums but many users aren’t familiar with it. Learning this easy and powerful programming method can help you save time, learn the command-line better, and banish tedious file management tasks.

What Is Shell Scripting?

Being a Linux user means you play around with the command-line. Like it or not, there are just some things that are done much more easily via this interface than by pointing and clicking. The more you use and learn the command-line, the more you see its potential. Well, the command-line itself is a program: the shell. Most Linux distros today use Bash, and this is what you’re really entering commands into.

Now, some of you who used Windows before using Linux may remember batch files. These were little text files that you could fill with commands to execute and Windows would run them in turn. It was a clever and neat way to get some things done, like run games in your high school computer lab when you couldn’t open system folders or create shortcuts. Batch files in Windows, while useful, are a cheap imitation of shell scripts.

cbr script

Shell scripts allow us to program commands in chains and have the system execute them as a scripted event, just like batch files. They also allow for far more useful functions, such as command substitution. You can invoke a command, like date, and use it’s output as part of a file-naming scheme. You can automate backups and each copied file can have the current date appended to the end of its name. Scripts aren’t just invocations of commands, either. They’re programs in their own right. Scripting allows you to use programming functions – such as ‘for’ loops, if/then/else statements, and so forth – directly within your operating system’s interface. And, you don’t have to learn another language because you’re using what you already know: the command-line.

That’s really the power of scripting, I think. You get to program with commands you already know, while learning staples of most major programming languages. Need to do something repetitive and tedious? Script it! Need a shortcut for a really convoluted command? Script it! Want to build a really easy to use command-line interface for something? Script it!

Before You Begin

Before we begin our scripting series, let’s cover some basic information. We’ll be using the bash shell, which most Linux distributions use natively. Bash is available for Mac OS users and Cygwin on Windows, too. Since it’s so universal, you should be able to script regardless of your platform. In addition, so long as all of the commands that are referenced exist, scripts can work on multiple platforms with little to no tweaking required.

Scripting can easily make use of “administrator” or “superuser” privileges, so it’s best to test out scripts before you put them to work. Also use common sense, like making sure you have backups of the files you’re about to run a script on. It’s also really important to use the right options, like –i for the rm command, so that your interaction is required. This can prevent some nasty mistakes. As such, read through scripts you download and be careful with data you have, just in case things go wrong.

At their core, scripts are just plain text files. You can use any text editor to write them: gedit, emacs, vim, nano… This list goes on. Just be sure to save it as plain text, not as rich text, or a Word document. Since I love the ease of use that nano provides, I’ll be using that.

Script Permissions and Names

Scripts are executed like programs, and in order for this to happen they need to have the proper permissions. You can make scripts executable by running the following command on it:

chmod +x ~/somecrazyfolder/script1

This will allow anyone to run that particular script. If you want to restrict its use to just your user, you can use this instead:

chmod u+x ~/somecrazyfolder/script1

In order to run this script, you would have to cd into the proper directory and then run the script like this:

cd ~/somecrazyfolder

./script1

To make things more convenient, you can place scripts in a “bin” folder in your home directory:

~/bin

In many modern distros, this folder no longer is created by default, but you can create it. This is usually where executable files are stored that belong to your user and not to other users. By placing scripts here, you can just run them by typing their name, just like other commands, instead of having to cd around and use the ‘./’ prefix.

Before you name a script, though, you should the following command to check if you have a program installed that uses that name:

which [command]

A lot of people name their early scripts “test,” and when they try to run it in the command-line, nothing happens. This is because it conflicts with the test command, which does nothing without arguments. Always be sure your script names don’t conflict with commands, otherwise you may find yourself doing things you don’t intend to do!

Scripting Guidelines

guidelines

As I mentioned before, every script file is essentially plain text. That doesn’t mean you can write what you want all willy-nilly, though. When a text file is attempted to be executed, shells will parse through them for clues as to whether they’re scripts or not, and how to handle everything properly. Because of this, there are a few guidelines you need to know.

  1. Every script should being with “#!/bin/bash”
  2. Every new line is a new command
  3. Comment lines start with a #
  4. Commands are surrounded by ()

The Hash-Bang Hack

When a shell parses through a text file, the most direct way to identify the file as a script is by making your first line:

#!/bin/bash

If you use another shell, substitute its path here. Comment lines start with hashes (#), but adding the bang (!) and the shell path after it is a sort of hack that will bypass this comment rule and will force the script to execute with the shell that this line points to.

New Line = New Command

Every new line should be considered a new command, or a component of a larger system. If/then/else statements, for example, will take over multiple lines, but each component of that system is in a new line. Don’t let a command bleed over into the next line, as this can truncate the previous command and give you an error on the next line. If your text editor is doing that, you should turn off text-wrapping to be on the safe side. You can turn off text wrapping in nano bit hitting ALT+L.

Comment Often with #s

If you start a line with a #, the line is ignored. This turns it into a comment line, where you can remind yourself of what the output of the previous command was, or what the next command will do. Again, turn off text wrapping, or break you comment into multiple lines that all begin with a hash. Using lots of comments is a good practice to keep, as it lets you and other people tweak your scripts more easily. The only exception is the aforementioned Hash-Bang hack, so don’t follow #s with !s. ;-)

Commands Are Surrounded By Parentheses

In older days, command substitutions were done with single tick marks (`, shares the ~ key). We’re not going to be touching on this yet, but as most people go off and explore after learning the basics, it’s probably a good idea to mention that you should use parentheses instead. This is mainly because when you nest – put commands inside other commands – parentheses work better.

Your First Script

Let’s start with a simple script that allows you to copy files and append dates to the end of the filename. Let’s call it “datecp”. First, let’s check to see if that name conflicts with something:

which cp

You can see that there’s no output of the which command, so we’re all set to use this name.

Let’s create a blank file in the ~/bin folder:

touch ~/bin/datecp

touch

And, let’s change the permission now, before we forget:

chmod

Let’s start building our script then. Open up that file in your text editor of choice. Like I said, I like the simplicity of nano.

nano ~/bin/datecp

And, let’s go ahead and put in the prerequisite first line, and a comment about what this script does.

hashbang hack

Next, let’s declare a variable. If you’ve ever taken algebra, you probably know what a that is. A variable allows us to store information and do things with it. Variables can “expand” when referenced elsewhere. That is, instead of displaying their name, they will display their stored contents. You can later tell that same variable to store different information, and any instruction that occurs after that will use the new information. It’s a really fancy placeholder.

What will we put in out variable? Well, let’s store the date and time! To do this, we’ll call upon the date command.

Take a look at the screenshot below for how to build the output of the date command:

date output

You can see that by adding different variables that start with %, you can change the output of the command to what you want. For more information, you can look at the manual page for the date command.

Let’s use that last iteration of the date command, “date +%m_%d_%y-%H.%M.%S”, and use that in our script.

date in script

If we were to save this script right now, we could run it and it would give us the output of the date command like we’d expect:

date script output

But, let’s do something different. Let’s give a variable name, like date_formatted to this command. The proper syntax for this is as follows:

variable=$(command –options arguments)

And for us, we’d build it like this:

date_formatted=$(date +%m_%d_%y-%H.%M.%S)

date as variable

This is what we call command substitution. We’re essentially telling bash that whenever the variable “date_formatted” shows up, to run the command inside the parentheses. Then, whatever output the commands gives should be displayed instead of the name of the variable, “date_formatted”.

Here’s an example script and its output:

echo date script

echo date output

Note that there are two spaces in the output. The space within the quotes of the echo command and the space in front of the variable are both displayed. Don’t use spaces if you don’t want them to show up. Also note that without this added “echo” line, the script would give absolutely no output.

Let’s get back to our script. Let’s next add in the copying part of the command.

cp –iv $1 $2.$date_formatted

appended filename

This will invoke the copy command, with the –i and –v options. The former will ask you for verification before overwriting a file, and the latter will display what is being down on the command-line.

Next, you can see I’ve added the “$1” option. When scripting, a dollar sign ($) followed by a number will denote that numbered argument of the script when it was invoked. For example, in the following command:

cp –iv Trogdor2.mp3 ringtone.mp3

The first argument is “Trogdor2.mp3” and the second argument is “ringtone.mp3”.

Looking back at our script, we can see that we’re referencing two arguments:

appended filename

This means that when we run the script, we’ll need to provide two arguments for the script to run correctly. The first argument, $1, is the file that will be copied, and is substituted as the “cp –iv” command’s first argument.

The second argument, $2, will act as the output file for the same command. But, you can also see that it’s different. We’ve added a period and we’ve referenced the “date_formatted” variable from above. Curious as to what this does?

Here’s what happens when the script is run:

appended filename output

You can see that the output file is listed as whatever I entered for $2, followed by a period, then the output of the date command! Makes sense, right?

Now when I run the datecp command, it will run this script and allow me to copy any file to a new location, and automatically add the date and time to end of the filename. Useful for archiving stuff!

 


Shell scripting is at the heart of making your OS work for you. You don’t have to learn a new programming language to make it happen, either. Try scripting with some basic commands at home and start thinking of what you can use this for.

 

Do you script? Have any advice for newbies? Share your thoughts in the comments! There’s more to come in this series!

Yatri Trivedi is a monk-like geek. When he's not overdosing on meditation and geek news of all kinds, he's hacking and tweaking something, often while mumbling in 4 or 5 other languages.

  • Published 07/5/11

Comments (26)

  1. ChrisC

    I just started learning shell scripting the other day!! This is rather basic compared to what I was trying to accomplish the other day, however its a great guide for beginners!! Its quite nice to start learning stuff like this with a purpose in mind already; it helps motivate you quite a lot.

  2. Zenchaos

    I’ve been using a bash prompt for a while now and I still learned something. Nice tutorial!

  3. Myles Braithwaite

    Thanks for the great shell script idea. I updated the script a little so it would add the file extension at the end of (like “music.2011-07-05_09.4858.mp3″). https://gist.github.com/1064873

  4. Skiddie

    You should actually put ‘user’ scripts in sbin not bin.
    Also, the editor really needs to take a look at the spelling and grammar of this post.

  5. ShofneDL

    Actually you would want to place the script in a directory defined in the PATH variable.

    On my system-
    david@ASUS-PC:~$ echo $PATH
    /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games

  6. BradR

    I have been looking into scripting and learning more about the command line. Can anyone tell me a good place to get a guide/place to learn.

  7. H4CKN3T

    What distro are you using that bin is located in the home directory? I’ve been stuck on Backtrack and Ubuntu (which are the same now), but I’ve always seen the bin directory under the actual root folder ( / ), and under the “usr” directory (/usr/bin).

    @Skiddie I’m going to disagree with you on the sbin vs bin post. The “S” stands for “system” and requires root privs to execute. The “/bin” and “/sbin” directory are often used for system startup (I think).

    If you just absolutely hate typing ./ before you run your scripts I would put them in the “/usr/bin/” directory and leave the rest of them alone that is not located in your ~/ directory.

  8. Yinon

    Really nice tut for beginners. cheers mate.

  9. Samuel Dionne-Riel

    Hi!

    Your example script is well done, uses no deprecated or “old-style” bash idioms, making this tutorial useful, but there is one thing that might cause problems here: Quoting.

    Arguments are space-separated in the bash shell, it will tokenize when there is a space in the resulted expanded command. In your script, cp $1 $2.$date_formatted will work as intended as long as the expanded variables do not have spaces in them. If you call your script this way: datecp "my old name" "my new name" the expansion will result in this command: cp my new name my old name.the_date which actually has 6 arguments.

    To properly address this issue, the last line of the script should be: cp "$1" "$2.$date_formatted"

    The quoting will make it that bash will tokenize them as two arguments. There is more information about that at the bash-hackers wiki. ( http://wiki.bash-hackers.org/syntax/words#example ).

    Just a reminder for anyone working on bash scripting: Quote arguments properly. This is important.

    Also, two notes:

    The blog’s administrator should look into installing a plugin that properly allows code to be formatted as WordPress will replace some things like quoting and dashes for typographically better ones, but improper for programming/scripting
    Reader should manually rewrite dashes, single quotes and double quotes when having trouble with copied examples here. This comment included.

  10. Samuel Dionne-Riel

    @H4CKN3T
    You are right when asking “What distro are you using that bin is located in the home directory?” as it isn’t part of any current filesystem standards.

    Though, it is usual for many *nix enthusiasts to have a bin directory in their home directories with their personal scripts. I personally use the .bin directory as it is normally hidden (beginning with a dot).

    The last issue with that is having the directory in your command search path.

    The article says:

    In many modern distros, [the bin] folder no longer is created by default, but you can create it. This is usually where executable files are stored that belong to your user and not to other users. By placing scripts here, you can just run them by typing their name, just like other commands, instead of having to cd around and use the ‘./’ prefix.

    This wrong in saying that “By placing scripts here, you can just run them by typing their name”. It will not run if that directory is not in your search path. There are great chances that if you didn’t already have that directory that it won’t be in it.

    I won’t explain what it does, as Wikipedia has a great article on that. ( http://en.wikipedia.org/wiki/PATH_(variable)#Unix_and_Unix-like ).

    If you want to add a folder to that search path, you can concatenate one this way: export PATH=~/bin/:"$PATH". You can either do this every time you open a console (which is tedious) or add this command to your profile. I found out lately that the profile comportment can vary on certain platform so I will refrain actually saying how to do it.

    You can search some informations on either the .profile file, the .bash_profile or the .bashrc files and find out which is the better one in your particular case.

  11. Arun

    I’m a beginner. This Tutorial is very useful to know what is bash, shell scripting. I was struck with /bin directory, but thanks to @H4CKN3T for telling /usr/bin…. Nice Tutorial …

  12. trm96

    Good article, great for new users who want to play around with the terminal.

    BTW
    I know this article was meant as a command line tutorial but I usually type out scripts via gedit or kate in the GUI.
    Also great job on telling people to use nano rather than emacs or vi witch are not as easy to use.

  13. lonestar

    Oh ye of little faith.
    You DO NOT pollute /bin, /sbin, /usr/bin or /usr/sbin with self-made scripts.
    There are only _two_ possible places for them:
    1) ~/bin
    2) /usr/local/bin
    Personally I would advocate ~/bin, but /usr/local/bin is also (almost) acceptable …
    Also – neither ~/bin or /usr/local/bin may be in your path, so make sure you set your PATH-variable accordingly.

  14. Skiddie

    My bad, i actually meant /usr/local/sbin

  15. Samuel Dionne-Riel

    There seems to be some misconception about what folders are used here in the comments. Fortunately, there exists a standard that explains the use of everything in the filesystem, the Filesystem Hierarchy Standard or FHS.

    According to the FHS 2.3:

    – /sbin : System binaries
    Purpose : Utilities used for system administration (and other root-only commands) are stored in /sbin, /usr/sbin, and /usr/local/sbin. /sbin contains binaries essential for booting, restoring, recovering, and/or repairing the system in addition to the binaries in /bin. [18] Programs executed after /usr is known to be mounted (when there are no problems) are generally placed into /usr/sbin. Locally-installed system administration programs should be placed into /usr/local/sbin. [19]

    – /usr/sbin : Non-essential standard system binaries.
    Purpose : This directory contains any non-essential binaries used exclusively by the system administrator. System administration programs that are required for system repair, system recovery, mounting /usr, or other essential functions must be placed in /sbin instead. [29]

    The /usr/local/ hierarchy is a clone of the /usr/ one, but used for “locally installed software”, by which I mean, software that was not installed via the package management software.

    Furthermore, the home folder is not standardized by the FHS, and the only specification I know of that applies to the home folder is the XDG Base Directory specification, which specifies mainly where configuration and cache data should be stored in the home folder.

  16. Blake

    @BradR I learned from http://linuxcommand.org/learning_the_shell.php it was a helpful site for when your first learning the command line.

  17. Alex

    Hey Guys, could you help me please, I really appreciate your efforts to write this useful article, but I need you advice. I followed this instructions, and everything worked perfect, except last point.

    When I wite next:

    datecp big.mp3 short.mp3

    I receive:

    cp: target `short.mp3.07_08_11-13.01.38′ is not a directory

    What I have to do with it? Or how to fix it! Thanks in advance.

  18. Alex

    Wowww….Sorry for bother, I found out it’s my fault. It was in this line:

    Right varient: cp -iv $1 $2.$date_formatted

    My bad: cp +iv $1 $2.$date_formatted

    Now, it works, thank you very much for this tutorial.

  19. me

    nano isn’t the most usefull editor to use.

  20. Martin

    if no bother using vi, gedit would be alot better than nano for doing this (I know it’s gui). Except the grammar and misspells, what I first noticed vas the “cp -iv” was invoked, ok I get the -v option, but how likely would it be that this command is ever to overwrite an existing file?? Due to file name is dependent on the exact month, day, year, hour, minute AND second of the copy!? I know it just works fine with the -i option but kinda feels like more work than needed, except if you copy a file (that takes less than a second) and then accidently evokes the command a second time (keep in mind the whole process have to happen in less than a second for the “bug” to appear), then again if you did. It would be the exact same file copied twice, nothing critical would ever be likely to happen without the -i option, it would only be annoying to hit “y” IF it ever occurred ;)

    Ps. great work :)

  21. Rajneesh Gadge

    Very good article to start from.

    Keep it up!

    Looking forward to have its successor also.

  22. Troy

    I like it! I have ~/bin in my path.

  23. nordik_14

    Sorry but I have a little problem. Console says that I have not permission to run this scipt or comand, just it works when I write sudo. I have done about that of permission (chmod u+x /bin/datecp). Is not to bad but I would prefer do this without sudo any idea?. This tutorial is very good.

  24. Al

    Great post, but I still have one question, who the fuck actually uses scripts, and what for????!? “File management tasks?” Who has file management tasks??? Im a CS student, senior year and i have NEVER had file management tasks. Why should i learn scripting? please convince me.

  25. Mandi

    This may seem like a silly question, but I’m doing this in Windows and I was wondering what the file extension is for scripts like these. Thanks.

  26. Dazed_75

    I think most now agree that the proper place for user developed scripts is either ~/bin or /usr/local/bin but there has been no discussion of how to choose based on the [dis]advantages of each. Personally, I prefer ~/bin unless and until the script is fully developed and tested AND it is to be used by other users on the system.

    /usr/local/bin has the advantages that it is generally included in the default $PATH for users. It has the slight disadvantage of needing root permissions to put anything in it. The larger disadvantage is that it WILL be replaced on a new install though it should not be on a distribution upgrade.

    ~/bin has some advantages. If /home is a separate partition it need not be formatted on a new install so that its contents are preserved. It does not require root permissions. Its contents are generally not seen or usable by other users so is semi-private while developing a script. A slight disadvantage is that it is generally not in the default $PATH for a user so must be added or the ./script notation used. Another note is that on completion it is a trivial task to sudo mv script /usr/local/bin and re-permission it for other users to use if that is desired.

Get Free Articles in Your Inbox!

Join 134,000 newsletter readers

Email:

Go check your email!