105.2 Customize or write simple scripts¶

Weight: 4

Candidates should be able to customize existing scripts, or write simple new Bash scripts.

Objectives

Use standard sh syntax (loops, tests).
Use command substitution.
Test return values for success or failure or other information provided by a command.
Execute chained commands.
Perform conditional mailing to the superuser.
Correctly select the script interpreter through the shebang (#!) line.
Manage the location, ownership, execution and suid-rights of scripts.

Terms

for, while, test, if, read, seq, exec, ||, &&

Chaining commands¶

Three ways to put several commands on one line:

Separator	Runs the next command
`A ; B`	always, whatever happened to A
`A && B`	only if A succeeded (exit status 0)
`A \|\| B`	only if A failed (exit status not 0)

Think of them as logic. With A && B, if A fails the whole thing is already false, so B is skipped. With A || B || C, if A works the whole thing is already true, so B and C are skipped; if A fails, B is tried, and so on.

$ cd /tmp; ls
$ cp backup.gzip /backups/ && rm backup.gzip
$ cp backup.gz /backups/ && rm backup.gz    # delete the file ONLY if the copy worked
$ ping -c1 host || echo "host is down"      # message ONLY if the ping failed

The second line deletes the original only if the copy worked. That is the typical use of &&.

Exit status¶

Every command returns a number when it finishes, its exit status (or return value). 0 means success, anything else means some kind of failure. The special variable $? holds the status of the last command:

$ touch /chert
touch: cannot touch '/chert': Permission denied
$ echo $?
1
$ touch /tmp/11
$ echo $?
0
$ dummycommand
dummycommand: command not found
$ echo $?
127
$ test -e /
$ echo $?
0
$ test -e /nonexist
$ echo $?
1

&&, ||, if and while all decide based on this number. In your own scripts, end with exit 0 for success or exit 1 (or another number) for an error, so whoever runs the script can test the result.

Shell scripts and the shebang¶

A script is a text file with commands, run one after the other by an interpreter. Scripts automate jobs: for example, a do_backup.sh might compress some files, name the archive after today's date, copy it to a backup server, and delete archives older than a month.

The first line chooses the interpreter. It starts with #!, called the shebang, followed by the program's full path:

#!/bin/bash

# A very simple script

echo "Cheers from the script file! Current time is: "
date +%H:%M

#!/bin/bash runs the script with Bash. #!/bin/sh uses sh, a simpler shell that understands most of the same syntax.
The shebang works for any language: #!/usr/bin/python, #!/usr/bin/perl, #!/usr/bin/awk.
On every other line, # starts a comment, and blank lines are ignored. Do not mix up a comment # with the shebang #!.
A .sh ending is not required, it just helps you recognise scripts.

Another tiny script:

#!/bin/bash

echo
echo "We are learning! Wowww..."
echo

Running a script¶

How	Notes
`bash script.sh` or `sh script.sh`	the interpreter reads the file. The file only needs read permission
`./script.sh`	needs the execute bit: `chmod +x script.sh`
`script.sh`	works only if the script is in a directory listed in `PATH`
`source script.sh` or `. script.sh`	runs it in the current shell, not in a child
`exec ./script.sh`	replaces the current shell with the script. When it ends, the shell is gone. `exec -c` runs it in a clean, empty environment

The ./ is needed because Linux does not search the current directory for commands. Normally a script runs in a sub-shell (a child process), so it cannot change your current shell's variables. When it ends, you get your prompt back and its exit status is in $?. source and exec are the two exceptions above.

$ bash script.sh
Cheers from the script file! Current time is:
10:57
$ chmod +x myscript.sh
$ ./myscript.sh
$ exec ./longtask.sh          # this shell becomes longtask.sh; it does not come back

Inside a script, exec is often used to send all later output to a log file:

exec >> /var/log/myscript.log 2>&1

Permissions, ownership and SUID¶

For other users to run the script with bash script.sh, they need read permission: chmod o+r script.sh. To run it as ./script.sh, they need the execute bit.
A script can be given the SUID bit so normal users run it with its owner's (root's) rights. Then it is critical that only root can write to the file. Otherwise any user could edit it and run anything as root. Still, avoid SUID on shell scripts when you can: shell scripts are easy to trick, so most systems ignore SUID on them anyway. When a job genuinely needs extra rights, give it a narrow sudo rule instead.
Keep personal scripts in ~/bin and system-wide ones in /usr/local/bin, both of which are normally on PATH. A script should be owned sensibly and be executable:

$ ls -l /usr/local/bin/backup.sh
-rwxr-xr-x 1 root root 412 Sep  6 10:20 /usr/local/bin/backup.sh

Variables and parameters¶

Variables work as in 105.1: SOLUTION=42, no spaces around =, and quotes for values with spaces. Names cannot start with a digit, and uppercase names are the custom in scripts.

#!/bin/bash
NAME="Carol the scripting student"   # NAME=VALUE, quote values with spaces
echo "$NAME is learning!"

#!/bin/bash
NAME="Nagato the geeking guy"
echo "$NAME is learning! Wowww..."
echo "first argument: $1"
echo "there are $# arguments"

Parameters are special variables Bash fills in for the script:

Parameter	Holds
`$0`	the script's name
`$1`, `$2` ...	the first, second ... argument. From 10 on, use braces: `${10}`
`$#`	the number of arguments
`$*`	all the arguments
`$@`	all the arguments. In `"$@"`, each one is quoted separately
`$?`	the exit status of the last command
`$$`	the PID of the current shell
`$!`	the PID of the last program started in the background

${#VAR} gives the length of a variable:

$ OS=$(uname -o)
$ echo $OS
GNU/Linux
$ echo ${#OS}
9

Command substitution¶

Command substitution puts a command's output where you need it, for example into a variable. There are two forms, and they do the same thing:

$ OS=$(uname -o)
$ OS=`uname -o`

The $( ) form is easier to read and can be nested. A real-world case, naming a backup after the current date and time:

$ date +'%Y%m%d-%H%M'
20230408-1604
$ touch backup_`date +'%Y%m%d-%H%M'`.tar
$ touch backup_$(date +'%Y%m%d-%H%M').tar
$ FILES=$(ls -1)                          # $FILES now holds the file list
$ ls
backup_20230408-1604.tar

read: asking the user¶

read waits for the user to type a line and stores it in a variable. Without a variable name, it uses REPLY. With several names, each word goes into the next variable, and any extra words go into the last one:

#!/bin/bash
echo "what is your name?"
read NAME
echo "Hello $NAME"
read -p "Type your first name and last name: " FIRST LAST

Option	Does
`-p "text"`	shows a prompt, so you do not need a separate `echo`
`-t 10`	gives up after 10 seconds. `read` then fails, which `if` can test

if read -t 10 -p "Server address? " SERVER
then
    echo "Connecting to $SERVER ..."
else
    echo
    echo "Too late!"
fi

Arithmetic and arrays¶

Variables hold text. VAR=1 then VAR=$VAR+1 gives the text 1+1, not 2. To calculate with integers, use one of these:

Form	Example
`$(( ))`	`SUM=$(( $VAL1 + $VAL2 ))`
`expr`	SUM=`expr $VAL1 + $VAL2`, `expr 5 + 3`, `expr 5 \* 4` (escape the `*`)
`let`	`let VAR=VAR-1`

$(( )) also does % (remainder) and ** (power): $(( 1024**2 )) is 1048576.

Bash has one-dimensional arrays. Indexes start at 0, and you read elements with braces and square brackets:

$ SIZES=( 1048576 1073741824 )
$ echo ${SIZES[0]}
1048576
$ echo ${SIZES[1]}
1073741824
$ echo ${#SIZES[@]}
2

Form	Means
`declare -a SIZES`	declare an empty array
`SIZES=( a b c )`	create one from a list
`SIZES[0]=value`	set one element (no braces)
`${SIZES[0]}`	read one element
`${#SIZES[@]}` or `${#SIZES[*]}`	the number of elements
`FS=( $(cut -f 2 < /proc/filesystems) )`	fill an array from a command's output

Words become elements wherever there is a space, tab or newline. Those separators come from the IFS variable. IFS=$'\n' makes newlines the only separator.

Output: echo and printf¶

Command	Notes
`echo text`	adds a newline at the end
`echo -n text`	no newline
`echo -e "a\tb\n"`	understands escapes like `\t` (tab) and `\n` (newline). Keep the quotes
`printf "format" args`	fills placeholders: `%s` for text, `%d` for whole numbers. Adds no newline, so end the format with `\n`

OS=$(uname -o)
FREE=$(( 1000 * `sed -nre '2s/[^[:digit:]]//gp' < /proc/meminfo` ))
echo -e "Operating system:\t$OS"
echo -e "Unallocated RAM:\t$(( $FREE / 1024**2 )) MB"
printf "Operating system:\t%s\nUnallocated RAM:\t%d MB\n" $OS $(( $FREE / 1024**2 ))

Both forms print:

Operating system:   GNU/Linux
Unallocated RAM:    1491 MB

Conditions: test and if¶

if runs commands only when a command succeeds (exit status 0). The command it checks is usually test, written either as test expression (test condition) or, more often, as [ expression ]. The spaces inside the brackets are required.

if [ condition ]
then
    commands
else
    other commands
fi

else is optional, and fi (if backwards) closes the block. Putting then on the same line needs a ;:

if [ -x /bin/bash ] ; then
    echo "Confirmed: /bin/bash is executable."
else
    echo "No, /bin/bash is not executable."
fi

You can watch test on its own through $?:

$ test -d /etc
$ echo $?
0
$ [ -d /etc ]
$ echo $?
0

Testing files¶

Test	True if the path
`-e "$F"`	exists
`-f "$F"`	is a regular file
`-d "$F"`	is a directory
`-s "$F"`	exists and is not empty
`-r`, `-w`, `-x`	is readable, writable, executable by you
`-h` or `-L "$F"`	is a symbolic link
`-b`, `-c`	is a block or character device
`-p`, `-S`	is a pipe or a socket
`-u`, `-g`, `-k`	has SUID, SGID, the sticky bit
`-O`, `-G`	is owned by you, by your group
`-N "$F"`	was modified since it was last read
`"$A" -nt "$B"`	A is newer than B
`"$A" -ot "$B"`	A is older than B
`"$A" -ef "$B"`	A and B are hard links to the same file

Put double quotes around variables in tests. If the variable is empty and unquoted, test is missing an argument and fails with a syntax error.

Testing text and numbers¶

Test	True if
`-z "$T"`	T is empty
`-n "$T"`	T is not empty
`"$A" = "$B"` (or `==`)	the texts are equal
`"$A" != "$B"`	the texts differ
`$X -eq $Y`	equal
`$X -ne $Y`	not equal
`$X -lt $Y`, `-le`	less than, less or equal
`$X -gt $Y`, `-ge`	greater than, greater or equal

With real values:

Test	True when
`"a" = "b"`	the two strings are equal
`"a" != "b"`	the strings are not equal
`4 -lt 40`	first number is lower than the second
`5 -gt 15`	first number is greater than the second
`5 -ge 5`	greater than or equal
`5 -le 3`	lower than or equal
`9 -ne 2`	the numbers are not equal
`-f FILE`	FILE exists and is a regular file
`-s FILE`	FILE exists and is larger than zero bytes
`-x FILE`	FILE exists and is executable

= compares text, -eq compares numbers. Combine tests with ! (not), -a (and) and -o (or):

if [ ! -d "$FROM" -o ! -d "$TO" ]

A small example:

#!/bin/bash
kernel=$(uname -s)
if [ $kernel = "Linux" ]
then
    echo YES. You are using a Linux
else
    echo "Not a linux :("
fi

The same test with quotes, which is safer when a variable may be empty:

if [ "$kernel" = "Linux" ]
then
    echo "YES. You are using Linux"
else
    echo "Not a Linux :("
fi

case¶

case compares one value against several patterns, which is tidier than a long chain of if. Each pattern list ends with ), | separates alternatives, each block ends with ;;, and *) catches everything else. esac (case backwards) closes it:

#!/bin/bash
DISTRO=$1
echo -n "Distribution $DISTRO uses "
case "$DISTRO" in
    debian | ubuntu | mint)
        echo -n "the DEB"
        ;;
    centos | fedora | opensuse)
        echo -n "the RPM"
        ;;
    *)
        echo -n "an unknown"
        ;;
esac
echo " package format."

$ ./script.sh opensuse
Distribution opensuse uses the RPM package format.

shopt -s nocasematch before the case makes the matching ignore upper and lower case.

Loops¶

for¶

for runs the same commands once for each item in a list. The variable takes the next item each time:

for NUM in 1 1 2 3 5 8 13
do
    echo -n "$NUM is "
    if [ $(( $NUM % 2 )) -ne 0 ]
    then
        echo "odd."
    else
        echo "even."
    fi
done

1 is odd.
1 is odd.
2 is even.
3 is odd.
5 is odd.
8 is even.
13 is odd.

The list can be anything separated by spaces: numbers, words, file names. For a range of numbers, use seq:

You write	Gives
`seq 1 42` or `{1..42}`	1 to 42
`$(ls)`	the files in the current directory

for NUM in 1 2 3 4 5 6
do
    echo $NUM
done

for FILE in $(ls)
do
    echo $FILE
    wc -l $FILE
done

Bash also has a C-style for, handy with arrays:

SEQ=( 1 1 2 3 5 8 13 )
for (( IDX = 0; IDX < ${#SEQ[*]}; IDX++ ))
do
    echo "${SEQ[$IDX]}"
done

while and until¶

while repeats as long as its test succeeds. until repeats until its test succeeds:

VAR=52
while [ $VAR -gt 42 ]
do
    echo VAR is $VAR and it is still greater than 42
    let VAR=VAR-1
done

IDX=0
until [ $IDX -eq 7 ]
do
    echo $IDX
    IDX=$(( $IDX + 1 ))
done

So until [ $IDX -eq 7 ] does the same as while [ $IDX -lt 7 ]. Something inside the loop must change the condition, like the counter here. If the condition never changes, the loop runs forever: stop it with Ctrl+C.

Mailing the superuser¶

Scripts often report to the administrator by mail. With a mail system installed (the mail command comes in packages like mailutils), mail root sends a message to root. Interactively, it asks for the details:

$ mail root
Cc:
Subject: Hi there root
hello there. This is my mail

End the body with Ctrl+D. In a script, pipe the body in and give the subject with -s:

$ echo "Body!" | mail -s "Subject" root

Conditional mailing means mailing only when something goes wrong, using the exit status:

#!/bin/bash
tar czf /backups/home.tgz /home || echo "The backup failed on $(hostname)" | mail -s "Backup FAILED" root

Here mail runs only if tar fails. Another check, for a backup disk:

if ! df -h | grep -q '/backup'
then
    echo "Backup disk is not mounted" | mail -s "Backup alert" root
fi

A complete example¶

This script copies a list of files and folders to a backup disk. The list is kept in ~/.sync.list, one item per line:

#!/bin/bash

set -ef

# List of items to sync
FILE=~/.sync.list

# Origin directory
FROM=$1

# Destination directory
TO=$2

# Check if both directories are valid
if [ ! -d "$FROM" -o ! -d "$TO" ]
then
    echo Usage:
    echo "$0 <SOURCEDIR> <DESTDIR>"
    exit 1
fi

# Create array from file
mapfile -t LIST < $FILE

# Sync items
for (( IDX = 0; IDX < ${#LIST[*]}; IDX++ ))
do
    echo -e "$FROM/${LIST[$IDX]} → $TO/${LIST[$IDX]}";
    rsync -qa --delete "$FROM/${LIST[$IDX]}" "$TO";
done

Line by line:

set -ef: -e stops the script at the first failing command, -f turns off file name globbing. Both make surprises less likely.
$1 and $2 are the source and destination. If either is not a directory, the script prints how to use it and ends with exit 1.
mapfile -t LIST < $FILE reads the file into the array LIST, one line per element. -t removes the newline from each line. This keeps names with spaces, like Family Album, in one piece.
The loop prints each item (→ is an arrow, which needs echo -e) and copies it with rsync.

$ sync.sh /home/carol /media/carol/backup
/home/carol/Documents → /media/carol/backup/Documents
/home/carol/"To do" → /media/carol/backup/"To do"
/home/carol/Work → /media/carol/backup/Work
/home/carol/"Family Album" → /media/carol/backup/"Family Album"
/home/carol/.config → /media/carol/backup/.config

Summary¶

I build a script from the pieces in order. I chain commands with ; to always run the next one, && to run it only after success, and || to run it only after failure. All of these, plus if and the loops, read the exit status, where 0 means success; $? shows the last one and exit sets my own script's status. A script starts with a shebang like #!/bin/bash naming its interpreter, and every other # line is a comment. I run it with bash script.sh, or make it executable with chmod +x and run ./script.sh (or just its name if it is in PATH), keeping personal scripts in ~/bin and shared ones in /usr/local/bin. It runs in a sub-shell, while source runs it in my shell and exec replaces my shell with it. I avoid the SUID bit on scripts in favour of a narrow sudo rule, and if a script does get SUID, only root may be able to write to it.

Inside a script, $1, $2 and so on are the arguments, $# counts them, $@ and $* list them and $0 is the script name. $(command) or backticks capture output, read (with -p and -t) asks the user, $(( )), expr and let do integer math, and arrays use ${NAME[i]} with indexes from 0. I print with echo (-n, -e) or printf with %s and %d.

test, usually written [ ] with spaces, checks files (-e, -f, -d, -s, -x, -nt), text (=, !=, -z, -n) and numbers (-eq, -ne, -lt, -gt, -le, -ge), combined with !, -a and -o, and I quote variables in tests. if ... then ... else ... fi and case ... esac choose what to run. for walks a list, often made with seq or {1..42}; while loops while a test succeeds (often counting with let) and until loops until it does. Finally, a common admin trick is to report to the superuser with mail root or mail -s "subject" root, placed after || so the mail is sent only when something fails.