103.1 Work on the command line

Weight: 4

Candidates should be able to interact with shells and commands using the command line. The objective assumes the Bash shell.

Objectives

• Use single shell commands and one-line command sequences to perform basic tasks on the command line.
• Use and modify the shell environment including defining, referencing, and exporting environment variables.
• Use and edit command history.
• Invoke commands inside and outside the defined path.

Terms

bash, echo, env, export, pwd, set, unset, type, which, man, uname, history, .bash_history, Quoting

Shells and Bash

You issue your commands in a shell. The shell is your command line interface, and you have various options for it.

To reach your shell you either log in to the system in text mode, or run one of the various Terminal Emulators in your GUI. Some samples are gnome-terminal, konsole, xterm.

After running the Terminal Emulator or logging into text mode, you are in the shell and you can issue commands. bash (GNU Bourne Again shell) is the most common one, but you might use zsh, dash, ksh, csh, and others.

You can check where your general sh command links to:

$ readlink /bin/sh

Or check your $SHELL variable:

echo $SHELL

Real world use for readlink /bin/sh: on Debian and Ubuntu, /bin/sh points to dash, not bash. A script that starts with #!/bin/sh but uses bash-only syntax will fail on those systems and work fine on others. When a script breaks only on Ubuntu, this is the first thing to check.

Your bash has some internal commands that it understands without any external dependency (say cd, break, exec). If it does not understand something internally, it will try to run it as an external executable.

You can use the type command to determine this:

[nagato@fedora ~]$ type cd
cd is a shell builtin
[nagato@fedora ~]$ type ls
ls is aliased to `ls --color=auto'
[nagato@fedora ~]$ type ping
ping is /usr/bin/ping

How bash decides what to run:

you type a command
        |
        v
  is it an alias?  ---- yes ---> expand the alias, start again
        | no
        v
  is it a builtin? ---- yes ---> bash runs it itself, no file needed
        | no                     (cd, break, exec, kill, ...)
        v
  search each directory in $PATH, left to right
        |
        +--- found ---> run that file
        |
        +--- not found ---> "command not found"

A fuller type example, querying several commands at once:

$ type uname cp kill which
uname is hashed (/bin/uname)
cp is /bin/cp
kill is a shell builtin
which is /usr/bin/which

Breaking this down:

• cp is /bin/cp = a normal program, a real file on disk.
• kill is a shell builtin = part of bash itself, no file involved.
• which is /usr/bin/which = another normal program.
• uname is hashed (/bin/uname) = uname is a normal program too, but it was run recently, so bash saved its location in a hash table to find it faster next time. After a reboot, type uname would describe it as a plain binary again.

hash -d clears the hash table.

cd, pwd & uname

cd

cd changes directory. This includes . (current directory) and .. (parent directory).

You can point to directories in two ways:

1. Absolute Paths: like /home/nagato/lpic1/lesson3.1
2. Relative Paths: like lpic1/lesson3.1. Here we are not adding the / at the beginning, so bash will try to find the lpic1 directory where we are now.

The ~ character means home directory of the user issuing the command.

It is also possible to issue cd without any parameters. It will move you to your home directory. So these 3 commands are all equal:

cd
cd ~
cd $HOME

pwd

Shows your current directory:

[nagato@fedora lesson3.1]$ pwd
/home/nagato/lpic1/lesson3.1

A practical follow-up: if you create a file without naming a location, it lands in whatever pwd reports:

$ pwd
/home/frank
$ touch newfile
$ ls
newfile

uname

Gives you data about the system. Common switches are:

Option	Description
-s	Print the kernel name. This is the default if no option is specified.
-n	Print the nodename or hostname.
-r	Print the release of the kernel. This option is often used with module-handling commands.
-v	Print the version of the kernel.
-m	Print the machine's hardware (CPU) name.
-o	Print the operating system name.
-a	Print all of the above information.

Example:

[nagato@fedora lesson3.1]$ uname -a
Linux fedora 5.14.0-60.fc35.aarch64 #1 SMP Mon Aug 30 16:30:42 UTC 2021 aarch64 aarch64 aarch64 GNU/Linux

The same option on a different machine:

$ uname -a
Linux base 4.18.0-18-generic #19~18.04.1-Ubuntu SMP Fri Apr 5 10:22:13 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Reading that line left to right: Linux is the kernel name, base is the hostname, 4.18.0-18-generic is the kernel release, then the kernel build version string, then x86_64 for a 64-bit CPU.

Real world examples for the options that are not obvious:

• uname -r is the one you need before building or loading a kernel module. Module directories are named after the exact kernel release, as in /lib/modules/5.14.0-60.fc35.aarch64/. This is why the table says it is often used with module-handling commands.
• uname -m tells you whether to download the x86_64 or aarch64 build of a package. On the Fedora output above it reports aarch64, so an x86_64 package would not run.
• uname -n gives the hostname, useful inside a script that runs on many machines and needs to know which one it is on.

Two more options: -p for processor type and -i for hardware platform. Both are marked non-portable, so they may return unknown on some systems.

Getting Help

Most of the commands we use have a complete manual, accessible via the man command. It uses the less pager by default, and contains the documentation, switches and parameters of commands and utilities.

Make yourself familiar with man by reading the manual of the yes command:

$ man yes

Man pages are categorized in different sections (books). You can check these by reading the manual of man itself:

$ man man
$ man 5 passwd

The reason man 5 passwd exists is that the name passwd appears in more than one section. Section 1 is the passwd command that changes your password. Section 5 is the /etc/passwd file format. Without the number, you get section 1 and may never find the file documentation you were actually looking for.

The standard sections are 1 user commands, 2 system calls, 3 library calls, 4 special files including devices, 5 file formats and config files, 6 games, 7 miscellaneous, 8 system administration commands.

Here is what a real man page looks like, using man uname. The top of it:

$ man uname
UNAME(1) User Commands UNAME(1)
NAME
  uname - print system information
SYNOPSIS
  uname [OPTION]...
DESCRIPTION
  Print certain system information. With no OPTION, same as -s.
  -a, --all
  print all information, in the following order, except omit -p
  and -i if unknown:
  -s, --kernel-name
  print the kernel name

Note the UNAME(1) in the header. That (1) is the section number, so this page is the user command, not a file format.

There is also apropos. man only works when you already know the exact command name. When you do not, apropos searches through man page names and descriptions for a keyword:

$ apropos kernel
systemd-udevd-kernel.socket (8) - Device event managing daemon
uname (2) - get name and information about current kernel
urandom (4) - kernel random number source devices

Real world use: you remember there is a command that shows block devices but not what it is called. Running apropos block surfaces lsblk in the results, and from there man lsblk gives you the options.

One more, for builtins. Since builtins are part of bash and not separate files, some have no man page of their own. The help command documents those instead.

Special characters and Quoting/Escaping

In the computer world, some characters have special meanings. For example in bash, the * character will expand to all files. In these cases, if you want to use this character without this expansion, you have to Quote it or Escape it. In many cases this is done by adding a \ character before it:

$ echo 2 \* 3 = 6
2 * 3 = 6

These are the characters with special meanings that you need to quote if you use them in your commands:

* ?[]'"\$;&()|^<>

Note that there is a space character in the list above.

Since \ has a specific meaning itself, if you want to use the back-slash without its escaping usage, you have to quote your back-slash with another back-slash, \\.

The three ways to protect a character, side by side:

  no quoting        bash reads special characters and acts on them
  $ touch my big file
        |
        +--> creates 3 files: my, big, file

  double quotes "   protects most things, but $ ` \ still work
  $ echo "$HOME"
        |
        +--> /home/nagato        the variable still expanded

  single quotes '   protects everything, nothing is expanded
  $ echo '$HOME'
        |
        +--> $HOME               printed literally

  backslash \       protects the single character right after it
  $ touch my\ big\ file
        |
        +--> creates 1 file: my big file

Here is the space problem in full. Running touch with an unquoted three-word name creates three separate files, because bash treats each space as a separator between arguments:

$ touch my big file
$ ls
my big file

That output line looks like one filename but it is three, printed side by side. Doing it correctly, with quotes:

$ touch "my big file"
$ ls
'my big file'

Now ls shows a single file, and quotes it in the output because the name contains spaces. Single quotes work for the same job:

$ rm 'my big file'

And the backslash form does the same thing one character at a time:

$ touch my\ big\ file

The difference between the two quote types matters. Single quotes preserve the literal value of every character. Double quotes preserve everything except $, a backtick, \, and in certain cases !. The proof:

$ echo "$mynewvar"
goodbye
$ echo '$mynewvar'
$mynewvar

Real world use: single quotes are what you want when passing a regular expression or a password to a command, because you do not want bash touching any of it. Double quotes are what you want around a filename variable, as in rm "$filename", because you do want $filename expanded but you also want it protected if the name contains spaces.

In addition to escaping, you can use \ to create some special characters. For example, as you cannot type a return character, you create it via \n (new line):

nagato@funlife:~$ echo -e "hello\nthere"
hello
there

Some other cases are:

Escape sequence	Function
\a	Alert (bell)
\b	Backspace
\c	Suppress trailing newline (same function as -n option)
\f	Form feed (clear the screen on a video display)
\n	New line
\r	Carriage return
\t	Horizontal tab

Note that echo needs the -e option before it will act on these sequences. Without -e, echo "hello\nthere" prints the characters \n literally.

Real world use for \t: building a quick aligned report from a script, as in echo -e "name\tsize\tdate", so the columns line up without counting spaces.

On bash you can use \ to break a command into more lines:

$ echo You know slashes! But this \
is another \
usage
You know slashes! But this is another usage

Real world use: a long rsync or docker run command with many options becomes readable when split across lines this way, and it still runs as one single command.

Characters with a special meaning to the shell, which need quoting or escaping to be used literally: & ; | * ? " ' [ ] ( ) { } $ < > # / \ ! ~ ^ and space.

Shell environment variables

Environment Variables contain some configs and information about the shell. For example, your default editor is set in the EDITOR variable. You can query the value of a shell variable like this:

[nagato@fedora ~]$ echo $EDITOR
/usr/bin/nano

It is possible to check all the env variables using the set or env command.

These are some of the most used bash environment variables:

Name	Function
USER	The name of the logged-in user
PATH	List of directories to search for commands, colon separated
EDITOR	Default editor
HISTFILE	Where bash should save its history (normally .bash_history)
HOSTNAME	System hostname
PS1	The Prompt! Play with it
UID	The numeric user id of the logged-in user
HOME	The user's home directory
PWD	The current working directory
SHELL	The name of the shell
$	The process id (or PID) of the running bash shell (or other) process
PPID	The process id of the process that started this process, that is, the id of the parent process
?	The exit code of the last command

When trying to access the value, you should add a $ to the beginning of the variable name.

$ echo $USER $UID
nagato 1000
$ echo $SHELL $HOME $PWD
/bin/bash /home/nagato /home/nagato/lpic

To define a new EV (Environment Variable), or change or delete one, we can do:

$ MYMOOD=happy
$ echo I am $MYMOOD
I am happy
$ MYMOOD="Even Happier" # space has a specific meaning
$ unset MM

Note the comment on the third line. The value has a space in it, so it needs quotes. Without them, bash would read Even as the value and try to run Happier as a command.

There is no space on either side of the equal sign during assignment. MYMOOD = happy fails, because bash would read MYMOOD as a command name.

If you want new programs starting from this shell to have access to the variable you defined, you have to set them with export, or export them later.

$ export MYMOOD
$ export YOURMOOD="Not Confused"

This shows exactly why export is needed, by opening a child shell and looking for the variable:

$ myvar=hello
$ echo $myvar
hello
$ bash
$ echo $myvar
$

Typing bash opened a new shell inside the first one. The variable is gone there, because a plain assignment stays local to the shell that made it. Now the same test after exporting:

$ exit
$ export myvar
$ bash
$ echo $myvar
hello

What is happening:

   parent shell
   myvar=hello              (no export)
        |
        | type "bash", a child shell starts
        v
   child shell
   echo $myvar  --->  empty, the variable did not travel


   parent shell
   myvar=hello
   export myvar             (now marked for export)
        |
        | type "bash"
        v
   child shell
   echo $myvar  --->  hello, the variable travelled down

Variables travel down to children, never back up to parents. A script cannot change its parent shell's variables, which is why cd inside a script does not move the shell you ran it from.

To remove a variable, use unset, without the $:

$ unset myvar
$ echo $myvar
$

set and env are not interchangeable. set outputs all variables and functions, including local ones. env shows only exported environment variables. The proof:

$ mynewvar=goodbye
$ echo $mynewvar
goodbye
$ env | grep mynewvar
$ set | grep mynewvar
mynewvar=goodbye

env returned nothing, because mynewvar was never exported. set found it. Both show PATH, because PATH is exported:

$ set | grep PATH
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
[...]

Sample env output, showing the kind of thing that lives there:

$ env
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
XDG_RUNTIME_DIR=/run/user/1000
XAUTHORITY=/run/user/1000/gdm/Xauthority
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/etc/xdg
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
GJS_DEBUG_TOPICS=JS ERROR;JS LOG
[...]

Global bash configs are stored at /etc/profile, and each user has their own config at ~/.profile, ~/.bash_profile and ~/.bash_logout. If you need a permanent change, add your configs to these. Everything set at the command line disappears when the shell closes.

Path

When you issue a command, bash will run it if it is an internal bash command. Otherwise, bash will go and check the PATH variable's directories one by one and try to find it there. If not found, it gives you an error. If you want to run something on a specific path, you have to explicitly describe the location:

$ echo $PATH
/home/nagato/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games

The directories are separated by colons, and searched left to right. The first match wins.

But what happens if I try to run tar? Let's check with the which, type and whereis commands:

nagato@funlife:~$ which tar
/bin/tar
nagato@funlife:~$ type tar
tar is /bin/tar
nagato@funlife:~$ whereis tar
tar: /usr/lib/tar /bin/tar /usr/include/tar.h /usr/share/man/man1/tar.1.gz

The three commands answer three different questions. which gives the one file that would actually run. type says what kind of thing it is. whereis finds the binary, the source, and the man page, so it returns more than just the executable.

A cooler example is ping on Fedora:

[nagato@fedora ~]$ whereis ping
ping: /usr/bin/ping /usr/sbin/ping /usr/share/man/man8/ping.8.gz
[nagato@fedora ~]$ which ping
/usr/bin/ping
[nagato@fedora ~]$ /usr/sbin/ping 4.2.2.4
PING 4.2.2.4 (4.2.2.4) 56(84) bytes of data.
64 bytes from 4.2.2.4: icmp_seq=1 ttl=50 time=160 ms

This is the interesting case. There are two ping binaries on this system, one in /usr/bin and one in /usr/sbin. whereis found both. which reported only /usr/bin/ping, because that directory comes first in PATH. To run the other one, type its full path.

That is why, when you want to say "run this_program in this directory", you issue ./this_program. You are explicitly telling bash where the file is. In Linux, the current directory . is not part of PATH by default.

Why . is left out on purpose: if . were in your PATH, someone could drop a malicious file named ls into a shared directory. The moment you ran ls while standing in that directory, you would run their file instead of the real one.

This is what a broken PATH feels like. Running unset PATH erases it, and then almost nothing works by name:

$ unset PATH
$ sudo cat /etc/shadow

That fails, because sudo itself is a binary at /usr/bin/sudo and bash no longer knows where to look. It only works if you give every full path yourself:

$ /usr/bin/sudo /bin/cat /etc/shadow

Now the useful direction, adding a directory to PATH for the current session:

$ touch myfiles/myscript.sh
$ echo '#!/bin/bash' >> myfiles/myscript.sh
$ echo 'echo Hello' >> myfiles/myscript.sh
$ chmod +x myfiles/myscript.sh
$ myscript.sh
Hello

This works after running export PATH="/home/yourname/myfiles:$PATH". Note the shape of that value: the new directory, then a colon, then $PATH itself. That keeps everything already in the path and puts the new directory in front. This does not survive a reboot, so for a permanent change it goes in ~/.profile.

Command history

Bash saves its history in ~/.bash_history. You can cat it and see its contents, or run the history command. You can also use the below key combinations to access your previous commands:

Key (Combination)	Usage
Up and Down Arrow	Move in the history
Ctrl+R	Backward Search
Ctrl+O	Run the command you found with Ctrl+R
!!	Run the last command
!10	Run command number 10
!text	search backward for text, and run the first found command

If you want to clear your history, issue HISTSIZE=0.

Real world use for Ctrl+R: you ran a long ssh command with a key path and a port number two days ago. Press Ctrl+R, type a few letters of the hostname, and bash pulls the whole line back. Faster than pressing the up arrow forty times.

Real world use for !!: you run a command, it fails because you are not root. sudo !! reruns the exact same line with sudo in front, without retyping it.

To search history before you run anything, pipe history to grep:

$ history | grep bash_history
1605 sudo find /home -name ".bash_history" | xargs grep sudo

The 1605 at the start is the command's sequence number. That number is what you would use with !1605 to run it again.

.bash_history is a hidden file in your home directory. The dot at the start of the name is what makes it hidden, so plain ls will not show it. You need ls -a:

$ ls /home/frank
newfile
$ ls -a /home/frank
. .. .bash_history .bash_logout .bashrc .profile .ssh newfile

One detail that matters and is easy to miss: your most recent commands are often not in .bash_history yet. They go into a live history in memory first, and are only written to the file when you exit the session. So the file and the history command can disagree.

   you type a command
        |
        v
   history in memory        <--- "history" command reads this
        |
        | only when the session exits
        v
   ~/.bash_history          <--- "cat .bash_history" reads this

Comparing the two directly:

$ history 20
$ tail -n 20 .bash_history

history 20 limits the output to the last 20 entries instead of printing everything.

Exiting the Shell

The exit command exits the shell. Same as CTRL+d.

If you run a command inside parentheses, that command will be run inside a sub-shell. exec will run a command and close the current shell.

exec does not close the shell and then run the command. It replaces the shell process with the command, reusing the same PID. The shell is gone because the command took its place, not because it was shut down first. The visible result is the same, your session ends when that command finishes.

Real world use for a sub-shell: running (cd /tmp && ls) lists /tmp but leaves you standing exactly where you were. The cd happened inside the sub-shell and died with it. Without the parentheses, you would end up in /tmp.

Summary

I have a Linux system where every command I type goes into a shell, and on my machines that shell is bash. When I type something, bash first checks whether it is an alias, then whether it is one of its own builtins like cd or exec, and only then goes looking through the directories listed in PATH. I can ask which of those three a command is with type, get just its location with which, or get its binary, source and man page together with whereis. When I do not know a command's name at all, apropos searches the man page descriptions for me, and once I have the name, man gives me the full documentation, with a section number when the same name means two different things.

I can find out where I am with pwd, and what system I am on with uname. I mostly use uname -a for everything at once, but uname -r is the one that matters when I am dealing with kernel modules, since the module directories are named after the exact kernel release.

My shell carries a set of environment variables holding things like PATH, HOME, EDITOR and SHELL. I read one with echo and a $ in front of the name, list them all with env or set, and delete one with unset. A plain assignment only lives in the shell I made it in, so if I want a child process to see it I have to export it. Variables travel down to children and never back up to parents. Nothing I set at the prompt survives closing the shell, so anything permanent goes into ~/.profile or /etc/profile.

Bash also remembers what I have typed. The history command reads a live list in memory, while ~/.bash_history is the file on disk, and the file only catches up when the session ends, so the two can disagree. I pull old commands back with the up arrow, search them with Ctrl+R, or rerun the last one with !!. The last thing I have to stay careful about is quoting. Characters like *, $ and even a plain space mean something to bash, so I wrap things in single quotes when I want them completely untouched, double quotes when I still want variables expanded, and a backslash when it is just one character in the way.