101.2 Boot the system¶

Weight: 3

Candidates should be able to guide the system through the booting process.

Objectives

Provide common commands to the boot loader and options to the kernel at boot time
Demonstrate knowledge of the boot sequence from BIOS/UEFI to boot completion
Understanding of SysVinit and systemd
Awareness of Upstart
Check boot events in the log files

Terms

dmesg, journalctl, BIOS, UEFI, bootloader, kernel, initramfs, init, SysVinit, systemd

The boot process¶

Understand the boot process well. While the machine boots you have very little control and few commands to troubleshoot with, so you must know what happens at each step.

POST (Power-On Self-Test): the motherboard firmware checks the basic hardware (RAM, CPU).
The firmware (BIOS or UEFI) loads the bootloader.
The bootloader (GRUB) loads the Linux kernel, based on its configuration and the commands you give it.
The kernel sets up the hardware and memory, uses the initramfs to reach the root filesystem, and runs the init program.
init (systemd or SysVinit) starts the services: networking, a web server, the graphical interface and so on.

  power on
     |
     v
  firmware (BIOS / UEFI) --- POST, activates video, keyboard, storage
     |
     v
  bootloader (GRUB) -------- menu, kernel parameters
     |
     v
  kernel + initramfs ------- hardware detection, mounts the real root filesystem
     |
     v
  init (PID 1) ------------- systemd / SysVinit / Upstart start the services

BIOS boot¶

The BIOS (Basic Input/Output System) is firmware: a program stored in a non-volatile memory chip on the motherboard, separate from the disks, and run every time the computer is powered on. It is the older method. It can start a bootloader from an internal or external hard disk, a CD/DVD, a USB flash drive or a network server, in the order set in the BIOS configuration utility.

From a hard disk, the BIOS uses the MBR (Master Boot Record): the first 512 bytes of a disk with the standard DOS partition scheme.

  MBR: first 512 bytes of the disk
  +--------------------------------------+------------------------+-----------+
  | first stage of the bootloader        | partition table        | signature |
  | (bootstrap) 440 bytes                | 64 bytes               | 2 bytes   |
  +--------------------------------------+------------------------+-----------+

440 bytes is tiny, so the bootloader works in stages. If the MBR does not hold the right data, the system cannot boot (unless you use another method).

The BIOS boot steps:

POST finds simple hardware failures as soon as the machine is powered on.
The BIOS activates the basic components: video output, keyboard, storage.
The BIOS loads the first stage of the bootloader from the MBR (the first 440 bytes of the first device in the BIOS order).
The first stage calls the second stage, which shows the boot options and loads the kernel.

UEFI boot¶

UEFI (Unified Extensible Firmware Interface) is the modern firmware. Unlike the BIOS, it understands partitions and filesystems, and it does not use the MBR. It reads its settings from its NVRAM (non-volatile memory on the motherboard). These settings point to EFI applications to run automatically or from a boot menu: bootloaders, operating system selectors, diagnostic and repair tools.

EFI applications live on a special partition, the ESP (EFI System Partition):

formatted with FAT (FAT12, FAT16 or FAT32; ISO-9660 on optical media),
mounted on /boot/efi in Linux,
not shared with other filesystems like the root filesystem or user data,
the applications are .efi files in its EFI directory.

The UEFI boot steps:

POST finds simple hardware failures.
UEFI activates the basic components: video output, keyboard, storage.
UEFI reads the settings in NVRAM and runs the pre-defined EFI application from the ESP. Usually this is a bootloader.
The bootloader loads the kernel.

ESP partition (FAT, mounted at /boot/efi)
        |
        contains .efi files (EFI applications)
        |
        UEFI reads NVRAM -> knows which .efi file to run
        |
        usually: that .efi file IS the bootloader (GRUB)
        |
        bootloader loads the kernel

Secure Boot is a UEFI feature that only runs signed EFI applications, authorized by the hardware maker. It protects against malicious software, but can make it hard to install operating systems the manufacturer does not cover.

Is this system UEFI? If /sys/firmware/efi exists, yes:

$ ls /sys/firmware/efi

The bootloader: GRUB¶

The bootloader starts the minimum hardware needed, then finds and runs the operating system. You could point UEFI at any program, but on Linux we normally use GRUB (Grand Unified Bootloader), the most popular bootloader on x86.

When the BIOS or UEFI calls it, GRUB shows a list of the operating systems and kernels you can boot. If the menu does not appear, hold Shift while GRUB starts (BIOS), or press Esc (UEFI). From the menu you can choose a kernel and pass parameters to it. (GRUB configuration is covered in 102.2.)

Kernel parameters¶

Most kernel parameters look like option=value. You do not usually need them, but they help find and fix problems:

Parameter	Effect
`acpi=off`	disable ACPI support
`init=/bin/bash`	use another program as init. With `/bin/bash` you get a shell right after the kernel boots
`systemd.unit=graphical.target`	the systemd target to activate
`1` or `S`	boot into runlevel 1, single-user mode (recovery). systemd accepts SysV runlevel numbers too
`mem=512M`	limit the RAM the system can use, useful for virtual machines
`maxcpus=2`	limit the number of visible processors. `maxcpus=0` turns off multi-processor support, like `nosmp`
`quiet`	hide most boot messages
`vga=792`	choose a video mode (here 1024x768x24)
`vga=ask`	show a list of video modes to choose from
`root=/dev/sda3`	use a different root partition than the one set in the bootloader
`rootflags=...`	mount options for the root filesystem
`ro`	mount the root filesystem read-only at first
`rw`	allow writing to the root filesystem at first mount

To make parameters permanent, add them to the GRUB_CMDLINE_LINUX line in /etc/default/grub, then generate a new GRUB configuration (every time that file changes):

# grub-mkconfig -o /boot/grub/grub.cfg

The parameters used for the current boot are in /proc/cmdline:

$ cat /proc/cmdline

Kernel and initramfs¶

The bootloader loads the kernel into RAM. The kernel takes the CPU and sets up the basics: hardware configuration and memory addressing. Then it opens the initramfs (initial RAM filesystem).

The initramfs is an archive with a small temporary root filesystem. Its job is to give the kernel the modules (drivers) it needs to reach the real root filesystem, for example the driver for your disk controller.

Bootloader loads kernel
        v
Kernel loads initramfs into RAM
        v
initramfs gives kernel the drivers it needs
        v
Kernel mounts the REAL root filesystem (your disk)
        v
Kernel mounts the other filesystems in /etc/fstab
        v
Kernel runs init (PID 1)
        v
initramfs is discarded from RAM

Strictly speaking, the operating system is just the kernel and its parts that control the hardware and processes. In daily talk we use the word for the whole set of programs too.

System initialization: init¶

After the kernel, the system needs many other parts. During initialization two kinds of things start:

scripts, for short tasks that run and finish,
services, also called daemons, which keep running in the background.

Whatever tool a distribution uses, it must at least start, stop and restart services. The system does this itself after an update, and you do it after you change a service's configuration file. It should also be able to start just a minimum set of services, for maintenance.

The init program has PID 1 and starts everything else:

# which init
/sbin/init
# readlink -f /sbin/init
/usr/lib/systemd/systemd
# ps -p 1
PID TTY TIME     CMD
1   ?   00:00:06 systemd
$ pstree                                        # the process tree, with init at the top

There are three init systems to know:

Init system	Key facts
SysVinit	based on Unix System V. Uses runlevels, numbered 0 to 6. Only runlevels 0, 1 and 6 mean the same on every distribution. People liked it because it follows the Unix philosophy. Not used much now, but still found on old and even some new machines
systemd	the modern system and service manager, now the default on most major distributions. Starts services in parallel, uses sockets and D-Bus to start services, starts daemons on demand, watches processes with cgroups, and controls services by their dependencies. Has a compatibility layer for SysV commands and runlevels. Hated by Linux elitists for not following Unix principles, but widely adopted by major distros
Upstart	awareness only. Another init replacement, by Canonical (Ubuntu), released in 2007, focused on a faster boot by starting services in parallel. Used by older Ubuntu releases, now replaced by systemd. Can still be found in Google's ChromeOS

systemd¶

systemd is built around units. A unit can be a service, a group of services, or an action. Every unit has a name, a type and a configuration file. There are 12 unit types: automount, device, mount, path, scope, service, slice, snapshot, socket, swap, target and timer.

systemctl works with the units, and journalctl shows the logs.

# systemctl list-units
# systemctl list-units --type=target
# systemctl get-default                         # the default target (targets start groups of services)

Unit files are found in these places, highest priority first:

/etc/systemd/system/
/run/systemd/system/
/usr/lib/systemd/system/

# systemctl list-unit-files
# systemctl cat ntpd.service
# systemctl cat graphical.target

Working with services:

# systemctl start sshd
# systemctl stop sshd
# systemctl restart sshd
# systemctl reload sshd                         # the service re-reads its own configuration
# systemctl daemon-reload                       # re-reads the systemd configs (unit files), e.g. of sshd
# systemctl status sshd
# systemctl is-active sshd
# systemctl is-failed sshd
# systemctl enable sshd                         # start at boot
# systemctl disable sshd                        # do not start at boot
# systemctl is-system-running                   # running, degraded, maintenance, initializing, starting, stopping
# systemctl --failed                            # units that failed

SysV init scripts¶

On SysVinit systems, services are controlled by scripts in /etc/init.d/, which are close to normal Bash scripts:

# /etc/init.d/ntpd status
# /etc/init.d/ntpd stop
# /etc/init.d/ntpd start
# /etc/init.d/ntpd restart

Runlevels and targets are covered in 101.3.

Checking boot events: dmesg¶

Errors during boot do not always stop the system, but they can make it misbehave. They leave messages that tell you when and how the problem happened. Even without errors, these messages help with tuning.

Linux shows these messages while booting. Some desktops hide them behind a splash screen: press Esc to see them, or Ctrl+Alt+F1.

The kernel keeps its messages, including the boot messages, in the kernel ring buffer, a memory area. The messages are kept even when an animation hides them, but they are lost when the system is turned off. dmesg shows the ring buffer:

$ dmesg
[    5.262389] EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
[    5.460286] systemd[1]: systemd 237 running in system mode.
[    5.480138] systemd[1]: Detected architecture x86-64.
[    5.481767] systemd[1]: Set hostname to <torre>.
[    5.636866] systemd[1]: Created slice System Slice.
[    5.641661] systemd[1]: Starting Load Kernel Modules...
[    5.661672] EXT4-fs (sda1): re-mounted. Opts: errors=remount-ro
[    5.694322] lp: driver loaded but no devices found
[    5.897421] systemd-journald[352]: Received request to flush runtime journal from PID 1

The output can be hundreds of lines. The number at the start of each line is the seconds since the kernel started loading. Here you can see the kernel handing over to systemd.

$ dmesg -H                                      # human-readable, in a pager
# dmesg --clear                                 # empty the ring buffer

After the boot, the syslog daemon saves the boot messages in /var/log/dmesg. Many systems also keep a text file of the boot: /var/log/boot on Debian, /var/log/boot.log on Red Hat.

Checking boot events: journalctl¶

On systemd systems, journalctl shows the boot messages with -b (--boot) or -k (--dmesg, kernel messages only). systemd also keeps the logs of earlier boots. --list-boots lists them, numbered relative to the current boot (0):

$ journalctl --list-boots
 -4 9e5b3eb4952845208b841ad4dbefa1a6 Thu 2019-10-03 13:39:23 -03—Thu 2019-10-03 13:40:30 -03
 -3 9e3d79955535430aa43baa17758f40fa Thu 2019-10-03 13:41:15 -03—Thu 2019-10-03 14:56:19 -03
 -2 17672d8851694e6c9bb102df7355452c Thu 2019-10-03 14:56:57 -03—Thu 2019-10-03 19:27:16 -03
 -1 55c0d9439bfb4e85a20a62776d0dbb4d Thu 2019-10-03 19:27:53 -03—Fri 2019-10-04 00:28:47 -03
  0 08fbbebd9f964a74b8a02bb27b200622 Fri 2019-10-04 00:31:01 -03—Fri 2019-10-04 10:17:01 -03

$ journalctl -b 0                               # this boot (same as --boot=0)
$ journalctl -b -1                              # the previous boot
$ journalctl -b -2                              # the boot before that
$ journalctl -k                                 # kernel messages
$ journalctl -u kernel                          # all previous kernel logs too

$ journalctl -b 0
oct 04 00:31:01 ubuntu-host kernel: EXT4-fs (sda1): mounted filesystem with ordered data mode. Opts: (null)
oct 04 00:31:01 ubuntu-host systemd[1]: systemd 237 running in system mode.
oct 04 00:31:01 ubuntu-host systemd[1]: Detected architecture x86-64.
oct 04 00:31:01 ubuntu-host systemd-journald[352]: Journal started
oct 04 00:31:01 ubuntu-host systemd-modules-load[335]: Inserted module 'lp'
oct 04 00:31:01 ubuntu-host systemd[1]: Starting Flush Journal to Persistent Storage...

Other useful journalctl options:

# journalctl --no-pager                         # do not use less
# journalctl -n 10                              # only the last 10 lines
# journalctl -S -1d                             # since 1 day ago
# journalctl -xe                                # the last messages, with explanations
# journalctl -u ntp                             # only the ntp unit
# journalctl _PID=1234                          # only this process

systemd stores its logs in /var/log/journal/, and not as plain text, so you need journalctl to read them.

When the system cannot boot after loading the kernel and initramfs, start it from another boot medium (a live USB, for example), mount its filesystem and search the files in its /var/log/. journalctl -D (--directory) reads journal files from a directory other than /var/log/journal/.

/var/log/messages¶

After init comes up, the syslog daemon logs system messages to /var/log/messages (called /var/log/syslog on some systems). It has timestamps and survives reboots. The kernel still writes its own messages to the ring buffer as well, and there are many other logs in /var/log/.

Summary¶

I have a computer with a motherboard that has firmware (BIOS or UEFI) on it. When I press the power button, the firmware runs a Power-On Self Test (POST), checking the RAM and CPU, and wakes up video, keyboard and storage. If everything is okay, it starts the bootloader, which is located on the hard disk. A BIOS loads the 440-byte first stage of the bootloader from the MBR, which calls the second stage, while UEFI reads its NVRAM and runs an .efi application, usually GRUB, from the FAT-formatted ESP mounted on /boot/efi (Secure Boot allows only signed ones, and /sys/firmware/efi tells me I am on UEFI).

The bootloader's job is to load the Linux kernel based on its configuration. In the GRUB menu (Shift on BIOS, Esc on UEFI) I can pass kernel parameters such as init=/bin/bash, 1 or S for single-user mode, systemd.unit=, root=, ro, mem= or quiet, make them permanent in GRUB_CMDLINE_LINUX in /etc/default/grub followed by grub-mkconfig -o /boot/grub/grub.cfg, and read the current ones in /proc/cmdline.

The Linux kernel is the core of the operating system, but it needs additional data to operate. This data is in a file called initramfs, which contains the drivers the kernel needs to reach the real root filesystem. The kernel mounts that filesystem and the /etc/fstab entries, and runs the init program (PID 1) to start everything the system needs: SysVinit with runlevels and scripts in /etc/init.d/, systemd with units managed by systemctl, or the now-retired Upstart. So, when the system boots, the firmware loads the bootloader, the bootloader loads the kernel, the kernel gets the initramfs, and init starts the services.

To check boot events I read the kernel ring buffer with dmesg, use journalctl -b, -k and --list-boots (with -b -1 for the previous boot and -D for another directory) on systemd, and look at /var/log/dmesg, /var/log/boot.log and /var/log/messages or /var/log/syslog.