Configuring and Fixing the GRUB Bootloader: From Kernel Parameters to the Rescue Shell

1. GRUB's role in the boot process

GRUB, short for Grand Unified Bootloader, is the piece of software that runs between firmware and kernel on both classic BIOS systems and UEFI systems. After the power on self test, the firmware either loads the Master Boot Record with the GRUB stage 1 code, or, on UEFI, loads the file grubx64.efi from the EFI system partition directly, handing control over to GRUB.

GRUB then loads its own configuration, optionally displays a menu with several kernel versions and boot options, and finally hands control to the selected kernel together with its initramfs. On a server without a physical monitor this step usually happens invisibly, but it remains just as critical: a broken GRUB setup blocks every boot, no matter how healthy the kernel and root filesystem themselves are.

A key distinction for understanding GRUB is the difference between the configuration source and the file actually read at boot time. Administrators almost never edit grub.cfg directly, but rather the templates in /etc/default/grub and /etc/grub.d, from which grub2-mkconfig generates the actual configuration file.

2. /etc/default/grub: the central configuration file

The file /etc/default/grub contains shell variables that are read by the scripts in /etc/grub.d. The most important ones are GRUB_DEFAULT for the default entry, GRUB_TIMEOUT for the menu wait time, GRUB_CMDLINE_LINUX_DEFAULT for kernel parameters in normal operation, and GRUB_CMDLINE_LINUX for parameters that should also apply in recovery mode.

On production servers a low GRUB_TIMEOUT makes sense since the menu is rarely operated manually anyway, while GRUB_TIMEOUT_STYLE=hidden only shows the menu when shift is held during boot, keeping the boot sequence visually clean. GRUB_DISABLE_OS_PROBER=true stops GRUB from scanning other partitions for foreign operating systems on every update, which on pure Linux servers only costs time and, in rare cases, creates incorrect entries.

# /etc/default/grub, a typical server configuration
GRUB_DEFAULT=0
GRUB_TIMEOUT=3
GRUB_TIMEOUT_STYLE=menu
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release 2>/dev/null || echo Debian)"
GRUB_CMDLINE_LINUX_DEFAULT="quiet"
GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0"
GRUB_DISABLE_OS_PROBER=true

3. Setting kernel parameters permanently through the bootloader

Kernel parameters meant to apply only to the current boot can be edited directly in the GRUB menu with the e key, but disappear again after a reboot. For permanent changes, such as cgroup parameters on container hosts, transparent_hugepage=never on database servers, or a specific IOMMU setting for PCI passthrough, the change belongs in GRUB_CMDLINE_LINUX_DEFAULT or GRUB_CMDLINE_LINUX.

After every change to /etc/default/grub, the configuration has to be regenerated before it takes effect on the next boot. Debian and Ubuntu use the wrapper update-grub, while RHEL based systems use grub2-mkconfig with an explicit target path. Forgetting this step is a common source of confusion when new parameters do not seem to apply despite a correct file.

# Debian/Ubuntu: regenerate the configuration
sudo update-grub

# RHEL/Alma/Rocky, BIOS system
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

# RHEL/Alma/Rocky, UEFI system
sudo grub2-mkconfig -o /boot/efi/EFI/almalinux/grub.cfg

# Check the active kernel parameters of the running system
cat /proc/cmdline

4. Customizing the boot menu: order, timeout, and hidden entries

The order of menu entries comes from the numerically sorted scripts in /etc/grub.d. The script 10_linux generates entries for installed kernels, and by default the newest kernel appears first and is preselected. With GRUB_DEFAULT=saved combined with grub-set-default, a specific entry can instead be pinned as the permanent default, independent of newly installed kernel versions.

For servers, a tidy menu matters more than visual extras: GRUB_TIMEOUT_STYLE=countdown shows only a countdown instead of the full menu, and GRUB_TERMINAL=console skips graphical elements, reducing the dependency on a working framebuffer configuration, which particularly helps with serial consoles and remote management through IPMI or iLO.

5. Managing multi kernel environments cleanly

On production systems, the previous kernel usually stays installed after an update so a rollback through the Advanced Options submenu in the GRUB menu remains possible in case of trouble. Debian and RHEL based distributions limit how many kernel versions get retained through the respective package manager, for example installonly_limit in dnf.conf on RHEL systems.

Anyone who deliberately wants to pin an older kernel as the default, for instance because a newer kernel causes driver problems with a RAID controller, can find the matching menu entry with grep against grub.cfg and then pin it permanently with grub-set-default, instead of selecting it manually in the menu on every boot.

# List available menu entries, including Advanced Options
grep -E "^menuentry|^submenu" /boot/grub2/grub.cfg

# Pin a specific entry as the permanent default (index or title string)
sudo grub2-set-default "Advanced options for AlmaLinux (10.0.99-original.el9.x86_64)"

# Check the currently saved default entry
sudo grub2-editenv list

6. Custom menu entries through 40_custom

Manual entries, for example a rescue image, a memory test, or an alternative kernel boot with special parameters for troubleshooting, belong in /etc/grub.d/40_custom or a dedicated executable file in /etc/grub.d. Direct edits to grub.cfg, on the other hand, get silently overwritten on every run of grub2-mkconfig and are lost.

A custom script in /etc/grub.d should start with a three digit number to control its position in the menu order, and must be executable, otherwise grub2-mkconfig skips it silently. This is a common source of confusion when a new entry does not show up in the menu after editing.

# /etc/grub.d/45_rescue_debug, a custom entry with an extra debug parameter
cat <<'EOF' | sudo tee /etc/grub.d/45_rescue_debug
#!/bin/sh
exec tail -n +3 $0
menuentry 'Kernel Debug Boot (systemd.log_level=debug)' {
    insmod gzio
    insmod part_gpt
    insmod xfs
    set root='hd0,gpt2'
    linux /vmlinuz root=/dev/mapper/vg-root ro systemd.log_level=debug
    initrd /initramfs.img
}
EOF
sudo chmod +x /etc/grub.d/45_rescue_debug
sudo grub2-mkconfig -o /boot/grub2/grub.cfg

7. Recovery through the GRUB rescue shell

If the server ends up at a prompt reading grub> or grub rescue> instead of the menu, GRUB failed to load its configuration or important modules, usually because the reference to the boot partition is wrong or core.img got corrupted. In this minimal shell, ls reveals the available partition layout, and a manual set root followed by linux and initrd commands can still force a one time boot.

This manual boot is only a bridge for the current session and does not survive a reboot. What it does provide is enough time to log in to the running system and permanently repair the actual GRUB installation with grub2-install and grub2-mkconfig, instead of having to intervene manually on every boot.

# In the grub rescue shell: list available devices and partitions
grub rescue> ls
(hd0) (hd0,gpt2) (hd0,gpt1)

# Check and manually set the root partition and kernel path
grub rescue> ls (hd0,gpt2)/boot
grub rescue> set root=(hd0,gpt2)
grub rescue> set prefix=(hd0,gpt2)/boot/grub2
grub rescue> insmod normal
grub rescue> normal

8. grub-install and chroot repair from a live system

If the manual rescue shell is not enough, for example because core.img or the entire EFI system partition is damaged, the most reliable path is a live system or a rescue image from the hosting provider. There, the root partition and, on UEFI systems, the EFI system partition get mounted, essential virtual filesystems get bind mounted, and a chroot switches into the actual system.

Inside the chroot, grub2-install behaves exactly as it would on the running system itself, writing the boot code either into the MBR or creating a new grubx64.efi inside the EFI system partition along with a boot entry in the NVRAM firmware table. A final grub2-mkconfig run ensures the configuration matches the kernel that is actually installed.

# From the live system: mount the root partition and system paths
sudo mount /dev/mapper/vg-root /mnt
sudo mount /dev/sda1 /mnt/boot/efi
for fs in dev proc sys run; do sudo mount --bind /$fs /mnt/$fs; done
sudo chroot /mnt /bin/bash

# Inside the chroot: reinstall the bootloader
grub2-install --target=x86_64-efi --efi-directory=/boot/efi --bootloader-id=almalinux
grub2-mkconfig -o /boot/efi/EFI/almalinux/grub.cfg
exit

9. Best practices and common pitfalls

Before changing kernel parameters or the boot menu, it is worth testing through the provider's IPMI or rescue console, since a wrong root reference or a missing kernel module immediately leaves a remote server unreachable without physical access. A snapshot or backup before major GRUB changes is the cheapest insurance on virtualized servers.

Secure Boot adds another layer of complexity to manual grub-install calls, because the resulting EFI binary must be signed, otherwise the firmware refuses to start it. On systems with Secure Boot enabled, grub2-install should run through the distribution packages shim and grub2-efi rather than manually copying an unsigned binary.

After every kernel update it is worth a quick check with grep against grub.cfg to confirm the new kernel actually shows up as an entry and picked up the intended parameters from GRUB_CMDLINE_LINUX_DEFAULT, instead of blindly trusting the package manager's automatic flow.

10. Summary

11. FAQ: GRUB Bootloader