Mastodon

Linux Foundationals: Storage and Memory - Block Layer, LUKS2, Btrfs and zram



A dimly lit desk with a monitor showing a layered storage stack diagram from filesystem down to an NVMe drive, an opened laptop with a terminal, and a notebook with memory and swap sketches

At the end of part 1, systemd had done a switch_root and was running from “the real root filesystem”. That phrase hid a lot. On a default Fedora install, the root filesystem is a subvolume of a Btrfs pool, which lives on a device-mapper device, which decrypts a LUKS2 container, which sits on a partition of an NVMe drive. And the swap device that showed up in the dependency tree wasn’t a disk at all.

This is the second article in the Linux Foundationals series. It covers the two resources every process on the system competes for: storage and memory. They belong together more than it seems at first. The page cache is where the filesystem and the memory manager meet, swap is storage used as memory, and zram is memory used as storage used as memory.

The reference system is again Fedora 45, and this is the part of the series where Fedora and RHEL differ the most. RHEL has no Btrfs and installs XFS on LVM instead. It also doesn’t enable zram swap or systemd-oomd by default. The block layer, device mapper, LUKS2, systemd-cryptenroll and everything about how the kernel manages memory apply to both, so I’ll point out the differences where they matter.

By the end of this article you’ll know what happens between a read() call and the NVMe drive, how LUKS2 stores its keys and what a TPM2 enrollment does and doesn’t protect against, why df is the wrong tool on Btrfs, how to check what compression really saves you, what “free memory” means on Linux, and who decides which process dies when memory runs out.

Table of Contents

The Storage Stack at a Glance

Every layer in the storage stack presents the same simple interface to the layer above it and hides how it’s implemented. On my laptop, a read from a file in /home passes through these layers:

┌────────────────────────┐
│        process         │  read(), write(), mmap()
└───────────┬────────────┘
            │ system call
┌───────────┴────────────┐
│          VFS           │  dentries, inodes, mount table,
│      + page cache      │  cached file contents
└───────────┬────────────┘
            │ only on a cache miss
┌───────────┴────────────┐
│         Btrfs          │  subvolumes "root" and "home",
│                        │  CoW, checksums, zstd compression
└───────────┬────────────┘
            │ block I/O (bio) to /dev/mapper/luks-03ef...
┌───────────┴────────────┐
│     device mapper      │  dm-crypt target:
│        (dm-0)          │  AES-XTS, key from the LUKS2 header
└───────────┬────────────┘
            │ same bio, remapped to /dev/nvme0n1p3
┌───────────┴────────────┐
│      block layer       │  partitions, request queues (blk-mq),
│                        │  I/O scheduler
└───────────┬────────────┘
            │
┌───────────┴────────────┐
│      NVMe driver       │  hardware queues, one per CPU
└───────────┬────────────┘
            │
        the drive

I’ll go through this from the bottom up, because that’s the order in which the layers come into existence during boot: the kernel finds the drive, the initramfs unlocks the LUKS container, the root subvolume gets mounted. Memory comes after that, starting at the place where both worlds meet: the page cache.

The Block Layer

A block device is the kernel’s abstraction for anything that stores data in fixed-size, randomly addressable blocks. lsblk shows all of them as a tree:

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS
NAME               SIZE TYPE  FSTYPE      MOUNTPOINTS
zram0                8G disk  swap        [SWAP]
nvme0n1          953.9G disk
├─nvme0n1p1        600M part  vfat        /boot/efi
├─nvme0n1p2          1G part  ext4        /boot
└─nvme0n1p3      952.3G part  crypto_LUKS
  └─luks-03ef... 952.3G crypt btrfs       /home
                                          /

That’s the layout from part 1, now with types. There’s one physical disk (nvme0n1) with three partitions, one crypt device stacked on top of the third partition, and a second “disk” called zram0 that has no hardware behind it. The Btrfs filesystem on the crypt device is mounted twice, at / and at /home. All three of these facts get their own section below.

Names you can rely on

Kernel names like nvme0n1 or sda depend on the order in which drivers find devices. With more than one disk, they’re not guaranteed to be stable across boots. That’s why nothing in the boot chain uses them. The BLS entry from part 1 said root=UUID=5c88..., and /etc/fstab does the same:

UUID=5c88...  /          btrfs  subvol=root,compress=zstd:1,x-systemd.device-timeout=0  0 0
UUID=9faf...  /boot      ext4   defaults                                                1 2
UUID=943A-9AB3  /boot/efi  vfat   umask=0077,shortname=winnt                            0 2
UUID=5c88...  /home      btrfs  subvol=home,compress=zstd:1,x-systemd.device-timeout=0  0 0

udev maintains the stable names as symlinks under /dev/disk/:

ls /dev/disk/
# by-diskseq  by-id  by-label  by-partlabel  by-partuuid  by-path  by-uuid

# Filesystem UUIDs and labels of everything, read from the superblocks
lsblk -f
sudo blkid

by-uuid and by-label come from the filesystem superblock, by-partuuid and by-partlabel from the GPT, by-id from the hardware’s model and serial number, and by-path from the physical connection. For anything you write into a configuration file, use one of these, never the kernel name.

bios, queues and schedulers

Inside the kernel, every I/O request to a block device is a bio: a small structure that says “read (or write) these memory pages from (or to) this sector range on this device”. Filesystems create bios, stacking drivers like device mapper remap them, and at the bottom the block layer turns them into requests for the hardware driver.

Since kernel 5.0, there’s only one way requests reach a driver: the multi-queue block layer, blk-mq. It has per-CPU software queues that feed into one or more hardware queues. For an NVMe drive, which brings many hardware queues of its own, a request goes from the CPU that issued it almost straight to the device. Between the two sits the optional I/O scheduler:

cat /sys/block/nvme0n1/queue/scheduler
# [none] mq-deadline kyber bfq

The value in brackets is the active one, and for NVMe it’s none. That isn’t an oversight. A scheduler reorders and merges requests to make the best of a slow device, and a drive that answers in microseconds gains nothing from that except CPU overhead. For slower devices, Fedora ships a udev rule (60-block-scheduler.rules) that selects bfq for SATA and SAS disks, SD cards and eMMC, where fair queueing between processes makes a noticeable difference in desktop responsiveness.

Scheduler Idea Typical use
none No reordering, straight to the hardware queues NVMe, fast virtual disks
mq-deadline Guarantees a maximum latency per request, prefers reads SATA SSDs, servers
bfq Fair budgets per process, low latency for interactive tasks Rotating disks, SD cards, desktops
kyber Throttles queues to hit latency targets Fast devices under mixed load

The rest of /sys/block/<device>/queue/ describes what the device told the kernel about itself: rotational (0 for flash), logical_block_size, discard_granularity (non-zero if it supports TRIM) and more. To watch a device at work, iostat from the sysstat package is still the standard tool:

# Extended statistics, one line per device, every 2 seconds
iostat -x 2

Look at r_await and w_await (average time per request in milliseconds) and aqu-sz (average queue length) rather than at %util, which is meaningless for devices that process many requests in parallel.

Device Mapper

The block layer has a framework for building virtual block devices out of other block devices: device mapper. A device-mapper device is defined by a table. Each line maps a range of sectors of the virtual device to a target, a kernel module that decides what happens to bios in that range.

That’s a small idea with large consequences, because almost everything in Linux storage that isn’t a filesystem is built on it:

Target What it does Who uses it
linear Maps a range to a range on another device LVM logical volumes
striped Spreads data over several devices LVM striping
crypt Encrypts and decrypts every bio LUKS, plain dm-crypt
thin, thin-pool Thin provisioning and snapshots LVM thin pools, Stratis
raid Wraps the MD RAID personalities LVM RAID
multipath Several paths to the same SAN LUN multipathd
verity Read-only integrity checking against a hash tree Android, image-based systems
integrity Per-sector checksums LUKS2 with integrity

LVM, LUKS and multipath are not three subsystems. They’re three userspace tools (lvm, cryptsetup, multipathd) that write tables for the same kernel framework. That’s also why they stack in any order: a dm device is a block device, so it can be the backing device of another dm device.

On my laptop there’s exactly one dm device, and you can look at its table:

sudo dmsetup ls --tree
sudo dmsetup table
luks-03ef... (252:0)
 └─ (259:3)

luks-03ef...: 0 1997047808 crypt aes-xts-plain64 :64:logon:cryptsetup:03ef...-d0 0 259:3 32768 1 allow_discards

Read the table line from left to right: starting at sector 0, for 1997047808 sectors (952.3 GiB), use the crypt target with the cipher aes-xts-plain64. The key is 64 bytes long and isn’t printed, it’s referenced from the kernel keyring. The backing device is 259:3 (major and minor number of nvme0n1p3), and the encrypted data starts at sector 32768 of that partition. The first 16 MiB before that offset are the LUKS2 header. allow_discards means TRIM requests from the filesystem are passed down to the drive.

The device shows up as /dev/dm-0, and udev creates the readable name /dev/mapper/luks-03ef... as a symlink to it. With rootflags=subvol=root and this device, the kernel command line from part 1 is now completely explained.

On a RHEL system, the same commands show one linear table per logical volume (rhel-root, rhel-swap, rhel-home). The principle is identical.

LUKS2: Encryption at the Block Level

dm-crypt does the encryption, but it knows nothing about passphrases. It takes a raw key and encrypts sectors. LUKS (Linux Unified Key Setup) is the on-disk format that solves key management, and cryptsetup is the tool that reads it and sets up the dm-crypt table.

The central idea of LUKS is that your passphrase never encrypts your data. The data is encrypted with a random volume key, generated once when the container is created. The volume key is stored in the header, encrypted separately for each keyslot. Every keyslot holds its own copy of the volume key, wrapped with a key derived from one passphrase (or one token).

/dev/nvme0n1p3
┌──────────────────────────────────────────────────────────────┐
│ LUKS2 header (16 MiB)                                        │
│ ┌──────────────────────────────────────────────────────────┐ │
│ │ binary header + JSON metadata (stored twice)             │ │
│ │   keyslots: 0 → argon2id, 1 → pbkdf2                     │ │
│ │   tokens:   0 → systemd-fido2, points at keyslot 1       │ │
│ │   segments: 0 → aes-xts-plain64, offset 16 MiB           │ │
│ ├──────────────────────────────────────────────────────────┤ │
│ │ keyslot area: the volume key, encrypted once per keyslot │ │
│ └──────────────────────────────────────────────────────────┘ │
├──────────────────────────────────────────────────────────────┤
│ encrypted data (the Btrfs filesystem), sector by sector      │
└──────────────────────────────────────────────────────────────┘

This design has three practical consequences:

  • Changing a passphrase is instant. Only one keyslot is rewritten, not 950 GiB of data.
  • Several ways to unlock can coexist. A passphrase in slot 0, a TPM2-sealed key in slot 1, a recovery key in slot 2. Any one of them opens the container.
  • The header is everything. If the header is destroyed, the volume key is gone, and no passphrase in the world brings the data back.

cryptsetup luksDump shows the header in readable form. Here’s the one from my laptop, with the salts, the digest and the credential data cut out:

sudo cryptsetup luksDump /dev/nvme0n1p3
LUKS header information
Version:        2
Epoch:          5
Metadata area:  16384 [bytes]
Keyslots area:  16744448 [bytes]
UUID:           03ef...
Label:          (no label)
Subsystem:      (no subsystem)
Flags:          (no flags)

Data segments:
  0: crypt
        offset: 16777216 [bytes]
        length: (whole device)
        cipher: aes-xts-plain64
        sector: 512 [bytes]

Keyslots:
  0: luks2
        Key:        512 bits
        Priority:   normal
        Cipher:     aes-xts-plain64
        Cipher key: 512 bits
        PBKDF:      argon2id
        Time cost:  7
        Memory:     1048576
        Threads:    4
        Salt:       ...
        AF stripes: 4000
        AF hash:    sha256
        Area offset:32768 [bytes]
        Area length:258048 [bytes]
        Digest ID:  0
  1: luks2
        Key:        512 bits
        Priority:   normal
        Cipher:     aes-xts-plain64
        Cipher key: 512 bits
        PBKDF:      pbkdf2
        Hash:       sha512
        Iterations: 1000
        Salt:       ...
        AF stripes: 4000
        AF hash:    sha512
        Area offset:290816 [bytes]
        Area length:258048 [bytes]
        Digest ID:  0
Tokens:
  0: systemd-fido2
        fido2-credential:
                    ...
        fido2-salt: ...
        fido2-rp:   io.systemd.cryptsetup
        fido2-clientPin-required:
                    true
        fido2-up-required:
                    true
        fido2-uv-required:
                    false
        Keyslot:    1
Digests:
  0: pbkdf2
        Hash:       sha256
        Iterations: 372893
        Salt:       ...
        Digest:     ...

A few things are worth pointing out. The data segment starts at an offset of 16777216 bytes, which is the sector 32768 from the dm table. The key is 512 bits because XTS mode splits it into two 256-bit AES keys.

Keyslot 0 is my passphrase. It uses Argon2id as its key derivation function, with a memory cost of 1 GiB: every single passphrase guess costs an attacker a gigabyte of RAM and a noticeable amount of time, which makes brute-forcing on GPUs impractical. That’s also the reason unlocking takes a second or two even with the right passphrase.

Keyslot 1 looks alarmingly weak in comparison: plain PBKDF2 with only 1000 iterations. It isn’t, and the reason is in the Tokens: section. Token 0 is of type systemd-fido2 and points at keyslot 1. The key for this slot doesn’t come from a human. It’s a random secret that a FIDO2 security key computes from the stored credential and salt, and only after I’ve entered the token’s PIN (clientPin-required) and touched it (up-required, user presence). A key with that much entropy doesn’t need stretching, so systemd uses the cheapest KDF setting there is. This is the keyslot I use every day. The passphrase in slot 0 is the fallback.

The digest at the end is how cryptsetup knows that a keyslot produced the right result: it’s a hash of the volume key, and both keyslots refer to it with Digest ID: 0. Both slots unlock the same volume key.

Everyday cryptsetup

# Add a second passphrase (asks for an existing one first)
sudo cryptsetup luksAddKey /dev/nvme0n1p3

# Change the passphrase of a keyslot
sudo cryptsetup luksChangeKey /dev/nvme0n1p3

# Remove a keyslot
sudo cryptsetup luksKillSlot /dev/nvme0n1p3 2

# Check which keyslot a passphrase opens, without opening anything
sudo cryptsetup open --test-passphrase --verbose /dev/nvme0n1p3

# Status of the active mapping
sudo cryptsetup status luks-03ef...

And the one command everyone with an encrypted disk should have run once:

sudo cryptsetup luksHeaderBackup /dev/nvme0n1p3 \
    --header-backup-file /run/media/user/backup/luks-header-nvme0n1p3.img

The target here is a mounted USB stick, on purpose. The backup contains the keyslots, so it’s exactly as sensitive as the disk itself: whoever has it and a valid passphrase can open the container, even after you’ve changed that passphrase on the live disk. Store it offline, not on the disk it belongs to.

How the container gets opened at boot

In part 1, a unit called systemd-cryptsetup@luks\x2d03ef....service appeared below cryptsetup.target. Now it’s clear where it comes from. Two sources describe which containers to open:

  • The kernel command line: rd.luks.uuid=luks-03ef... tells the initramfs that this container is needed for the root filesystem.
  • /etc/crypttab: the table of encrypted devices, the counterpart to /etc/fstab.
# /etc/crypttab
# <name>       <device>     <keyfile>  <options>
luks-03ef...   UUID=03ef...  none       discard

systemd-cryptsetup-generator is another generator like the fstab generator from part 1. It turns both sources into systemd-cryptsetup@.service instances, in the initramfs and on the running system. none in the third column means “ask for a passphrase”, and discard is where the allow_discards flag in the dm table came from. Dracut copies the relevant lines of /etc/crypttab into the initramfs, which is why part 1 listed it among the files that require a dracut -f after changes.

A word on discard: passing TRIM through an encrypted device reveals to someone with access to the raw disk which sectors are unused. For a laptop SSD that trade is almost always worth it, since the drive needs the information for wear leveling and performance. If you need to hide even the fill level of a disk, leave it out.

TPM2 Auto-Unlock with systemd-cryptenroll

Typing a long passphrase on every boot is the price of disk encryption, and on servers it’s often a price nobody can pay, because nobody is standing in front of the machine. The TPM2 chip that practically every machine of the last decade contains offers a way out: let the hardware hold the key, and let it release the key only if the machine booted the way you expect.

Measured boot and PCRs

Part 1 explained that Secure Boot verifies: each stage refuses to start the next one if the signature is wrong. In parallel, a second mechanism measures: each stage hashes the next one and sends the hash to the TPM before executing it. The TPM stores these measurements in Platform Configuration Registers (PCRs).

A PCR can’t be written, only extended: the new value is the hash of the old value concatenated with the new measurement. That makes a PCR a fingerprint of the entire sequence of things measured into it since power-on. There’s no way to set a PCR to a chosen value, and no way to undo a measurement without a reboot.

PCR What gets measured into it
0 Firmware code
1 Firmware configuration (setup options, boot order)
2, 3 Option ROMs and their configuration
4 Boot loader code: every EFI binary that gets executed (shim, GRUB)
5 GPT partition table
7 Secure Boot state: on or off, PK, KEK, db, dbx, and the certificates used to verify the boot binaries
8 GRUB: commands and the kernel command line
9 GRUB: every file it reads, including kernel and initramfs
11 Unified Kernel Images: kernel, initrd and command line, measured by systemd-stub
14 shim: the MOK certificates

You can look at the current values:

systemd-analyze has-tpm2
sudo systemd-analyze pcrs

The TPM can seal a secret against a policy such as “PCR 7 must have exactly this value”. It will then only hand the secret back while that’s true. systemd-cryptenroll uses this to store a LUKS key: it generates a random key, adds it to a new LUKS keyslot, seals it with the TPM, and stores the sealed blob together with the policy as a token in the LUKS2 header. Nothing secret is stored outside the TPM in usable form, and the sealed blob is useless on any other machine.

Enrolling the TPM

The enrollment is a single command:

sudo systemd-cryptenroll --tpm2-device=auto --tpm2-pcrs=7 --tpm2-with-pin=yes /dev/nvme0n1p3
🔐 Please enter current passphrase for disk /dev/nvme0n1p3: ••••••••••••••••
🔐 Please enter TPM2 PIN: ••••••
🔐 Please enter TPM2 PIN (repeat): ••••••
New TPM2 token enrolled as key slot 2.

Three options matter here:

  • --tpm2-device=auto picks the only TPM in the machine.
  • --tpm2-pcrs=7 binds the key to the Secure Boot state. Pay attention to this one: in current systemd versions the default is to bind to no PCRs at all. Without the option, the TPM releases the key on this machine no matter what was booted.
  • --tpm2-with-pin=yes additionally requires a PIN. More on why in a moment.

Existing keyslots aren’t touched. On a container like mine, with a passphrase in slot 0 and a FIDO2 token in slot 1, the TPM2 key goes into slot 2. Running the tool without options lists what’s enrolled:

sudo systemd-cryptenroll /dev/nvme0n1p3
SLOT TYPE
   0 password
   1 fido2
   2 tpm2

The last step is telling systemd-cryptsetup to use the TPM at boot, with one more option in /etc/crypttab:

luks-03ef...   UUID=03ef...  none  discard,tpm2-device=auto

Then rebuild the initramfs, so the new crypttab and the TPM2 libraries end up in it:

sudo dracut -f
lsinitrd -m /boot/initramfs-$(uname -r).img | grep tpm2
# tpm2-tss

Current dracut versions pull in the tpm2-tss module by themselves as soon as crypttab contains tpm2-device=. The drop-in from part 1 (add_dracutmodules+=" tpm2-tss ") forces it and doesn’t hurt.

On the next boot, the passphrase prompt is replaced by a short PIN prompt, or by no prompt at all if you enrolled without a PIN.

What this protects against, and what it doesn’t

This is the part most tutorials skip, and it’s the most important one.

Without a PIN, bound to PCR 7 only, the disk unlocks whenever this machine boots with Secure Boot in the same state and with the same set of trusted certificates. That protects against someone removing the drive, and against someone booting with Secure Boot switched off. It does not protect a stolen laptop very well, for two reasons:

  • PCR 7 doesn’t say which signed operating system was booted, only which certificate verified it. A Fedora live USB stick boots through the same Microsoft-signed shim with the same Fedora certificate, ends up with the same PCR 7 value, and can ask the TPM for the key.
  • As part 1 pointed out, Secure Boot doesn’t cover the initramfs or the kernel command line, and neither does PCR 7. Both live on the unencrypted /boot partition. Someone with physical access can modify the initramfs to save the key after the TPM has released it.

Without a PIN, the machine also boots all the way to the login screen by itself. From then on, the only thing between an attacker and your data is the login prompt and every network service the machine runs, while the decryption key sits in RAM.

Binding to more PCRs closes these holes on paper. PCR 9 covers the kernel and the initramfs, PCR 8 the command line. In practice, these values change with every kernel update, so the TPM refuses the key after each one and you’re back to typing the passphrase and re-enrolling. Nobody keeps that up for long.

A PIN changes the picture. The TPM only releases the key if the PCRs match and the PIN is correct, and it enforces a lockout after too many wrong attempts in hardware. This is why a six-digit PIN in a TPM can be stronger than a mediocre passphrase without one: the passphrase can be attacked offline with as much hardware as the attacker can afford, the PIN can’t be attacked offline at all. For a laptop, TPM2 with PIN is the combination I’d recommend. For a server in a locked rack, where the realistic threat is a decommissioned or stolen disk, TPM2 without PIN is a reasonable trade.

The proper fix for the unverified initramfs is the one part 1 already named: Unified Kernel Images. With a UKI, kernel, initramfs and command line are measured into PCR 11 as one unit, and the distribution can ship a signed policy that says “any UKI signed by this key is fine”. The TPM key then survives kernel updates without re-enrollment and still doesn’t unlock for a tampered initramfs. That’s a topic for part 8.

When PCR 7 changes

PCR 7 is stable across kernel updates, which is why it’s the usual choice. It does change when the Secure Boot databases change: a dbx revocation update delivered by fwupd, the new Microsoft 2023 certificates being added to db, or someone toggling Secure Boot in the firmware setup. After that, the TPM refuses to unseal, systemd-cryptsetup falls back to asking for the passphrase, and you enroll again:

sudo systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --tpm2-with-pin=yes /dev/nvme0n1p3

--wipe-slot=tpm2 removes the old TPM2 keyslot and its token in the same step. This fallback is the reason to never remove the passphrase slot. A mainboard replacement, a firmware update that resets the TPM, or a dead TPM chip all end the same way: without another keyslot the data is gone. If you don’t want to keep a memorable passphrase around, generate a recovery key and store it somewhere safe:

sudo systemd-cryptenroll --recovery-key /dev/nvme0n1p3

It prints a long, high-entropy key (and a QR code) exactly once. The same tool also enrolls FIDO2 tokens with --fido2-device=auto, and as the header dump earlier showed, that’s what I use on my own laptop. The crypttab counterpart is fido2-device=auto. My own crypttab line from above doesn’t carry it, because systemd-cryptsetup also looks for tokens in the LUKS2 header by itself, but stating it explicitly is the documented way and makes the intent visible. If you carry a YubiKey or Nitrokey anyway, it’s a good alternative to the TPM: the secret lives on a device that isn’t stolen together with the laptop, and the PIN and touch requirements are enforced by the token.

To get rid of the TPM enrollment completely, wipe the slot and remove tpm2-device=auto from /etc/crypttab, followed by another dracut -f:

sudo systemd-cryptenroll --wipe-slot=tpm2 /dev/nvme0n1p3

The VFS: One Tree, Many Filesystems

With the container open, there’s a block device with a filesystem on it. Before looking at Btrfs, one layer deserves a short stop: the Virtual File System. It’s the reason cat doesn’t need to know whether a file lives on Btrfs, ext4, NFS or in /proc.

The VFS defines four objects that every filesystem has to provide:

Object What it represents
superblock One mounted filesystem instance
inode One file or directory: owner, mode, size, timestamps, where the data is. No name.
dentry A name in a directory, pointing at an inode. This is where hard links come from: two dentries, one inode.
file One open file: an inode plus a position and flags. What a file descriptor refers to.

When a process calls open("/home/user/notes.txt"), the VFS walks the path component by component through the dentry cache, crosses from one filesystem into another wherever something is mounted, checks permissions (and asks SELinux), and finally calls into the filesystem that owns the inode. Path lookups are so frequent that the dentry cache is one of the most heavily optimized structures in the kernel. It’s also a visible consumer of memory: sudo slabtop usually shows dentry and the filesystems’ inode caches near the top.

The mount table ties the filesystems into one tree. findmnt shows it:

findmnt --real
TARGET        SOURCE                              FSTYPE OPTIONS
/             /dev/mapper/luks-03ef...[/root]     btrfs  rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=257,subvol=/root
├─/home       /dev/mapper/luks-03ef...[/home]     btrfs  rw,relatime,seclabel,compress=zstd:1,ssd,discard=async,space_cache=v2,subvolid=256,subvol=/home
└─/boot       /dev/nvme0n1p2                      ext4   rw,relatime,seclabel
  └─/boot/efi /dev/nvme0n1p1                      vfat   rw,relatime,fmask=0077,dmask=0077,...

--real hides the two dozen virtual filesystems (proc, sysfs, tmpfs, cgroup2 and friends) that are also mounted on any modern system. Leave it out once to see how much of what looks like files on Linux isn’t stored anywhere. The mount table is also not global: every mount namespace has its own, which is the basis for containers and for systemd’s sandboxing options. That’s part 3.

The second thing the VFS provides is the page cache, the in-memory cache of file contents that sits between every filesystem and the block layer. It gets its own section in the memory half of this article.

Btrfs on Fedora

Fedora has used Btrfs as the default filesystem for desktop installs since Fedora 33. The findmnt output above already shows the two defining features of the default setup: one filesystem mounted twice with different subvol= options, and compress=zstd:1.

If you’ve read the ZFS article in the FreeBSD series, most concepts will be familiar: copy-on-write, checksums on everything, cheap snapshots, a shared pool instead of fixed-size partitions. The implementations and the tooling are very different, and Btrfs has some sharp edges of its own.

Copy-on-write in one paragraph

A traditional filesystem like ext4 or XFS overwrites data in place. Btrfs never does. When a block changes, the new version is written to free space, and then the metadata that points at it is updated, which is again done by writing new metadata blocks, all the way up to the root of the tree. The last step is a single atomic update of the superblock to point at the new root. Until that happens, the old state is fully intact on disk. After a crash, the filesystem is either in the old state or the new one, never in between, and there’s no journal to replay and no fsck at boot.

Everything else follows from this. A snapshot is just a second root pointing at the same tree, so it’s instant and initially takes no space. Checksums for every data and metadata block can be stored in the parent, which makes silent corruption detectable. And a reflink copy of a file shares all of its data blocks with the original until one of them is modified.

Subvolumes

A subvolume is an independent file tree inside a Btrfs filesystem. It looks like a directory, it can be mounted like a filesystem, and it’s the unit that snapshots work on. All subvolumes share the same pool of space.

sudo btrfs subvolume list /
ID 256 gen 569311 top level 5 path home
ID 257 gen 569311 top level 5 path root
ID 258 gen 530397 top level 257 path var/lib/machines

Fedora’s layout is deliberately simple. The top level of the filesystem (always subvolume ID 5) contains two subvolumes, root and home, and nothing else. /etc/fstab mounts them at / and /home, and the IDs match the subvolid= values in the findmnt output above. The top level itself isn’t mounted anywhere. The third subvolume wasn’t created by the installer. It’s nested inside root (top level 257), and it comes from systemd, which creates /var/lib/machines as a subvolume when it finds itself on Btrfs.

What does the split buy you compared to a single tree? Three things:

  • No sizing decisions. / and /home share 952 GiB. There’s no “root is full while home has 300 GiB free”, the classic problem of partition and LVM layouts.
  • Independent snapshots. A snapshot covers exactly one subvolume and stops at nested ones. You can snapshot and roll back the system without touching your home directory, and the other way around.
  • Reinstall without losing data. The installer can create a new root subvolume and reuse the existing home.

It’s the same reasoning that gives a ZFS-on-root FreeBSD system separate datasets for /home and /var/log, just with far fewer of them.

Creating subvolumes of your own doesn’t need root if you own the parent directory:

btrfs subvolume create ~/vm-images
btrfs subvolume show ~/vm-images

A good reason to do that is to exclude something from snapshots: container storage, VM images, build caches.

Snapshots

# A read-only snapshot of the root subvolume
sudo mkdir /.snapshots
sudo btrfs subvolume snapshot -r / /.snapshots/root-before-upgrade

# Delete it again
sudo btrfs subvolume delete /.snapshots/root-before-upgrade

Getting a single file back is a plain cp out of the snapshot directory. Rolling back the whole system is possible too, and part 1 already showed the mechanism: the root subvolume is chosen by rootflags=subvol=root on the kernel command line. Create a writable snapshot from the read-only one, press e in GRUB, change rootflags=subvol= to point at it, and the system boots into the old state.

Three caveats before you rely on that. First, keeping snapshots in /.snapshots inside the root subvolume is fine for getting files back, but awkward for rollbacks. Once you boot into a clone, the original root is no longer mounted at /, and /.snapshots is gone from view until you mount the top level (mount -o subvolid=5) somewhere. Setups that are built for rollback keep their snapshots outside the root subvolume for that reason, either directly in the top level or in a dedicated snapshots subvolume next to root and home. Second, /boot is a separate ext4 partition on Fedora. Kernels and initramfs images aren’t part of any Btrfs snapshot, so a snapshot from before a kernel update needs a kernel that’s still installed. Third, Fedora doesn’t set up any of this automatically. There’s no snapshot before dnf upgrade and no boot menu entry for old states unless you install and configure something like snapper yourself. If you want atomic, bootable system states out of the box, that’s what the image-based Fedora variants are for, and they’re the topic of part 8.

And the obligatory warning: a snapshot lives on the same disk in the same filesystem. It protects against rm -rf and bad upgrades, not against a dead SSD or a lost laptop. btrfs send and btrfs receive can replicate snapshots to another disk, which is the Btrfs building block for a real backup strategy.

The same sharing mechanism works on the level of single files:

cp --reflink=always fedora.qcow2 fedora-test.qcow2

The copy of a 40 GiB image takes a fraction of a second and no additional space, because both files point at the same extents. Only the blocks that later change in one of the two get written separately. Recent versions of cp try this automatically (--reflink=auto is the default), so on Btrfs and XFS you’re probably using reflinks without knowing it.

Why df is the wrong tool

Free space on Btrfs is harder to state than on other filesystems, and df gives you an estimate at best. The reason is two-level allocation. Btrfs first carves the raw device into chunks (typically 1 GiB for data), each dedicated to either data or metadata with a specific redundancy profile. Files are then stored inside chunks. The honest view comes from Btrfs’s own tool:

sudo btrfs filesystem usage /
Overall:
    Device size:                 952.27GiB
    Device allocated:            470.02GiB
    Device unallocated:          482.24GiB
    Device missing:                  0.00B
    Device slack:                    0.00B
    Used:                        351.55GiB
    Free (estimated):            594.62GiB      (min: 353.50GiB)
    Free (statfs, df):           594.62GiB
    Data ratio:                       1.00
    Metadata ratio:                   2.00
    Global reserve:              512.00MiB      (used: 0.00B)
    Multiple profiles:                  no

Data,single: Size:458.01GiB, Used:345.63GiB (75.46%)

Metadata,DUP: Size:6.00GiB, Used:2.96GiB (49.36%)

System,DUP: Size:8.00MiB, Used:80.00KiB (0.98%)

How to read it:

  • Device allocated (470 GiB) is how much of the disk has been assigned to chunks. Used (351 GiB) is how much of that actually contains something. The difference is free space inside existing chunks.
  • Device unallocated (482 GiB) hasn’t been assigned to anything yet. This is the number to watch.
  • Data,single means file contents are stored once. Metadata,DUP means all metadata is stored twice on the same device, hence the metadata ratio of 2.00. If one copy of a metadata block fails its checksum, Btrfs repairs it from the other. For data on a single disk, corruption is detected and reported, but can’t be repaired.

You may have noticed that the two “Free” lines are identical here, so df is telling the truth on this machine. That’s because this is the easy case: one disk, data stored once, and 482 GiB still unallocated. The two numbers drift apart when unallocated space gets close to zero, because df can’t express “there’s room for data but not for metadata”. They also diverge on filesystems with several devices or mixed RAID profiles, where the usable space depends on which profile future chunks are allocated with.

The situation that used to give Btrfs a bad name is a filesystem where all space is allocated but not used: plenty of room in data chunks, no unallocated space left, and the metadata chunks full. Then even deleting a file can fail with “No space left on device”, because deleting needs to write new metadata. Modern kernels reclaim empty and nearly empty chunks automatically, so this has become rare. If you ever get there, a filtered balance compacts data chunks that are less than half full and returns the freed chunks to the unallocated pool:

sudo btrfs balance start -dusage=50 /

Don’t run a full, unfiltered balance as routine maintenance. It rewrites the entire filesystem and helps nothing.

Transparent compression

Fedora mounts with compress=zstd:1: every file is compressed with zstd at level 1 on write and decompressed on read, invisible to applications. Level 1 is the fastest setting, and on anything faster than a SATA SSD it’s chosen so that compression never becomes the bottleneck. On slow storage like SD cards it often makes the system faster, because less data has to cross the slow device.

Btrfs doesn’t blindly compress everything. For each file, it tests the beginning of the data, and if that doesn’t shrink, it marks the file as incompressible and stops trying. Already compressed formats (video, JPEG, archives, most of a browser cache) are therefore stored as they are, at almost no CPU cost.

Regular tools can’t tell you what compression saves. ls and du report uncompressed sizes. The tool for the job is compsize:

sudo dnf install compsize
sudo compsize -x /
sudo compsize -x /home

-x keeps compsize from crossing mount points. Since / and /home are separate mounts, compsize -x / only looks at the root subvolume, so I run it once for each:

# /
Processed 468921 files, 311027 regular extents (365860 refs), 258873 inline, 609368 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       80%       35G          44G          47G
none       100%       30G          30G          31G
zstd        34%      4.6G          13G          16G
prealloc   100%      168M         168M         112M

# /home
Processed 1625518 files, 1707709 regular extents (1936499 refs), 867091 inline, 2380944 fragments.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       79%      310G         393G         414G
none       100%      280G         280G         282G
zstd        27%       30G         112G         131G
prealloc   100%      162M         162M         321M

Read the first block like this: 30 GiB of data on the root subvolume aren’t compressed and are stored as they are. Another 13 GiB were compressed down to 4.6 GiB, 34% of their original size. In total, 44 GiB of data occupy 35 GiB on disk. prealloc is space that programs reserved with fallocate() and haven’t written yet. Referenced is larger than Uncompressed because of reflinks and snapshots: some extents are referenced more than once.

The two rows tell two different stories. Whatever Btrfs does compress shrinks to about a third on / and to about a quarter on /home, which is a very good result for the cheapest zstd level. But by volume, most of what’s on this disk isn’t compressible at all: photos, video, archives, container layers and VM images all end up in the none row, either because they’re compressed already or because they’re NOCOW files (more on those below). The bottom line for the whole disk is 345 GiB used for 437 GiB of data, so compression saves me a bit over 90 GiB, or about 20%. A system that holds mostly binaries, libraries, source code and logs does far better than that. The numbers depend entirely on what you store.

Changing the settings:

# Try a different level right now (not persistent)
sudo mount -o remount,compress=zstd:3 /home

# Persistent: change compress=zstd:1 in /etc/fstab, then
sudo systemctl daemon-reload

# Per directory: new files below it are compressed even without the mount option
btrfs property set ~/logs compression zstd

Mount options only affect newly written data. Existing files stay as they are until they’re rewritten. btrfs filesystem defragment -r -czstd <path> rewrites them, but be careful: defragmenting breaks the sharing between snapshots and reflinked copies, so on a filesystem with many snapshots it can increase disk usage dramatically.

Checksums and scrub

Every block Btrfs reads is verified against its checksum, and a mismatch results in an I/O error instead of silently wrong data. Blocks that are never read are never checked, though. A scrub reads everything and verifies it:

sudo btrfs scrub start /
sudo btrfs scrub status /

# Error counters per device since the filesystem was created
sudo btrfs device stats /

On a single-disk laptop, scrub can repair metadata (thanks to DUP) and tell you exactly which files have damaged data, which is still far better than finding out when you open them a year later. Fedora doesn’t schedule scrubs. A monthly timer is a sensible addition on any machine whose data you care about.

When copy-on-write hurts

CoW has a cost: a file that’s constantly modified in the middle gets fragmented, because every changed block is written somewhere new. For most files nobody notices. For large files with lots of small random writes, namely VM disk images and database files, it can become a real performance problem on top of the fragmentation that those formats already produce internally.

The escape hatch is the NOCOW attribute:

mkdir ~/vm-images
chattr +C ~/vm-images
lsattr -d ~/vm-images
# ---------------C------ /home/user/vm-images

Files created in that directory from now on are overwritten in place. The attribute only works on empty files, so set it on the directory before putting images there. It’s a trade, not a free win: NOCOW files have no checksums and no compression. libvirt sets the attribute by itself when it creates a storage pool on Btrfs, so for the default /var/lib/libvirt/images you don’t have to do anything.

TRIM

The mount options included discard=async, which Btrfs enables by itself on devices that support it. Freed extents are collected and reported to the drive in batches a little later, which avoids the latency spikes that synchronous discard used to cause. Together with discard in /etc/crypttab, that’s the complete chain: Btrfs frees an extent, dm-crypt passes the discard through, the SSD learns that the blocks are unused. Fedora additionally enables fstrim.timer, which runs a weekly fstrim over all mounted filesystems. For Btrfs with async discard it’s redundant, but for the ext4 /boot and the ESP it isn’t.

And on RHEL?

RHEL removed Btrfs in version 8 and never brought it back. The default there is XFS on LVM: a volume group, logical volumes for root, swap and optionally home, and XFS on top. XFS is an excellent, very scalable filesystem and supports reflinks, but it has no built-in snapshots, compression or data checksums. Snapshots come from LVM thin pools, or from Stratis, which bundles XFS and thin provisioning behind a simpler interface. If you want a checksumming CoW filesystem on RHEL, you’re in third-party territory, as described in the articles about ZFS on RHEL and bcachefs on RHEL 10.

Memory: Page Cache and Anonymous Pages

Now to the other half. The kernel manages memory in pages of 4 KiB, and for understanding everything that follows, only one distinction matters. Every page a process uses is one of two kinds:

  • File-backed pages are copies of something that exists on disk: the contents of files that were read or written, and the code of every running program and shared library, which is just mmap()ed files. Collectively, they’re the page cache.
  • Anonymous pages have no file behind them: heap, stack, everything a program allocates with malloc(). tmpfs and shared memory are technically file pages, but behave like anonymous ones, because there’s no disk file behind them either.

The difference matters at the moment the kernel needs memory back:

File-backed, clean File-backed, dirty Anonymous
Reclaim by Dropping the page Writing it to disk, then dropping it Writing it to swap, then dropping it
Cost Free now, a disk read later if needed again A disk write A swap write
Without swap Possible Possible Impossible

Keep the bottom right cell in mind. It’s the reason for the entire second half of this article.

Reading free correctly

free -h
               total        used        free      shared  buff/cache   available
Mem:            29Gi       5.1Gi        19Gi        94Mi       5.6Gi        24Gi
Swap:          8.0Gi          0B       8.0Gi

This is my laptop shortly after booting, with a browser and a few terminals open. used is memory claimed by processes and by the kernel itself that can’t simply be dropped. Most of it is anonymous memory, which could go to swap. The rest is kernel memory like page tables and unreclaimable slab, which can’t go anywhere. buff/cache is the page cache: 5.6 GiB of file contents that the kernel keeps around because the RAM would otherwise sit idle. And available is the number that answers “how much can I still allocate?”. It’s the kernel’s estimate of free memory plus everything it could reclaim without swapping.

On a machine that’s been running for a week, free will be close to zero and buff/cache will have taken everything. That’s not a problem and not a leak. Unused RAM is wasted RAM, and the page cache is given back the instant something else needs it. The only number worth alerting on is available.

/proc/meminfo has the details behind the summary:

grep -E '^(MemTotal|MemAvailable|Cached|AnonPages|Shmem|Dirty|SwapTotal|SwapFree):' /proc/meminfo
MemTotal:       31447104 kB
MemAvailable:   26066684 kB
Cached:          5670400 kB
SwapTotal:       8388604 kB
SwapFree:        8388604 kB
Dirty:              2204 kB
AnonPages:       3512632 kB
Shmem:             95768 kB

3.5 GiB of anonymous memory, 5.6 GiB of page cache, and about 2 MiB of dirty pages waiting to be written. Shmem contains everything in tmpfs filesystems, and on Fedora that includes /tmp. A 4 GiB file in /tmp is 4 GiB of memory that can only be reclaimed by swapping it out.

Writes go to memory first

The page cache also explains why writes seem so fast. A write() copies data into the page cache, marks the pages dirty, and returns. The data reaches the disk later, written by kernel flusher threads once the pages are old enough (30 seconds by default) or once there are too many of them:

sysctl vm.dirty_background_ratio vm.dirty_ratio
# vm.dirty_background_ratio = 10
# vm.dirty_ratio = 20

At 10% of available memory in dirty pages, background writeback starts. At 20%, processes that write are throttled until the disk catches up. An application that needs to know its data is on stable storage has to call fsync(), which is what every database does. It’s also why copying a large file to a slow USB stick “finishes” quickly and then sync or the unmount takes five minutes: the copy was to memory.

Reclaim, and why swap matters

As long as there’s free memory, the kernel hands it out. When free memory drops below a threshold, the kswapd kernel thread wakes up and starts reclaiming in the background: it looks for pages that haven’t been used recently and frees them, by one of the three routes in the table above. If kswapd can’t keep up, processes that allocate memory have to do reclaim themselves before their allocation succeeds (direct reclaim), and that’s when the system starts to feel slow.

Now consider a system without any swap. Anonymous pages can’t be reclaimed at all, so the only thing the kernel can free is page cache. Under growing memory pressure it drops more and more of it, including the pages that hold the code of running programs. Those programs need their code to continue, so the pages are read back from disk, displacing other pages that are needed a moment later. The system spends all of its time reading the same executable pages over and over. The mouse pointer freezes, SSH logins time out, and it can stay in this state for many minutes before the kernel finally concludes that it’s out of memory. That’s thrashing, and it’s the main failure mode of a swapless Linux machine.

So swap isn’t “emergency RAM”. It’s what gives the kernel a choice. With swap, an anonymous page that hasn’t been touched for hours (a background tab, an idle daemon) can be moved out to make room for page cache that’s actually in use.

How the kernel weighs that choice is controlled by vm.swappiness. It’s not a percentage and not a threshold. It’s the relative cost the kernel assumes for bringing back a swapped-out page compared to re-reading a file page, on a scale from 0 to 200. At 100, both count as equally expensive. Fedora uses the kernel default:

sysctl vm.swappiness
# vm.swappiness = 60

A value below 100 says “swapping is more expensive than re-reading files”, which was a safe assumption when swap meant a partition on a rotating disk. It’s much less true for what Fedora uses as swap.

Swap on zram

Since Fedora 33 there’s no swap partition in a default install. Instead there’s the device that has been sitting at the top of the lsblk output all along:

swapon --show
zramctl
NAME       TYPE      SIZE USED PRIO
/dev/zram0 partition   8G   0B  100

NAME       ALGORITHM DISKSIZE DATA COMPR TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle         8G   4K   80B   12K         [SWAP]

zram is a kernel module that provides a block device whose contents are stored compressed in RAM. Formatted as swap, it gives the kernel a place to put anonymous pages that it wants to evict: the page is compressed and stored in memory, taking a fraction of its original size. Swapping a page out costs CPU time instead of disk I/O, and getting it back takes microseconds.

That sounds like a trick, using memory to save memory. It works because idle anonymous memory compresses very well. Ratios of 3:1 are typical, so evicting 3 GiB of idle pages into zram frees roughly 2 GiB net.

How it’s set up

The device is created by zram-generator, yet another systemd generator. At boot it reads its configuration and produces the units that were visible in part 1’s dependency tree: systemd-zram-setup@zram0.service, which sets the size and runs mkswap, and dev-zram0.swap, which activates it. The entire default configuration on Fedora is two lines:

# /usr/lib/systemd/zram-generator.conf
[zram0]
zram-size = min(ram, 8192)

The device is as large as the installed RAM, capped at 8 GiB. On my 32 GiB laptop that gives the 8 GiB device above.

The most common misunderstanding about zram is that this size is memory taken away from the system. It isn’t. DISKSIZE is the maximum amount of uncompressed data the device accepts. An empty zram device uses practically nothing, as the zramctl output above shows: 12 KiB in total. With 32 GiB of RAM my laptop practically never swaps, so here’s an example of what the same line looks like on a machine that does:

NAME       ALGORITHM DISKSIZE  DATA  COMPR  TOTAL STREAMS MOUNTPOINT
/dev/zram0 lzo-rle         8G  2.9G 812.4M 842.6M         [SWAP]

DATA is how much has been swapped out (2.9 GiB of pages), COMPR is what it compressed to, and TOTAL is the RAM the device really occupies including overhead. Here 2.9 GiB of idle memory cost about 840 MiB, a ratio of roughly 3.5:1. Even a completely full 8 GiB device would use around 2.5 GiB of RAM at that ratio.

The priority of 100 only matters if you add a disk swap as well: the kernel fills swap devices in order of priority, and regular swap devices get negative priorities by default, so zram is used first.

Tuning it

Settings go into /etc/systemd/zram-generator.conf, which replaces the file in /usr/lib:

# /etc/systemd/zram-generator.conf
[zram0]
zram-size = min(ram / 2, 16384)
compression-algorithm = zstd
sudo systemctl daemon-reload
sudo systemctl restart systemd-zram-setup@zram0.service

Restarting the unit swaps everything back in first, so do it when there’s enough free memory. lzo-rle is the kernel’s default algorithm and optimized for speed. zstd compresses noticeably better at a higher CPU cost per page, which is usually a good trade on machines with little RAM and a bad one on machines that swap heavily under load. This is also the place to pick up vm.swappiness again. Fedora keeps the kernel default of 60, even though that value encodes the assumption that swap is slower than re-reading files, and swapping to zram is the faster of the two. Some zram-only setups therefore raise it to somewhere between 100 and 180, so the kernel prefers compressing cold anonymous pages over evicting page cache that’s in use:

# /etc/sysctl.d/99-swappiness.conf
vm.swappiness = 150
sudo sysctl --system

Whether that helps depends on the workload. On a machine with plenty of RAM you won’t notice a difference. On a small machine that lives in its swap, it’s worth an experiment, with PSI (see below) as the measurement.

To switch zram off, create an empty /etc/systemd/zram-generator.conf or remove the zram-generator-defaults package.

When would you still want disk swap? There are two cases. The first is hibernation: the memory image has to survive power-off, and zram obviously can’t hold it. Fedora doesn’t set up hibernation by default, and with Secure Boot active, kernel lockdown restricts it as well. The second is a workload with a lot of anonymous memory that is truly idle for long periods and doesn’t compress well. A disk swap with lower priority behind zram covers that.

Don’t confuse zram with zswap. zswap is a compressed cache in front of a disk swap device: pages are compressed in RAM first and written to the disk swap when the cache fills up. It needs a real swap device to work. Fedora uses zram and leaves zswap off. Running both is pointless, because zswap would be compressing pages on their way into a device that compresses them again.

On RHEL, none of this is active by default. The installer creates a swap logical volume, and zram-generator is available in AppStream if you want it.

Measuring Memory Pressure: PSI

How do you know whether a system is short on memory? Free memory isn’t the answer, as we’ve seen. Swap usage isn’t either: a full swap with no activity just means some idle pages were moved out long ago. The question that matters is whether processes are losing time because of memory. Since kernel 4.20, there’s a direct measurement for that: Pressure Stall Information.

cat /proc/pressure/memory
some avg10=0.00 avg60=0.00 avg300=0.00 total=25
full avg10=0.00 avg60=0.00 avg300=0.00 total=25
  • some is the share of time in which at least one task was stalled waiting for memory: doing direct reclaim, waiting for a page to be swapped in, or waiting for evicted file pages to be read back.
  • full is the share of time in which all non-idle tasks were stalled at once. Nothing productive happened during that time.
  • avg10, avg60 and avg300 are averages over the last 10, 60 and 300 seconds in percent, and total is the accumulated stall time in microseconds.

My laptop has lost 25 microseconds to memory pressure since boot, so there’s nothing to see here. On a machine in trouble, some avg10 climbs into double digits, and a sustained full value above a few percent means the machine is thrashing. The same files exist for cpu and io, and that makes PSI a great first look at any slow system: it tells you in three cat commands which resource processes are waiting for.

PSI is also tracked per cgroup. Every directory below /sys/fs/cgroup/ has its own memory.pressure file, which tells you whether one specific service or one user session is suffering, even when the system as a whole is fine. That per-cgroup view is what the next component is built on.

Out of Memory: The Kernel OOM Killer and systemd-oomd

Reclaim can fail. If every page is in use, the page cache is at its minimum and swap is full, an allocation can’t be satisfied. At that point something has to die.

The kernel’s OOM killer

The kernel’s answer is the OOM killer. It computes a badness score for every process, essentially its memory footprint including swap, adjusted by oom_score_adj, a per-process value between -1000 (never kill) and 1000 (kill first). It then kills the process with the highest score and logs its decision:

# Kills by the kernel OOM killer in the current boot
journalctl -k -b | grep -i -E 'out of memory|oom-kill'

# The score of a process, and its adjustment
cat /proc/$$/oom_score /proc/$$/oom_score_adj

systemd sets these adjustments for you. Unit files can use OOMScoreAdjust=, and important system services ship with negative values so that they’re picked last.

The OOM killer works, and it has two problems. The first is that it acts late. The kernel only invokes it when reclaim has definitively failed, and as described above, a system can spend minutes thrashing before that point. From the kernel’s point of view, it’s still making progress as long as it can drop and re-read page cache. From the user’s point of view, the machine has been dead for a while. The second problem is that it kills one process, chosen by size. That may be one worker of a multi-process application, leaving the rest in a broken state, or a large and innocent process instead of the many small ones that caused the problem.

You can watch the kernel OOM killer in a safe way, confined to a cgroup with a memory limit. This runs tail /dev/zero, which allocates memory until something stops it, in a scope that’s allowed 1 GiB and no swap:

systemd-run --user --scope -p MemoryMax=1G -p MemorySwapMax=0 tail /dev/zero
# Running as unit: run-r3f1c....scope
# Killed

journalctl -k -b | grep -i 'out of memory' | tail -1
# Memory cgroup out of memory: Killed process 48211 (tail) total-vm:1052640kB, anon-rss:1047552kB, ...

The limit applies to the cgroup, so the kernel OOM killer only considers processes inside it. The rest of the system doesn’t notice. How cgroups work is the topic of part 3.

systemd-oomd

Fedora enables a second line of defense that acts earlier: systemd-oomd, a userspace daemon that watches PSI and kills before the system becomes unresponsive. It replaced earlyoom on Fedora Workstation and has been the default on all desktop variants since Fedora 34.

The idea is to act on the symptom that users care about. systemd-oomd doesn’t wait for memory to run out. It monitors the memory pressure of selected cgroups, and when the pressure stays above a limit for long enough, it picks a cgroup below the monitored one and kills everything in it. Fedora’s configuration consists of three small files:

# /usr/lib/systemd/oomd.conf.d/10-oomd-defaults.conf
[OOM]
DefaultMemoryPressureDurationSec=20s

# /usr/lib/systemd/system/system.slice.d/10-oomd-per-slice-defaults.conf
[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=80%

# /usr/lib/systemd/user/slice.d/10-oomd-per-slice-defaults.conf
[Slice]
ManagedOOMMemoryPressure=kill
ManagedOOMMemoryPressureLimit=80%

In words: system.slice (all system services) and every slice of every user’s session are monitored. If the full memory pressure of one of them stays above 80% for more than 20 seconds, meaning its processes were completely stalled for more than 80% of that time, systemd-oomd acts. oomctl shows the live view:

oomctl
Dry Run: no
Swap Used Limit: 90.00%
Default Memory Pressure Limit: 60.00%
Default Memory Pressure Duration: 20s
System Context:
        Memory: Used: 5.1G, Total: 29.9G
        Swap: Used: 0B, Total: 7.9G
Swap Monitored CGroups:
Memory Pressure Monitored CGroups:
        Path: /system.slice
                Memory Pressure Limit: 80.00%
                Memory Pressure Duration: 20s
                Pressure: Avg10: 0.00, Avg60: 0.00, Avg300: 0.00, Total: 21us
                Current Memory Usage: 1.3G
                ...
        Path: /user.slice/user-1000.slice/user@1000.service
                Memory Pressure Limit: 80.00%
                Memory Pressure Duration: 20s
                Pressure: Avg10: 0.00, Avg60: 0.00, Avg300: 0.00, Total: 0
                Current Memory Usage: 6.2G
                ...
        Path: /user.slice/user-1000.slice/user@1000.service/app.slice
                Memory Pressure Limit: 80.00%
                ...

Two details in this output are worth a closer look. “Swap Monitored CGroups” is empty: systemd-oomd also has a second, swap-based trigger (ManagedOOMSwap=), which kills when both memory and swap are more than 90% full. Fedora 45 doesn’t enable it for any unit, so only the pressure-based policy is active. And the limit applies to the pressure of the monitored cgroup, not to the whole machine, so a user session that’s thrashing gets dealt with even while system services still run fine.

Who gets killed

When the limit is exceeded, systemd-oomd looks at the cgroups below the monitored one and picks the one with the most reclaim activity, which is the one the kernel is working hardest to squeeze memory out of. It then sends SIGKILL to every process in that cgroup.

This is where the desktop integration pays off. GNOME and KDE start every application in its own systemd scope below app.slice:

systemd-cgls --user-unit app.slice | head -20

So “the cgroup with the most reclaim activity” is a whole application: all processes of the browser, or the IDE together with its language servers. That’s a much more sensible unit to kill than a single PID. It also means the kill is clean from the system’s point of view. The application is gone completely, not left half alive.

The decision shows up in the journal. My laptop has never had a systemd-oomd kill, so this is an example of what such a message looks like, not a capture:

journalctl -u systemd-oomd -b
systemd-oomd[1423]: Killed /user.slice/user-1000.slice/user@1000.service/app.slice/app-org.mozilla.firefox-b1f0....scope due to memory pressure for /user.slice/user-1000.slice/user@1000.service/app.slice being 84.12% > 80.00% for > 20s with reclaim activity

The flip side is everything that isn’t cleanly separated into cgroups. Depending on the terminal emulator, all commands started in one terminal window may share a single scope. If a compiler run in such a window eats all the memory, systemd-oomd kills the scope, and the terminal with all its tabs goes with it. The fix is to put heavy jobs into their own scope, ideally with a limit of their own, so they get throttled and killed alone:

systemd-run --user --scope -p MemoryHigh=12G -p MemoryMax=16G make -j16

Individual units can also be protected. ManagedOOMPreference=avoid in a unit tells systemd-oomd to pick it only when there’s no other candidate, and omit takes it out of the selection completely. Use both sparingly: every unit that’s exempt makes it more likely that something else gets killed in its place.

How the pieces work together

Here’s the whole sequence on a default Fedora desktop when an application starts eating memory:

memory fills up
      │
      ▼
kswapd reclaims: drops cold page cache,
compresses idle anonymous pages into zram        ← invisible, the normal case
      │
      ▼
zram fills, reclaim gets harder,
processes stall in direct reclaim                ← PSI "some" rises, things feel slow
      │
      ▼
"full" pressure in the session > 80% for 20 s
systemd-oomd kills the busiest app cgroup        ← one application dies, desktop survives
      │
      ▼
(only if that didn't happen or didn't help)
allocation fails, kernel OOM killer picks
the process with the highest oom_score           ← last resort

zram plays a second role in this chain besides saving memory. It buys time. With swap, memory pressure builds up gradually and measurably, which gives systemd-oomd a window to react. Without swap, a system goes from “fine” to “livelocked” so quickly that a userspace daemon may not even get scheduled in time. That’s why the systemd-oomd documentation strongly recommends having swap, and why Fedora introduced zram first and systemd-oomd one release later.

On servers, the picture is different. RHEL ships systemd-oomd but doesn’t enable it, and that’s a defensible default. A database server with a carefully sized buffer pool doesn’t want a daemon making kill decisions based on pressure heuristics. There, explicit MemoryMax= and MemoryHigh= limits per service, with the kernel OOM killer acting inside the cgroup that exceeded its limit, are the more predictable tool.

Cheat Sheet

Question Command
What block devices exist, and how are they stacked? lsblk -f
Which I/O scheduler does a disk use? cat /sys/block/<dev>/queue/scheduler
What does a device-mapper device do? sudo dmsetup table
Which keyslots and tokens does a LUKS container have? sudo systemd-cryptenroll <dev>, sudo cryptsetup luksDump <dev>
Back up the LUKS header sudo cryptsetup luksHeaderBackup <dev> --header-backup-file <file>
Re-enroll the TPM after a PCR change sudo systemd-cryptenroll --wipe-slot=tpm2 --tpm2-device=auto --tpm2-pcrs=7 --tpm2-with-pin=yes <dev>
What’s mounted where, with which options? findmnt --real
How full is the Btrfs filesystem really? sudo btrfs filesystem usage /
Which subvolumes exist? sudo btrfs subvolume list /
What does compression save? sudo compsize -x <mountpoint>, once per mount
Has the disk returned corrupt data? sudo btrfs device stats /, sudo btrfs scrub start /
How much memory is available? free -h, column available
How well does zram compress? zramctl
Are processes stalling on memory? cat /proc/pressure/memory
What does systemd-oomd monitor? oomctl
Who was killed, and by whom? journalctl -u systemd-oomd, journalctl -k --grep=oom

Common Pitfalls

Enrolling a TPM2 key without --tpm2-pcrs=. The default binds to no PCRs. The key is then only tied to the TPM chip, and anything that boots on that machine can unseal it. Always state the PCRs explicitly, and check the result with cryptsetup luksDump, which lists them in the token.

Treating TPM auto-unlock without a PIN as equivalent to a passphrase. It protects a disk that leaves the machine. It doesn’t protect a machine that leaves the building. Use a PIN on anything portable.

Removing the passphrase after enrolling the TPM. Firmware updates, dbx updates and hardware repairs all invalidate the TPM enrollment sooner or later. Keep a passphrase or a recovery key, and keep a header backup offline.

Trusting df on Btrfs. Look at btrfs filesystem usage and watch the unallocated space. Use compsize instead of du when you want to know what’s really on disk.

Running defragment or a full balance as “maintenance”. Defragmenting un-shares snapshot and reflink extents and can multiply disk usage. A full balance rewrites everything for no benefit. Neither is needed on a healthy filesystem.

Putting VM images on CoW without thinking about it. Set chattr +C on the directory before creating images in it, and remember that those files then have no checksums.

Reading the zram size as reserved memory. DISKSIZE is a limit for uncompressed data. Look at TOTAL in zramctl for what the device really costs.

Disabling swap “for performance”. Without swap, the kernel can only reclaim file pages, and the result under pressure is thrashing instead of swapping. It also takes away the reaction time systemd-oomd depends on.

Disabling systemd-oomd because it killed something. It killed something because the session was stalled 80% of the time for 20 seconds. Without it, the same situation ends with a frozen desktop and a later, less targeted kill by the kernel. Put the heavy job into its own scope with a memory limit instead.

Panicking about low free memory. Page cache is supposed to fill the RAM. Monitor available and PSI, not free.

Conclusion

The storage stack is a tower of small, well-defined layers. The block layer turns hardware into numbered sectors. Device mapper builds virtual block devices from tables, and dm-crypt is the one table line that encrypts every sector on my laptop. LUKS2 manages the keys for it, in a header that can hold a passphrase, a TPM2-sealed key and a recovery key side by side. On top of that, Btrfs turns one block device into a pool with subvolumes, snapshots, checksums and compression, and the VFS makes all of it look like one tree of files.

Memory follows a similar logic, with one distinction at its center: file-backed pages can be dropped and re-read, anonymous pages can only go to swap. Fedora’s defaults build on that. zram gives anonymous memory somewhere to go without touching the disk, PSI measures whether processes are actually losing time, and systemd-oomd uses that measurement to kill one application before the whole machine stalls. The kernel’s OOM killer is still there, as the last resort it was always meant to be.

The mental model to keep: keys are wrapped, never used directly, and the TPM only adds real security together with a PIN or a measured, signed boot chain. Free space on Btrfs is about unallocated chunks, and free memory on Linux is about available, not free. And swap isn’t overflow storage. It’s what gives the kernel, and systemd-oomd, room to make good decisions.

Several times in this article, the explanation ended with “that’s a cgroup”. The memory limits, the per-cgroup pressure files, the scopes that systemd-oomd kills, the slices in oomctl. Part 3 picks that up: cgroups v2 and how systemd maps slices, scopes and services onto them, namespaces and capabilities as the building blocks of containers, and how to use all of it to sandbox a service with a handful of lines in a unit file.


References

Comments

You can use your Mastodon or other ActivityPub account to comment on this article by replying to the associated post.

Search for the copied link on your Mastodon instance to reply.

Loading comments...