<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>Larvitz Blog - FreeBSD</title><link href="https://hofstede.it/" rel="alternate"/><link href="https://hofstede.it/feeds/freebsd.atom.xml" rel="self"/><id>https://hofstede.it/</id><updated>2026-11-05T00:00:00+01:00</updated><subtitle>FreeBSD, Linux, all things cleanly engineered</subtitle><entry><title>Hardware Keys Without GnuPG: The Rust OpenPGP Card Stack, Now in FreeBSD Ports</title><link href="https://hofstede.it/hardware-keys-without-gnupg-the-rust-openpgp-card-stack-now-in-freebsd-ports/" rel="alternate"/><published>2026-11-05T00:00:00+01:00</published><updated>2026-11-05T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-11-05:/hardware-keys-without-gnupg-the-rust-openpgp-card-stack-now-in-freebsd-ports/</id><summary type="html">&lt;p&gt;Three small Rust tools from the OpenPGP Card project replace the GnuPG stack for my everyday hardware key work: oct to manage OpenPGP cards and Nitrokeys, oct-git to sign Git commits, and openpgp-card-ssh-agent as a hardware-backed &lt;span class="caps"&gt;SSH&lt;/span&gt; agent. I ported all three to FreeBSD and they are now in the ports tree. This article explains what each tool does, how they share &lt;span class="caps"&gt;PIN&lt;/span&gt; handling, and how I set them up on Fedora and on my FreeBSD&amp;nbsp;T480.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A finger about to press the glowing touch button of a black USB security key on a dark desk, with glowing cables leading to three small screens showing a chip, a Git branch and an SSH prompt with a key" src="https://hofstede.it/images/2026-11-05-openpgp-card-rust-stack-freebsd.webp" title="One card, three small tools, on Linux and FreeBSD."&gt;&lt;/p&gt;
&lt;p&gt;I have kept my &lt;span class="caps"&gt;SSH&lt;/span&gt; and signing keys on hardware for years. The private key lives on an OpenPGP card or a Nitrokey, it never touches the disk, and a stolen laptop is a stolen laptop rather than a stolen identity. What changed over the last year is the software I use to talk to those cards. For &lt;span class="caps"&gt;SSH&lt;/span&gt; authentication and Git signing I no longer go through GnuPG at all. I use three small Rust programs from the &lt;a href="https://codeberg.org/openpgp-card"&gt;OpenPGP Card project&lt;/a&gt;&amp;nbsp;instead.&lt;/p&gt;
&lt;p&gt;I use them daily on Fedora and on my &lt;a href="https://hofstede.it/the-t480-is-back-my-freebsd-151-laptop-from-boot-to-browser/"&gt;FreeBSD T480&lt;/a&gt;. On FreeBSD I had been running them&amp;nbsp;from &lt;code&gt;~/.cargo/bin&lt;/code&gt;, which works but is not how software should arrive on a FreeBSD machine. So I wrote ports for them, and all three have now been committed to the FreeBSD ports&amp;nbsp;tree:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Port&lt;/th&gt;
&lt;th&gt;Binary&lt;/th&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Upstream&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.freshports.org/security/openpgp-card-tools/"&gt;&lt;code&gt;security/openpgp-card-tools&lt;/code&gt;&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;oct&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Inspect and administer cards&lt;/td&gt;
&lt;td&gt;&lt;a href="https://codeberg.org/openpgp-card/openpgp-card-tools"&gt;openpgp-card-tools&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.freshports.org/security/openpgp-card-tools-git/"&gt;&lt;code&gt;security/openpgp-card-tools-git&lt;/code&gt;&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;oct-git&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sign and verify Git commits and tags&lt;/td&gt;
&lt;td&gt;&lt;a href="https://codeberg.org/openpgp-card/oct-git"&gt;oct-git&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;a href="https://www.freshports.org/security/openpgp-card-ssh-agent/"&gt;&lt;code&gt;security/openpgp-card-ssh-agent&lt;/code&gt;&lt;/a&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;openpgp-card-ssh-agent&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; agent backed by the card&amp;#8217;s authentication key&lt;/td&gt;
&lt;td&gt;&lt;a href="https://codeberg.org/openpgp-card/ssh-agent"&gt;ssh-agent&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;At the time of&amp;nbsp;writing, &lt;code&gt;openpgp-card-tools&lt;/code&gt; is already available as a binary package in&amp;nbsp;the &lt;code&gt;latest&lt;/code&gt; repository. The other two will follow once the package builders have picked them up. Since the ports were added this quarter, none of them are&amp;nbsp;in &lt;code&gt;quarterly&lt;/code&gt; yet.&lt;/p&gt;
&lt;p&gt;This article explains what the three tools do, how they fit together, and how I set them up on both&amp;nbsp;systems.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-not-just-use-gnupg"&gt;Why Not Just Use&amp;nbsp;GnuPG?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-you-need"&gt;What You Need&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#on-freebsd"&gt;On&amp;nbsp;FreeBSD&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#on-fedora"&gt;On&amp;nbsp;Fedora&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#oct-managing-the-card"&gt;oct: Managing the Card&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#setting-up-a-new-card"&gt;Setting Up a New&amp;nbsp;Card&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#touch-confirmation"&gt;Touch&amp;nbsp;Confirmation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-public-side"&gt;The Public&amp;nbsp;Side&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#where-the-pin-lives"&gt;Where the &lt;span class="caps"&gt;PIN&lt;/span&gt; Lives&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-pinentry-trap-on-freebsd"&gt;The pinentry Trap on&amp;nbsp;FreeBSD&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#oct-git-signing-commits-with-the-card"&gt;oct-git: Signing Commits With the Card&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#verifying-signatures"&gt;Verifying&amp;nbsp;Signatures&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-freebsd-wrapper"&gt;The FreeBSD&amp;nbsp;Wrapper&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#openpgp-card-ssh-agent-ssh-keys-that-cannot-be-copied"&gt;openpgp-card-ssh-agent: &lt;span class="caps"&gt;SSH&lt;/span&gt; Keys That Cannot Be Copied&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#running-it"&gt;Running&amp;nbsp;It&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#on-fedora-systemd-socket-activation"&gt;On Fedora: systemd Socket&amp;nbsp;Activation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#on-freebsd-an-xfce-autostart-entry"&gt;On FreeBSD: An Xfce Autostart&amp;nbsp;Entry&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-day-with-the-stack"&gt;A Day With the&amp;nbsp;Stack&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#troubleshooting"&gt;Troubleshooting&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#porting-notes"&gt;Porting&amp;nbsp;Notes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="why-not-just-use-gnupg"&gt;Why Not Just Use&amp;nbsp;GnuPG?&lt;/h2&gt;
&lt;p&gt;GnuPG can do all of&amp;nbsp;this. &lt;code&gt;gpg-agent&lt;/code&gt; can act as an &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;agent, &lt;code&gt;scdaemon&lt;/code&gt; talks to the card, and Git&amp;nbsp;calls &lt;code&gt;gpg&lt;/code&gt; to sign commits. Plenty of people run exactly that setup, and I did for a long&amp;nbsp;time.&lt;/p&gt;
&lt;p&gt;My problem with it was never the cryptography. It was the moving parts. A Git signature goes&amp;nbsp;through &lt;code&gt;git&lt;/code&gt;, &lt;code&gt;gpg&lt;/code&gt;, &lt;code&gt;gpg-agent&lt;/code&gt; and &lt;code&gt;scdaemon&lt;/code&gt;, each with its own configuration and its own idea of which card is plugged in. When something breaks, the error tends to surface two processes away from the&amp;nbsp;cause. &lt;code&gt;scdaemon&lt;/code&gt; also likes to take exclusive control of the card, which gets in the way of every other program that wants to use it. On a laptop that suspends, resumes and sees cards come and go, I spent more time than I would like restarting&amp;nbsp;agents.&lt;/p&gt;
&lt;p&gt;The OpenPGP Card tools take a different&amp;nbsp;approach:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Each tool does one&amp;nbsp;job.&lt;/strong&gt; &lt;code&gt;oct&lt;/code&gt; manages&amp;nbsp;cards. &lt;code&gt;oct-git&lt;/code&gt; signs for Git. The &lt;span class="caps"&gt;SSH&lt;/span&gt; agent serves &lt;span class="caps"&gt;SSH&lt;/span&gt;. None of them tries to be a general-purpose OpenPGP&amp;nbsp;suite.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No long-lived card&amp;nbsp;connection.&lt;/strong&gt; &lt;code&gt;oct-git&lt;/code&gt; runs only when Git calls it and holds no permanent connection to the card. The &lt;span class="caps"&gt;SSH&lt;/span&gt; agent is a process, but it opens the card per operation and picks up cards that were plugged in after it&amp;nbsp;started.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;They talk &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; directly.&lt;/strong&gt; Underneath&amp;nbsp;is &lt;code&gt;pcscd&lt;/code&gt;, the same smart-card daemon every other card application on the system uses. There is no second card daemon in the&amp;nbsp;middle.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Shared &lt;span class="caps"&gt;PIN&lt;/span&gt; handling.&lt;/strong&gt; All of them get the User &lt;span class="caps"&gt;PIN&lt;/span&gt; through the same small&amp;nbsp;library, &lt;code&gt;openpgp-card-state&lt;/code&gt;, so you configure &lt;span class="caps"&gt;PIN&lt;/span&gt; storage once per card instead of once per&amp;nbsp;program.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Pure Rust.&lt;/strong&gt; The tools are built on &lt;a href="https://github.com/rpgp/rpgp"&gt;rPGP&lt;/a&gt; and RustCrypto, and they do not use any part of&amp;nbsp;GnuPG.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;GnuPG is still installed on my machines, and I still use it for other OpenPGP work. It just isn&amp;#8217;t in the path for &lt;span class="caps"&gt;SSH&lt;/span&gt; logins and commit signatures&amp;nbsp;anymore.&lt;/p&gt;
&lt;h2 id="what-you-need"&gt;What You&amp;nbsp;Need&lt;/h2&gt;
&lt;p&gt;The hardware is any OpenPGP card that your system can reach over &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt;. That includes Nitrokeys, YubiKeys in OpenPGP mode and classic OpenPGP smart cards in a &lt;span class="caps"&gt;USB&lt;/span&gt; reader. Supported algorithms depend on the&amp;nbsp;card. &lt;code&gt;oct info&lt;/code&gt; lists them. On the software side, the tools need the &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt;&amp;nbsp;daemon.&lt;/p&gt;
&lt;h3 id="on-freebsd"&gt;On&amp;nbsp;FreeBSD&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;pcsc-lite&lt;span class="w"&gt; &lt;/span&gt;libccid
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;pcscd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;pcscd&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If &lt;code&gt;oct list&lt;/code&gt; later finds your card as root but not as your normal user, don&amp;#8217;t go looking&amp;nbsp;in &lt;code&gt;devfs.rules&lt;/code&gt;. &lt;code&gt;pcscd&lt;/code&gt; runs as root and opens the reader itself, so device permissions are not the problem. The usual cause is the polkit check&amp;nbsp;in &lt;code&gt;pcscd&lt;/code&gt;, which turns away clients that are not in a session polkit considers active. You can allow your user with a polkit rule for&amp;nbsp;the &lt;code&gt;org.debian.pcsc-lite.access_pcsc&lt;/code&gt; and &lt;code&gt;org.debian.pcsc-lite.access_card&lt;/code&gt; actions&amp;nbsp;in &lt;code&gt;/usr/local/etc/polkit-1/rules.d/&lt;/code&gt;. The blunter option, which I use on my single-user laptop, is to&amp;nbsp;start &lt;code&gt;pcscd&lt;/code&gt; with &lt;code&gt;--disable-polkit&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;pcscd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--disable-polkit&amp;quot;&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;pcscd&lt;span class="w"&gt; &lt;/span&gt;restart
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That lets every local user talk to every card. Think about it before you copy it to a shared&amp;nbsp;machine.&lt;/p&gt;
&lt;p&gt;Then install the tools. Right now that means one package and two&amp;nbsp;ports:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools

&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/security/openpgp-card-tools-git&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;make&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;clean
&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/security/openpgp-card-ssh-agent&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;make&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;clean
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Once the other two packages appear&amp;nbsp;in &lt;code&gt;latest&lt;/code&gt;, this becomes one&amp;nbsp;line:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools-git&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-ssh-agent
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you build from ports,&amp;nbsp;install &lt;code&gt;rust&lt;/code&gt; and &lt;code&gt;cmake-core&lt;/code&gt; as packages&amp;nbsp;first. &lt;code&gt;cmake-core&lt;/code&gt; is only needed&amp;nbsp;for &lt;code&gt;openpgp-card-tools-git&lt;/code&gt;. Without the packages, the ports framework&amp;nbsp;builds &lt;code&gt;lang/rust&lt;/code&gt; from source, and that takes&amp;nbsp;hours.&lt;/p&gt;
&lt;p&gt;If you work from a graphical session, also install a graphical pinentry. I&amp;#8217;ll come back to why in the &lt;a href="#where-the-pin-lives"&gt;&lt;span class="caps"&gt;PIN&lt;/span&gt; section&lt;/a&gt;.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;pinentry-gtk2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="on-fedora"&gt;On&amp;nbsp;Fedora&lt;/h3&gt;
&lt;p&gt;Fedora&amp;nbsp;needs &lt;code&gt;pcscd&lt;/code&gt; running and, for now, the tools themselves from&amp;nbsp;Cargo:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;pcsc-lite&lt;span class="w"&gt; &lt;/span&gt;pcsc-lite-devel&lt;span class="w"&gt; &lt;/span&gt;rustc&lt;span class="w"&gt; &lt;/span&gt;cargo
sudo&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;pcscd.socket

cargo&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--locked&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools-git&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-ssh-agent
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The binaries land&amp;nbsp;in &lt;code&gt;~/.cargo/bin&lt;/code&gt;, which needs to be on&amp;nbsp;your &lt;code&gt;PATH&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="oct-managing-the-card"&gt;oct: Managing the&amp;nbsp;Card&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;oct&lt;/code&gt; is the administration tool. It is what I reach for when a new Nitrokey arrives, when I want to know which key sits in which slot, or when I need to change a &lt;span class="caps"&gt;PIN&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The first two commands you will&amp;nbsp;use:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;oct&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="c1"&gt;# idents of all connected cards, e.g. 000F:01234567&lt;/span&gt;
oct&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# cardholder, key fingerprints, algorithms, signature counter, PIN retries&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every card has an &lt;strong&gt;ident&lt;/strong&gt; made of a manufacturer &lt;span class="caps"&gt;ID&lt;/span&gt; and a serial number,&amp;nbsp;like &lt;code&gt;000F:01234567&lt;/code&gt;. When more than one card is connected, you select one&amp;nbsp;with &lt;code&gt;--card&lt;/code&gt;. The same ident shows up again later in the &lt;span class="caps"&gt;PIN&lt;/span&gt; configuration and&amp;nbsp;in &lt;code&gt;ssh-add -L&lt;/code&gt;, so it is worth knowing where to find&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;oct status -v&lt;/code&gt; adds the touch policy and key status for each&amp;nbsp;slot. &lt;code&gt;oct info&lt;/code&gt; shows what the card can do, including the supported&amp;nbsp;algorithms.&lt;/p&gt;
&lt;h3 id="setting-up-a-new-card"&gt;Setting Up a New&amp;nbsp;Card&lt;/h3&gt;
&lt;p&gt;A fresh OpenPGP card ships with the User &lt;span class="caps"&gt;PIN&lt;/span&gt; &lt;code&gt;123456&lt;/code&gt; and the Admin &lt;span class="caps"&gt;PIN&lt;/span&gt; &lt;code&gt;12345678&lt;/code&gt;. Change both&amp;nbsp;first:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;oct&lt;span class="w"&gt; &lt;/span&gt;pin&lt;span class="w"&gt; &lt;/span&gt;--card&lt;span class="w"&gt; &lt;/span&gt;000F:01234567&lt;span class="w"&gt; &lt;/span&gt;set-user
oct&lt;span class="w"&gt; &lt;/span&gt;pin&lt;span class="w"&gt; &lt;/span&gt;--card&lt;span class="w"&gt; &lt;/span&gt;000F:01234567&lt;span class="w"&gt; &lt;/span&gt;set-admin
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then give the card a key. You have two options, and they have different&amp;nbsp;trade-offs.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Generate the keys on the card.&lt;/strong&gt; The private key is created inside the hardware and can never leave&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;oct&lt;span class="w"&gt; &lt;/span&gt;admin&lt;span class="w"&gt; &lt;/span&gt;--card&lt;span class="w"&gt; &lt;/span&gt;000F:01234567&lt;span class="w"&gt; &lt;/span&gt;generate&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--userid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Alice Adams &amp;lt;alice@example.org&amp;gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--output&lt;span class="w"&gt; &lt;/span&gt;alice.pub&lt;span class="w"&gt; &lt;/span&gt;curve25519
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;--output&lt;/code&gt; writes the resulting public certificate. Other algorithm names&amp;nbsp;include &lt;code&gt;rsa4096&lt;/code&gt; and &lt;code&gt;nistp256&lt;/code&gt;, if the card supports them. The catch: if the card dies or gets lost, the key is gone. That is acceptable for an &lt;span class="caps"&gt;SSH&lt;/span&gt; authentication key that you can simply replace on your servers. It hurts more for a signing key that people have learned to&amp;nbsp;trust.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Import an existing key.&lt;/strong&gt; You create the key in software, keep an offline backup, and load it onto one or more&amp;nbsp;cards:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;oct&lt;span class="w"&gt; &lt;/span&gt;admin&lt;span class="w"&gt; &lt;/span&gt;--card&lt;span class="w"&gt; &lt;/span&gt;000F:01234567&lt;span class="w"&gt; &lt;/span&gt;import&lt;span class="w"&gt; &lt;/span&gt;key.priv
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If the key has more than one subkey per&amp;nbsp;role, &lt;code&gt;--sig-fp&lt;/code&gt;, &lt;code&gt;--dec-fp&lt;/code&gt; and &lt;code&gt;--auth-fp&lt;/code&gt; choose which subkey goes into which slot. This is how you get two cards with the same identity, one on your keyring and one in a&amp;nbsp;drawer.&lt;/p&gt;
&lt;h3 id="touch-confirmation"&gt;Touch&amp;nbsp;Confirmation&lt;/h3&gt;
&lt;p&gt;Many cards can require a physical touch before they use a key. I strongly recommend turning it on, at least for signing and&amp;nbsp;authentication:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;oct&lt;span class="w"&gt; &lt;/span&gt;admin&lt;span class="w"&gt; &lt;/span&gt;--card&lt;span class="w"&gt; &lt;/span&gt;000F:01234567&lt;span class="w"&gt; &lt;/span&gt;touch&lt;span class="w"&gt; &lt;/span&gt;--key&lt;span class="w"&gt; &lt;/span&gt;SIG&lt;span class="w"&gt; &lt;/span&gt;--policy&lt;span class="w"&gt; &lt;/span&gt;On
oct&lt;span class="w"&gt; &lt;/span&gt;admin&lt;span class="w"&gt; &lt;/span&gt;--card&lt;span class="w"&gt; &lt;/span&gt;000F:01234567&lt;span class="w"&gt; &lt;/span&gt;touch&lt;span class="w"&gt; &lt;/span&gt;--key&lt;span class="w"&gt; &lt;/span&gt;AUT&lt;span class="w"&gt; &lt;/span&gt;--policy&lt;span class="w"&gt; &lt;/span&gt;On
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The available policies depend on the&amp;nbsp;card. &lt;code&gt;Cached&lt;/code&gt; lets one touch cover several operations within 15 seconds, which is pleasant for an interactive rebase that signs a dozen commits.&amp;nbsp;The &lt;code&gt;Fixed&lt;/code&gt; variants cannot be changed again without a factory&amp;nbsp;reset.&lt;/p&gt;
&lt;p&gt;Touch is the part that protects you against a compromised host. More on that&amp;nbsp;below.&lt;/p&gt;
&lt;h3 id="the-public-side"&gt;The Public&amp;nbsp;Side&lt;/h3&gt;
&lt;p&gt;Two more commands give you the public half of the&amp;nbsp;card:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;oct&lt;span class="w"&gt; &lt;/span&gt;ssh&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="c1"&gt;# SSH public key of the authentication slot, for authorized_keys&lt;/span&gt;
oct&lt;span class="w"&gt; &lt;/span&gt;pubkey&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# OpenPGP certificate built from the card&amp;#39;s keys&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;oct ssh&lt;/code&gt; prints a line you can paste straight&amp;nbsp;into &lt;code&gt;~/.ssh/authorized_keys&lt;/code&gt; or into your Forgejo, Codeberg or GitHub&amp;nbsp;account. &lt;code&gt;oct pubkey&lt;/code&gt; reconstructs a certificate from what the card stores. The upstream documentation calls this export approximate, because the card doesn&amp;#8217;t store things like expiration dates. If you imported a key, your original certificate is the better one to&amp;nbsp;publish.&lt;/p&gt;
&lt;h2 id="where-the-pin-lives"&gt;Where the &lt;span class="caps"&gt;PIN&lt;/span&gt;&amp;nbsp;Lives&lt;/h2&gt;
&lt;p&gt;Before the other two tools make sense, you need to know how they get the &lt;span class="caps"&gt;PIN&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;GnuPG keeps the &lt;span class="caps"&gt;PIN&lt;/span&gt;&amp;nbsp;in &lt;code&gt;gpg-agent&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s cache. The OpenPGP Card tools hand that job to a small library called &lt;a href="https://codeberg.org/openpgp-card/state"&gt;openpgp-card-state&lt;/a&gt;, which&amp;nbsp;both &lt;code&gt;oct-git&lt;/code&gt; and the &lt;span class="caps"&gt;SSH&lt;/span&gt; agent use. Its configuration lives&amp;nbsp;in &lt;code&gt;~/.config/openpgp-card-state/config.toml&lt;/code&gt; and contains one entry per card, keyed by the&amp;nbsp;ident:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[[cards]]&lt;/span&gt;
&lt;span class="n"&gt;ident&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;000F:01234567&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;pin_storage&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Ephemeral&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;[[cards]]&lt;/span&gt;
&lt;span class="n"&gt;ident&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0006:12345678&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;pin_storage&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Keyring&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The backend decides where the &lt;span class="caps"&gt;PIN&lt;/span&gt; is&amp;nbsp;kept:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Backend&lt;/th&gt;
&lt;th&gt;Where the &lt;span class="caps"&gt;PIN&lt;/span&gt; lives&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Keyring&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The desktop secret store (Secret Service, macOS Keychain, Windows Credential Manager)&lt;/td&gt;
&lt;td&gt;The default. Survives reboots&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Ephemeral&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The Linux kernel keyring, expiring&amp;nbsp;after &lt;code&gt;pin_expiry&lt;/code&gt; seconds&lt;/td&gt;
&lt;td&gt;Linux only. Never written to disk&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Pinentry&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Nowhere. You get a prompt for every operation&lt;/td&gt;
&lt;td&gt;Works everywhere, no storage at all&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Direct&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;In plaintext&amp;nbsp;in &lt;code&gt;config.toml&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Meant for &lt;span class="caps"&gt;CI&lt;/span&gt; and tests&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;On Fedora I&amp;nbsp;use &lt;code&gt;Ephemeral&lt;/code&gt; for my Nitrokey&amp;nbsp;and &lt;code&gt;Keyring&lt;/code&gt; for a second card. On the FreeBSD laptop I&amp;nbsp;use &lt;code&gt;Pinentry&lt;/code&gt;,&amp;nbsp;because &lt;code&gt;Ephemeral&lt;/code&gt; is Linux-only and I don&amp;#8217;t run a Secret Service provider in my Xfce&amp;nbsp;session.&lt;/p&gt;
&lt;p&gt;It helps to think carefully about what the &lt;span class="caps"&gt;PIN&lt;/span&gt; protects. The upstream &lt;span class="caps"&gt;README&lt;/span&gt; is admirably honest about it: the User &lt;span class="caps"&gt;PIN&lt;/span&gt; mostly protects against someone who has stolen the card. Against malware running as your user on an unlocked desktop, a cached &lt;span class="caps"&gt;PIN&lt;/span&gt; is at best a weak defense, whichever backend holds it. &lt;strong&gt;Touch confirmation&lt;/strong&gt; is what actually stops a remote attacker, because malware cannot press the button on your Nitrokey. With touch enabled, a cached &lt;span class="caps"&gt;PIN&lt;/span&gt; becomes a reasonable convenience rather than a&amp;nbsp;risk.&lt;/p&gt;
&lt;h3 id="the-pinentry-trap-on-freebsd"&gt;The pinentry Trap on&amp;nbsp;FreeBSD&lt;/h3&gt;
&lt;p&gt;With&amp;nbsp;the &lt;code&gt;Pinentry&lt;/code&gt; backend, the tools call an executable&amp;nbsp;named &lt;code&gt;pinentry&lt;/code&gt; from &lt;code&gt;PATH&lt;/code&gt;. On&amp;nbsp;FreeBSD, &lt;code&gt;/usr/local/bin/pinentry&lt;/code&gt; is the curses variant. That works in a terminal, but it cannot prompt from a background agent, or from Git started by an editor. The result is an operation that just fails without a visible&amp;nbsp;prompt.&lt;/p&gt;
&lt;p&gt;The fix is to put a graphical pinentry first&amp;nbsp;on &lt;code&gt;PATH&lt;/code&gt; for these tools&amp;nbsp;only:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;~/.local/libexec/openpgp-card
ln&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bin/pinentry-gtk-2&lt;span class="w"&gt; &lt;/span&gt;~/.local/libexec/openpgp-card/pinentry
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The package is&amp;nbsp;called &lt;code&gt;pinentry-gtk2&lt;/code&gt; and the binary is&amp;nbsp;called &lt;code&gt;pinentry-gtk-2&lt;/code&gt;. The hyphen is not a&amp;nbsp;typo. &lt;code&gt;pinentry-qt&lt;/code&gt; works just as well if you prefer&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The agent&amp;#8217;s start script and the Git wrapper below both prepend that directory&amp;nbsp;to &lt;code&gt;PATH&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="oct-git-signing-commits-with-the-card"&gt;oct-git: Signing Commits With the&amp;nbsp;Card&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;oct-git&lt;/code&gt; is a replacement&amp;nbsp;for &lt;code&gt;gpg&lt;/code&gt; in exactly one role: being the program Git calls to create and verify OpenPGP signatures. It implements the small subset of GnuPG&amp;#8217;s command-line interface that Git uses, and nothing&amp;nbsp;more.&lt;/p&gt;
&lt;p&gt;Setting it up is four&amp;nbsp;lines:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;gpg.program&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bin/oct-git
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;user.signingkey&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;fingerprint&amp;gt;
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;commit.gpgsign&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
oct-git&lt;span class="w"&gt; &lt;/span&gt;--store-card-pin
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On Fedora with a Cargo install, the path&amp;nbsp;is &lt;code&gt;~/.cargo/bin/oct-git&lt;/code&gt; instead. &lt;code&gt;&amp;lt;fingerprint&amp;gt;&lt;/code&gt; is the fingerprint of your signing key, as shown&amp;nbsp;by &lt;code&gt;oct status&lt;/code&gt;, in hex without&amp;nbsp;spaces. &lt;code&gt;tag.gpgsign true&lt;/code&gt; does the same for&amp;nbsp;tags.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;--store-card-pin&lt;/code&gt; walks through all connected cards, checks whether a working &lt;span class="caps"&gt;PIN&lt;/span&gt; is already stored, and asks for one if not. With&amp;nbsp;the &lt;code&gt;Keyring&lt;/code&gt; or &lt;code&gt;Ephemeral&lt;/code&gt; backends, that means later signatures need no prompt. Press Enter to skip a card.&amp;nbsp;With &lt;code&gt;Pinentry&lt;/code&gt; there is nothing to store, so you get the prompt at signing time&amp;nbsp;instead.&lt;/p&gt;
&lt;p&gt;From then&amp;nbsp;on, &lt;code&gt;git commit&lt;/code&gt; signs with the card. When the card wants a&amp;nbsp;touch, &lt;code&gt;oct-git&lt;/code&gt; shows a desktop notification that&amp;nbsp;names &lt;code&gt;oct-git&lt;/code&gt; as the program asking. This sounds minor, but it makes a real difference. With a blinking Nitrokey and no explanation, you can&amp;#8217;t tell whether Git, &lt;span class="caps"&gt;SSH&lt;/span&gt; or something else wants the key. With the notification, you know what you are&amp;nbsp;confirming.&lt;/p&gt;
&lt;p&gt;Switching back to GnuPG is just a matter of&amp;nbsp;pointing &lt;code&gt;gpg.program&lt;/code&gt; at &lt;code&gt;gpg&lt;/code&gt; again. Your repositories don&amp;#8217;t care which program made the&amp;nbsp;signature.&lt;/p&gt;
&lt;h3 id="verifying-signatures"&gt;Verifying&amp;nbsp;Signatures&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;oct-git&lt;/code&gt; also verifies,&amp;nbsp;so &lt;code&gt;git log --show-signature&lt;/code&gt; and &lt;code&gt;git verify-commit&lt;/code&gt; work. It looks up the signer&amp;#8217;s certificate in the shared &lt;a href="https://datatracker.ietf.org/doc/draft-nwjh-openpgp-cert-d/"&gt;openpgp-cert-d&lt;/a&gt; directory. If it is not there, it tries keys.openpgp.org, pgpkeys.eu and keyserver.ubuntu.com. For certificates that are not on any keyserver, import them by&amp;nbsp;hand:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;oct-git&lt;span class="w"&gt; &lt;/span&gt;--import&lt;span class="w"&gt; &lt;/span&gt;colleague.pub
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two limitations are worth&amp;nbsp;knowing. &lt;code&gt;oct-git&lt;/code&gt; cannot yet update certificates that are already in the store. And it has no notion of trust levels, so a valid signature is reported with an &amp;#8220;Unverified&amp;#8221; trust level. The signature is cryptographically valid and the output shows the issuer&amp;#8217;s fingerprint, but deciding whether that fingerprint belongs to the right person is still up to you. For signing my own work, neither limitation matters to&amp;nbsp;me.&lt;/p&gt;
&lt;h3 id="the-freebsd-wrapper"&gt;The FreeBSD&amp;nbsp;Wrapper&lt;/h3&gt;
&lt;p&gt;On FreeBSD with&amp;nbsp;the &lt;code&gt;Pinentry&lt;/code&gt; backend, Git needs to find the graphical pinentry too. I&amp;nbsp;point &lt;code&gt;gpg.program&lt;/code&gt; at a three-line wrapper instead of the&amp;nbsp;binary:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# ~/.local/bin/oct-git-freebsd&lt;/span&gt;
&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.local/libexec/openpgp-card:&lt;/span&gt;&lt;span class="nv"&gt;$PATH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;PATH
&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bin/oct-git&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;700&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;~/.local/bin/oct-git-freebsd
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;gpg.program&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.local/bin/oct-git-freebsd&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="openpgp-card-ssh-agent-ssh-keys-that-cannot-be-copied"&gt;openpgp-card-ssh-agent: &lt;span class="caps"&gt;SSH&lt;/span&gt; Keys That Cannot Be&amp;nbsp;Copied&lt;/h2&gt;
&lt;p&gt;The third tool&amp;nbsp;replaces &lt;code&gt;ssh-agent&lt;/code&gt; (or &lt;code&gt;gpg-agent --enable-ssh-support&lt;/code&gt;) with an agent that offers the key in each connected card&amp;#8217;s authentication slot. When &lt;span class="caps"&gt;SSH&lt;/span&gt; needs a signature, the agent performs it on the card. The private key never exists anywhere else, so there is&amp;nbsp;no &lt;code&gt;~/.ssh/id_ed25519&lt;/code&gt; to steal, back up by accident or leave on an old&amp;nbsp;laptop.&lt;/p&gt;
&lt;p&gt;The agent supports &lt;span class="caps"&gt;RSA&lt;/span&gt;, &lt;span class="caps"&gt;NIST&lt;/span&gt; P-256/384/521 and Curve25519 keys. Apart from the PINs, it needs no configuration and keeps no state. Cards can be plugged in after it starts. It finds them on the next operation, which is exactly what I want on a&amp;nbsp;laptop.&lt;/p&gt;
&lt;h3 id="running-it"&gt;Running&amp;nbsp;It&lt;/h3&gt;
&lt;p&gt;The agent needs a socket to listen&amp;nbsp;on:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;openpgp-card-ssh-agent&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;unix://&lt;/span&gt;&lt;span class="nv"&gt;$XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="s2"&gt;/openpgp-card/ssh-agent.sock&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Point &lt;span class="caps"&gt;SSH&lt;/span&gt; at&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;SSH_AUTH_SOCK&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="s2"&gt;/openpgp-card/ssh-agent.sock&amp;quot;&lt;/span&gt;
ssh-add&lt;span class="w"&gt; &lt;/span&gt;-L
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;ssh-add -L&lt;/code&gt; lists the public key of every connected card, with the card&amp;#8217;s ident as the comment. To store a &lt;span class="caps"&gt;PIN&lt;/span&gt; for a card,&amp;nbsp;use &lt;code&gt;ssh-add -s&lt;/code&gt; with that&amp;nbsp;ident:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ssh-add&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;000F:01234567
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; will ask for a &amp;#8220;passphrase for &lt;span class="caps"&gt;PKCS&lt;/span&gt;#11&amp;#8221;. That wording is hard-coded in OpenSSH and has nothing to do with &lt;span class="caps"&gt;PKCS&lt;/span&gt;#11 here. Enter the card&amp;#8217;s User &lt;span class="caps"&gt;PIN&lt;/span&gt;. The agent tests it against the card and only saves it if the card accepts&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;If you don&amp;#8217;t want the card agent for everything, select it per host&amp;nbsp;in &lt;code&gt;~/.ssh/config&lt;/code&gt; instead of&amp;nbsp;setting &lt;code&gt;SSH_AUTH_SOCK&lt;/code&gt; globally:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Host git.example.org
    IdentityAgent /var/run/xdg/&amp;lt;username&amp;gt;/openpgp-card/ssh-agent.sock
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the runtime directory on my FreeBSD laptop. On Fedora it&amp;nbsp;is &lt;code&gt;/run/user/&amp;lt;uid&amp;gt;/&lt;/code&gt;. Use&amp;nbsp;whatever &lt;code&gt;echo $XDG_RUNTIME_DIR&lt;/code&gt; prints on your&amp;nbsp;machine.&lt;/p&gt;
&lt;h3 id="on-fedora-systemd-socket-activation"&gt;On Fedora: systemd Socket&amp;nbsp;Activation&lt;/h3&gt;
&lt;p&gt;On Linux the agent can be socket-activated&amp;nbsp;by &lt;code&gt;systemd --user&lt;/code&gt;. The upstream repository ships unit files in&amp;nbsp;its &lt;code&gt;contrib&lt;/code&gt; directory. I use them almost unchanged. The socket&amp;nbsp;unit, &lt;code&gt;~/.config/systemd/user/openpgp-card-ssh-agent.socket&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;A simple ssh-agent backed by OpenPGP card authentication keys&lt;/span&gt;

&lt;span class="k"&gt;[Socket]&lt;/span&gt;
&lt;span class="na"&gt;ListenStream&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;%t/openpgp-card/ssh-agent.sock&lt;/span&gt;
&lt;span class="na"&gt;SocketMode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;0600&lt;/span&gt;
&lt;span class="na"&gt;DirectoryMode&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;0700&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;sockets.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And the&amp;nbsp;service, &lt;code&gt;~/.config/systemd/user/openpgp-card-ssh-agent.service&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;A simple ssh-agent backed by OpenPGP card authentication keys&lt;/span&gt;
&lt;span class="na"&gt;Requires&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;openpgp-card-ssh-agent.socket&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;%h/.cargo/bin/openpgp-card-ssh-agent -H fd://&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;-H fd://&lt;/code&gt; tells the agent to take the listening socket that systemd hands&amp;nbsp;over. &lt;code&gt;%t&lt;/code&gt; is the runtime directory. Enable the socket and export the path in your&amp;nbsp;shell:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-ssh-agent.socket
&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;SSH_AUTH_SOCK&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="s2"&gt;/openpgp-card/ssh-agent.sock&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The agent now starts the first time something connects to the&amp;nbsp;socket.&lt;/p&gt;
&lt;p&gt;If you use&amp;nbsp;the &lt;code&gt;Keyring&lt;/code&gt; backend, the agent needs a Secret Service provider to be running. My units also&amp;nbsp;list &lt;code&gt;gnome-keyring-daemon.service&lt;/code&gt; in &lt;code&gt;Requires=&lt;/code&gt;, because gnome-keyring is my Secret Service provider. Do not copy that line blindly. On a stock &lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma installation, KWallet provides Secret Service and there is&amp;nbsp;no &lt;code&gt;gnome-keyring-daemon.service&lt;/code&gt;.&amp;nbsp;A &lt;code&gt;Requires=&lt;/code&gt; on a unit that doesn&amp;#8217;t exist will stop the agent from&amp;nbsp;starting.&lt;/p&gt;
&lt;p&gt;An &lt;code&gt;export&lt;/code&gt; in your shell only reaches programs started from that shell. A Git &lt;span class="caps"&gt;GUI&lt;/span&gt; or an editor launched from the desktop menu won&amp;#8217;t see it. On&amp;nbsp;Fedora, &lt;code&gt;environment.d&lt;/code&gt; sets the variable for the whole user session instead. Put this&amp;nbsp;in &lt;code&gt;~/.config/environment.d/50-openpgp-card-ssh-agent.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;SSH_AUTH_SOCK&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;/openpgp-card/ssh-agent.sock
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;GNOME&lt;/span&gt; and Plasma both pick this up at the next login. Fish users who still want it in the shell&amp;nbsp;write &lt;code&gt;set -gx SSH_AUTH_SOCK $XDG_RUNTIME_DIR/openpgp-card/ssh-agent.sock&lt;/code&gt; instead of&amp;nbsp;the &lt;code&gt;export&lt;/code&gt; line.&lt;/p&gt;
&lt;p&gt;If your machine has a second smart card stack that the agent should not touch, I wrote about &lt;a href="https://hofstede.it/card-wars-hiding-smartcard-readers-from-eager-rust-agents-with-ld_preload/"&gt;hiding a reader from the agent with LD_PRELOAD&lt;/a&gt;. That drops into this same service as an&amp;nbsp;override.&lt;/p&gt;
&lt;h3 id="on-freebsd-an-xfce-autostart-entry"&gt;On FreeBSD: An Xfce Autostart&amp;nbsp;Entry&lt;/h3&gt;
&lt;p&gt;FreeBSD has no user session manager&amp;nbsp;like &lt;code&gt;systemd --user&lt;/code&gt;, so on the T480 the agent starts with the Xfce session.&amp;nbsp;The &lt;code&gt;Exec&lt;/code&gt; line of&amp;nbsp;a &lt;code&gt;.desktop&lt;/code&gt; file doesn&amp;#8217;t expand shell variables, so the autostart entry calls a small&amp;nbsp;script:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# ~/.local/bin/start-openpgp-card-ssh-agent&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-eu

:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="p"&gt;:?XDG_RUNTIME_DIR must be set by the desktop session&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;socket_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="s2"&gt;/openpgp-card&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;socket&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket_dir&lt;/span&gt;&lt;span class="s2"&gt;/ssh-agent.sock&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;umask&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;077&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket_dir&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;700&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket_dir&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# The Rust card tools look for an executable named pinentry on PATH.&lt;/span&gt;
&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.local/libexec/openpgp-card:&lt;/span&gt;&lt;span class="nv"&gt;$PATH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;PATH

&lt;span class="c1"&gt;# A socket can survive a logout or an agent crash. Only keep it if&lt;/span&gt;
&lt;span class="c1"&gt;# an agent actually answers on it. ssh-add -l exits with 2 when it&lt;/span&gt;
&lt;span class="c1"&gt;# cannot reach an agent, and with 0 or 1 when one is listening.&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;rc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;SSH_AUTH_SOCK&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ssh-add&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;rc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$rc&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-ne&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Refusing to replace unexpected path: &lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bin/openpgp-card-ssh-agent&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;unix://&lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The autostart&amp;nbsp;entry, &lt;code&gt;~/.config/autostart/openpgp-card-ssh-agent.desktop&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Desktop Entry]&lt;/span&gt;
&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Application&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;OpenPGP Card SSH Agent&lt;/span&gt;
&lt;span class="na"&gt;Comment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Provide SSH authentication through the OpenPGP card&lt;/span&gt;
&lt;span class="na"&gt;Exec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/home/&amp;lt;username&amp;gt;/.local/bin/start-openpgp-card-ssh-agent&lt;/span&gt;
&lt;span class="na"&gt;TryExec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/home/&amp;lt;username&amp;gt;/.local/bin/start-openpgp-card-ssh-agent&lt;/span&gt;
&lt;span class="na"&gt;Terminal&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;false&lt;/span&gt;
&lt;span class="na"&gt;OnlyShowIn&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;XFCE&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;700&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;~/.local/bin/start-openpgp-card-ssh-agent
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Other desktops have equivalent autostart mechanisms. The script itself doesn&amp;#8217;t depend on&amp;nbsp;Xfce.&lt;/p&gt;
&lt;h2 id="a-day-with-the-stack"&gt;A Day With the&amp;nbsp;Stack&lt;/h2&gt;
&lt;p&gt;Once it is set up, there is very little to see, which is the point. My day on either machine looks like&amp;nbsp;this:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;I log in. The agent socket is there, from systemd on Fedora or from the autostart entry on&amp;nbsp;FreeBSD.&lt;/li&gt;
&lt;li&gt;I plug in the Nitrokey. Nothing happens yet, because nothing needs&amp;nbsp;to.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ssh&lt;/code&gt; to a server&amp;nbsp;or &lt;code&gt;git push&lt;/code&gt; to Forgejo. The agent finds the card, the card blinks, I touch it and I&amp;#8217;m&amp;nbsp;in.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;git commit&lt;/code&gt;. &lt;code&gt;oct-git&lt;/code&gt; signs, a notification tells me who is asking, I touch the card, and the commit carries a&amp;nbsp;signature.&lt;/li&gt;
&lt;li&gt;I unplug the card and walk away. Without it, nobody signs or logs in as me, including anything already running on the&amp;nbsp;laptop.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The same card, the same keys and nearly the same configuration work on both operating systems. The only real differences are how the agent starts and where the &lt;span class="caps"&gt;PIN&lt;/span&gt; is kept. My dotfiles handle those with a few template conditions in &lt;a href="https://hofstede.it/chezmoi-and-git-dotfiles-without-pretending-every-machine-is-the-same/"&gt;chezmoi&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id="troubleshooting"&gt;Troubleshooting&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;oct list&lt;/code&gt; shows no card.&lt;/strong&gt; Check&amp;nbsp;that &lt;code&gt;pcscd&lt;/code&gt; is running. Then check for&amp;nbsp;GnuPG&amp;#8217;s &lt;code&gt;scdaemon&lt;/code&gt;, which can hold the card exclusively. On my&amp;nbsp;T480, &lt;code&gt;oct list&lt;/code&gt; found nothing&amp;nbsp;while &lt;code&gt;scdaemon&lt;/code&gt; was running and found the card immediately after&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gpgconf&lt;span class="w"&gt; &lt;/span&gt;--kill&lt;span class="w"&gt; &lt;/span&gt;scdaemon
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Killing it is only a temporary fix. If GnuPG is still installed, the&amp;nbsp;next &lt;code&gt;gpg&lt;/code&gt; invocation can&amp;nbsp;start &lt;code&gt;scdaemon&lt;/code&gt; again and take the card back. That includes an editor plugin quietly&amp;nbsp;running &lt;code&gt;gpg --list-keys&lt;/code&gt;. If you never use GnuPG with the card,&amp;nbsp;tell &lt;code&gt;gpg-agent&lt;/code&gt; not to&amp;nbsp;start &lt;code&gt;scdaemon&lt;/code&gt; at&amp;nbsp;all:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;disable-scdaemon&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;~/.gnupg/gpg-agent.conf
gpgconf&lt;span class="w"&gt; &lt;/span&gt;--kill&lt;span class="w"&gt; &lt;/span&gt;gpg-agent
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you still need GnuPG&amp;#8217;s card support now and then,&amp;nbsp;put &lt;code&gt;pcsc-shared&lt;/code&gt; in &lt;code&gt;~/.gnupg/scdaemon.conf&lt;/code&gt; instead.&amp;nbsp;Then &lt;code&gt;scdaemon&lt;/code&gt; shares the reader with other &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; clients and doesn&amp;#8217;t lock them&amp;nbsp;out.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Signing or &lt;span class="caps"&gt;SSH&lt;/span&gt; fails without any prompt.&lt;/strong&gt; On FreeBSD this is almost always the&amp;nbsp;curses &lt;code&gt;pinentry&lt;/code&gt; being called from a context that has no terminal. Check that the graphical pinentry is first on&amp;nbsp;the &lt;code&gt;PATH&lt;/code&gt; the tool actually sees, not just the one in your&amp;nbsp;shell.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Nothing happens and the card is blinking.&lt;/strong&gt; It&amp;#8217;s waiting for a touch. That&amp;#8217;s the system working as&amp;nbsp;intended.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Debug output from the agent.&lt;/strong&gt; Start it&amp;nbsp;with &lt;code&gt;RUST_LOG=debug&lt;/code&gt; to see what it does with each&amp;nbsp;request.&lt;/p&gt;
&lt;h2 id="porting-notes"&gt;Porting&amp;nbsp;Notes&lt;/h2&gt;
&lt;p&gt;The ports themselves were not dramatic. These are Rust programs with a single real system&amp;nbsp;dependency, &lt;code&gt;pcsc-lite&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;USES=cargo&lt;/code&gt; in the FreeBSD ports framework does most of the work. Each port carries&amp;nbsp;a &lt;code&gt;Makefile.crates&lt;/code&gt; listing every crate it vendors, so the build is reproducible and runs without network&amp;nbsp;access.&lt;/p&gt;
&lt;p&gt;The time-consuming part was everything around the binaries. That meant working out the pinentry behavior and figuring out how to start an agent&amp;nbsp;without &lt;code&gt;systemd --user&lt;/code&gt;. The &lt;a href="https://codeberg.org/Larvitz/freebsd-openpgp-card-ports"&gt;port repository&amp;#8217;s &lt;span class="caps"&gt;README&lt;/span&gt;&lt;/a&gt; collects those&amp;nbsp;notes.&lt;/p&gt;
&lt;p&gt;Credit for the tools goes to Heiko Schaefer and Wiktor Kwapisiewicz and the rest of the OpenPGP Card project. They built a set of small, understandable programs for a job that used to require a large, opaque stack. All I did was make them easy to install on one more operating&amp;nbsp;system.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Hardware-backed keys only work in practice if they are less annoying than the alternative. For me, the Rust OpenPGP Card stack is what got&amp;nbsp;there. &lt;code&gt;oct&lt;/code&gt; handles the&amp;nbsp;card, &lt;code&gt;oct-git&lt;/code&gt; signs my commits&amp;nbsp;and &lt;code&gt;openpgp-card-ssh-agent&lt;/code&gt; replaces my &lt;span class="caps"&gt;SSH&lt;/span&gt; keys. Each tool does one job, none of them fights over the card, and all three share a single &lt;span class="caps"&gt;PIN&lt;/span&gt;&amp;nbsp;configuration.&lt;/p&gt;
&lt;p&gt;If you have a Nitrokey or an OpenPGP card in a drawer because the GnuPG setup felt like too much, give these a try. On FreeBSD it now starts&amp;nbsp;with &lt;code&gt;pkg install openpgp-card-tools&lt;/code&gt;, and soon the other two packages will be there as&amp;nbsp;well.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="linux"/><category term="fedora"/><category term="openpgp"/><category term="smartcard"/><category term="nitrokey"/><category term="ssh"/><category term="git"/><category term="rust"/><category term="ports"/><category term="pkg"/><category term="security"/></entry><entry><title>php2shell on arm64: Building Fixed PHP Packages Without Poudriere</title><link href="https://hofstede.it/php2shell-on-arm64-building-fixed-php-packages-without-poudriere/" rel="alternate"/><published>2026-10-22T00:00:00+02:00</published><updated>2026-10-22T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-10-22:/php2shell-on-arm64-building-fixed-php-packages-without-poudriere/</id><summary type="html">&lt;p&gt;The php2shell fix landed in the FreeBSD ports tree within a day, but packages had only been built for some &lt;span class="caps"&gt;CPU&lt;/span&gt; architectures, and my arm64 jail host was not among them. Instead of setting up Poudriere for a one-off job, I built the fixed &lt;span class="caps"&gt;PHP&lt;/span&gt; 8.5.12 in a throwaway Bastille jail, turned the result into a tiny local pkg repository and upgraded the affected jails from&amp;nbsp;it.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Diagram in three panels: the lang/php85 port at 8.5.12_1 on the 2026Q4 branch, a Bastille build jail running make package install clean on FreeBSD 15.1 aarch64, and a local pkg repository at file:///var/tmp/php85-local" src="https://hofstede.it/images/2026-10-22-php2shell-freebsd-arm64-packages-without-poudriere.webp" title="Patched port, throwaway build jail, local pkg repository: 32 packages, no build farm."&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-not-just-use-poudriere"&gt;Why Not Just Use&amp;nbsp;Poudriere?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-build-jail"&gt;The Build&amp;nbsp;Jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-ports-tree-on-the-right-branch"&gt;A Ports Tree on the Right&amp;nbsp;Branch&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#which-ports-which-options"&gt;Which Ports, Which&amp;nbsp;Options?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#building-the-packages"&gt;Building the Packages&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-mistake-where-did-my-packages-go"&gt;The Mistake: Where Did My Packages&amp;nbsp;Go?&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-local-repository"&gt;A Local&amp;nbsp;Repository&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#upgrading-the-target-jails"&gt;Upgrading the Target Jails&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#surprise-guests-in-the-second-jail"&gt;Surprise Guests in the Second&amp;nbsp;Jail&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#switching-back-to-official-packages"&gt;Switching Back to Official&amp;nbsp;Packages&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#pitfalls-the-checklist"&gt;Pitfalls: The&amp;nbsp;Checklist&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;On 2026-10-21 the &lt;span class="caps"&gt;PHP&lt;/span&gt; project released fixes for two critical vulnerabilities nicknamed &lt;strong&gt;php2shell&lt;/strong&gt; (&lt;span class="caps"&gt;CVE&lt;/span&gt;-2026-109618 and &lt;span class="caps"&gt;CVE&lt;/span&gt;-2026-109619, both &lt;span class="caps"&gt;CVSS&lt;/span&gt; 9.2). Every &lt;span class="caps"&gt;PHP&lt;/span&gt; version since &lt;span class="caps"&gt;PHP&lt;/span&gt; 5 is affected. The fixed releases are 8.5.12, 8.4.27, 8.3.36 and&amp;nbsp;8.2.35.&lt;/p&gt;
&lt;p&gt;My jail host runs FreeBSD 15.1 on &lt;strong&gt;arm64&lt;/strong&gt;, and three of its Bastille jails run &lt;span class="caps"&gt;PHP&lt;/span&gt;: a looking glass for my network, a DokuWiki and a Nextcloud. I stopped all three the same day and disabled their autostart. The plan was to wait for the fixed&amp;nbsp;packages.&lt;/p&gt;
&lt;p&gt;The FreeBSD side was&amp;nbsp;quick. &lt;code&gt;lang/php85&lt;/code&gt; was updated to 8.5.12 in the ports tree within a day,&amp;nbsp;in &lt;code&gt;main&lt;/code&gt; as well as in&amp;nbsp;the &lt;code&gt;2026Q4&lt;/code&gt; quarterly branch. But a fix in the ports tree is not a package yet. The package builders had only published fixed packages for some &lt;span class="caps"&gt;CPU&lt;/span&gt; architectures so far, and a day&amp;nbsp;later &lt;code&gt;pkg&lt;/code&gt; on my arm64 host still offered&amp;nbsp;8.5.11.&lt;/p&gt;
&lt;p&gt;So I built the packages myself. Without&amp;nbsp;Poudriere.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The short version:&lt;/strong&gt; a throwaway thin jail with the same release as the target jails, a&amp;nbsp;fresh &lt;code&gt;2026Q4&lt;/code&gt; ports checkout inside&amp;nbsp;it, &lt;code&gt;make package&lt;/code&gt; for &lt;span class="caps"&gt;PHP&lt;/span&gt; and every core extension the jails&amp;nbsp;use, &lt;code&gt;pkg repo&lt;/code&gt; over the results, and a local repository with a higher priority inside each target jail. Two of the three jails were back online the same&amp;nbsp;evening.&lt;/p&gt;
&lt;h2 id="why-not-just-use-poudriere"&gt;Why Not Just Use&amp;nbsp;Poudriere?&lt;/h2&gt;
&lt;p&gt;Poudriere is the right tool for building FreeBSD packages. If I built packages regularly, I would use it, and I would not be writing this&amp;nbsp;article.&lt;/p&gt;
&lt;p&gt;For a one-off job it is a lot of machinery: its own &lt;span class="caps"&gt;ZFS&lt;/span&gt; datasets, a build jail, a ports tree it manages itself, configuration, and by default it builds the entire dependency tree from source. I needed 32 small packages, once, and I wanted to get rid of everything&amp;nbsp;afterwards.&lt;/p&gt;
&lt;p&gt;The other options I&amp;nbsp;considered:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Keep waiting.&lt;/strong&gt; That was my first decision. But three services were offline, and I did not know when the arm64 builders would catch&amp;nbsp;up.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Build on the jail host itself.&lt;/strong&gt; The ports tree on my host turned out to be a tarball from June 2025, not a git checkout, and it did not even&amp;nbsp;know &lt;code&gt;lang/php85&lt;/code&gt;. Building there would also have pulled a pile of build dependencies onto a host that is otherwise managed with pkgbase and has very few packages installed. I like it that&amp;nbsp;way.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Build inside each target jail with the ports tree.&lt;/strong&gt; That puts compilers and build dependencies into internet-facing web jails and builds the same thing three&amp;nbsp;times.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A throwaway jail keeps everything that is needed for the build in one place I can delete with a single&amp;nbsp;command.&lt;/p&gt;
&lt;h2 id="the-build-jail"&gt;The Build&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;The target jails are Bastille thin jails on 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;, so the build jail is one too. Same release, same base system, same &lt;span class="caps"&gt;ABI&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-B&lt;span class="w"&gt; &lt;/span&gt;phpbuild&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.1-RELEASE&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.99/24&lt;span class="w"&gt; &lt;/span&gt;bastille0
bastille&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;phpbuild&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;boot&lt;span class="w"&gt; &lt;/span&gt;off
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;-B&lt;/code&gt; attaches the jail to the&amp;nbsp;existing &lt;code&gt;bastille0&lt;/code&gt; bridge like all my other jails. The jail network is NATed as a whole, so a free address is all the jail needs to reach the internet. The default route comes&amp;nbsp;from &lt;code&gt;bastille_network_gateway&lt;/code&gt; in&amp;nbsp;my &lt;code&gt;bastille.conf&lt;/code&gt;. Without that setting,&amp;nbsp;add &lt;code&gt;-g&lt;/code&gt; with your gateway to&amp;nbsp;the &lt;code&gt;create&lt;/code&gt; command. &lt;code&gt;boot off&lt;/code&gt; makes sure it does not come back after a reboot by&amp;nbsp;accident.&lt;/p&gt;
&lt;p&gt;One small trap: Bastille copies the&amp;nbsp;host&amp;#8217;s &lt;code&gt;/etc/resolv.conf&lt;/code&gt; into the jail. My host only has IPv6 resolvers, and the build jail only has IPv4. So it gets its own&amp;nbsp;resolver:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;nameserver 9.9.9.9&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bastille/jails/phpbuild/root/etc/resolv.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="a-ports-tree-on-the-right-branch"&gt;A Ports Tree on the Right&amp;nbsp;Branch&lt;/h2&gt;
&lt;p&gt;This is the step that matters most. Check which branch your jails get their packages&amp;nbsp;from:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# in one of the target jails&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;-vv&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-A1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;FreeBSD-ports:&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;FreeBSD-ports: {
    url             : &amp;quot;pkg+https://pkg.FreeBSD.org/FreeBSD:15:aarch64/quarterly&amp;quot;,
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;My jails&amp;nbsp;use &lt;code&gt;quarterly&lt;/code&gt;, so the ports tree has to be the matching quarterly&amp;nbsp;branch, &lt;code&gt;2026Q4&lt;/code&gt;. The ports tree decides which dependencies a port needs and in which versions, while the binary dependencies come from the quarterly repository.&amp;nbsp;A &lt;code&gt;main&lt;/code&gt; checkout can require newer dependency versions than the quarterly repository has, and packages built from it can end up depending on versions the target jails cannot get from their repository. Matching the branches avoids that&amp;nbsp;mismatch.&lt;/p&gt;
&lt;p&gt;From here on, the commands run inside the build jail until I say&amp;nbsp;otherwise:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;console&lt;span class="w"&gt; &lt;/span&gt;phpbuild

&lt;span class="c1"&gt;# build jail&lt;/span&gt;
env&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;ASSUME_ALWAYS_YES&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;yes&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;bootstrap&lt;span class="w"&gt; &lt;/span&gt;-f
pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;git-tiny
git&lt;span class="w"&gt; &lt;/span&gt;clone&lt;span class="w"&gt; &lt;/span&gt;--depth&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;2026Q4&lt;span class="w"&gt; &lt;/span&gt;https://git.FreeBSD.org/ports.git&lt;span class="w"&gt; &lt;/span&gt;/usr/ports
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Check that the fix is really&amp;nbsp;there:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# build jail&lt;/span&gt;
make&lt;span class="w"&gt; &lt;/span&gt;-C&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/lang/php85&lt;span class="w"&gt; &lt;/span&gt;-V&lt;span class="w"&gt; &lt;/span&gt;PKGNAME
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;php85-8.5.12_1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And compare the checksum&amp;nbsp;in &lt;code&gt;distinfo&lt;/code&gt; with the one php.net publishes. I want to know that the tarball the port will fetch is the one the &lt;span class="caps"&gt;PHP&lt;/span&gt; project&amp;nbsp;released:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# build jail&lt;/span&gt;
cat&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/lang/php85/distinfo
fetch&lt;span class="w"&gt; &lt;/span&gt;-qo&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;https://www.php.net/releases/index.php?json&amp;amp;version=8.5.12&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Both&amp;nbsp;said &lt;code&gt;08bc2eba2a588654ade1357174aa646b761556c862dda0678345ee29ed481209&lt;/code&gt; for &lt;code&gt;php-8.5.12.tar.xz&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="which-ports-which-options"&gt;Which Ports, Which&amp;nbsp;Options?&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;PHP&lt;/span&gt; on FreeBSD is split into many&amp;nbsp;packages. &lt;code&gt;lang/php85&lt;/code&gt; is the interpreter, and every core extension&amp;nbsp;(&lt;code&gt;php85-mbstring&lt;/code&gt;, &lt;code&gt;php85-gd&lt;/code&gt;, &lt;code&gt;php85-pdo_pgsql&lt;/code&gt;, &amp;#8230;) is its own port built from the same &lt;span class="caps"&gt;PHP&lt;/span&gt; tarball. All of them should go to 8.5.12, not just the&amp;nbsp;interpreter.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;PECL&lt;/span&gt; extensions such as APCu or imagick have their own version numbers and do not need to be rebuilt. The same goes for&amp;nbsp;the &lt;code&gt;php85-extensions&lt;/code&gt; meta port and applications&amp;nbsp;like &lt;code&gt;dokuwiki-php85&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;To get the list, I went back to the host and asked the stopped jails&amp;nbsp;directly. &lt;code&gt;pkg -r&lt;/code&gt; works on a jail&amp;#8217;s root directory without starting&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;j&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;lg&lt;span class="w"&gt; &lt;/span&gt;nextcloud&lt;span class="w"&gt; &lt;/span&gt;wiki&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bastille/jails/&lt;span class="nv"&gt;$j&lt;/span&gt;/root&lt;span class="w"&gt; &lt;/span&gt;query&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%o %v&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;php85*&amp;#39;&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;$2 ~ /^8\.5\.11/ {print $1}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bastille/jails/phpbuild/root/tmp/ports.txt
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Writing it straight into the build&amp;nbsp;jail&amp;#8217;s &lt;code&gt;/tmp&lt;/code&gt; saves a copy step later. That gave&amp;nbsp;me &lt;code&gt;lang/php85&lt;/code&gt; plus 31 core extensions. The list is sorted alphabetically, which is not the order they can be built in, so I used it for the checks below and ordered the build by&amp;nbsp;hand.&lt;/p&gt;
&lt;p&gt;Next, the options. My jails run the official packages, which are built with the port defaults, so the defaults should match. I checked anyway by comparing the options recorded in the installed packages with what the port would&amp;nbsp;build:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host: options that are ON in the installed packages&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;j&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;lg&lt;span class="w"&gt; &lt;/span&gt;nextcloud&lt;span class="w"&gt; &lt;/span&gt;wiki&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bastille/jails/&lt;span class="nv"&gt;$j&lt;/span&gt;/root&lt;span class="w"&gt; &lt;/span&gt;query&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%n %Ok %Ov&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;php85*&amp;#39;&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;$3 == &amp;quot;on&amp;quot; {print $1, $2}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bastille/jails/phpbuild/root/tmp/opts-installed.txt

&lt;span class="c1"&gt;# build jail: options the port would select&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;origin&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;make&lt;span class="w"&gt; &lt;/span&gt;-C&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/&lt;span class="nv"&gt;$origin&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-V&lt;span class="w"&gt; &lt;/span&gt;PKGBASE&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;make&lt;span class="w"&gt; &lt;/span&gt;-C&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/&lt;span class="nv"&gt;$origin&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-V&lt;span class="w"&gt; &lt;/span&gt;SELECTED_OPTIONS&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$o&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;/tmp/ports.txt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/tmp/opts-port.txt

diff&lt;span class="w"&gt; &lt;/span&gt;/tmp/opts-installed.txt&lt;span class="w"&gt; &lt;/span&gt;/tmp/opts-port.txt
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The only difference was&amp;nbsp;the &lt;code&gt;php85-extensions&lt;/code&gt; meta port, which I was not rebuilding anyway. So:&amp;nbsp;no &lt;code&gt;make config&lt;/code&gt;,&amp;nbsp;just &lt;code&gt;BATCH=yes&lt;/code&gt; and the&amp;nbsp;defaults.&lt;/p&gt;
&lt;h2 id="building-the-packages"&gt;Building the&amp;nbsp;Packages&lt;/h2&gt;
&lt;p&gt;First create the packages directory. &lt;strong&gt;Do not skip this.&lt;/strong&gt; More on that in a&amp;nbsp;moment:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# build jail&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/packages
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then a small script, saved&amp;nbsp;as &lt;code&gt;/root/build-php.sh&lt;/code&gt; in the build jail. It builds the interpreter first, then the extensions that others need at build time&amp;nbsp;(&lt;code&gt;pdo&lt;/code&gt; for&amp;nbsp;the &lt;code&gt;pdo_*&lt;/code&gt; drivers, &lt;code&gt;dom&lt;/code&gt; and &lt;code&gt;xml&lt;/code&gt;), then the&amp;nbsp;rest:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e
&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;BATCH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;yes
&lt;span class="nv"&gt;P&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/usr/ports

&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;lang/php85&lt;span class="w"&gt; &lt;/span&gt;databases/php85-pdo&lt;span class="w"&gt; &lt;/span&gt;textproc/php85-dom&lt;span class="w"&gt; &lt;/span&gt;textproc/php85-xml&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;archivers/php85-phar&lt;span class="w"&gt; &lt;/span&gt;archivers/php85-zip&lt;span class="w"&gt; &lt;/span&gt;archivers/php85-zlib&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;converters/php85-iconv&lt;span class="w"&gt; &lt;/span&gt;converters/php85-mbstring&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;databases/php85-pdo_mysql&lt;span class="w"&gt; &lt;/span&gt;databases/php85-pdo_pgsql&lt;span class="w"&gt; &lt;/span&gt;databases/php85-pdo_sqlite&lt;span class="w"&gt; &lt;/span&gt;databases/php85-sqlite3&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;devel/php85-intl&lt;span class="w"&gt; &lt;/span&gt;devel/php85-pcntl&lt;span class="w"&gt; &lt;/span&gt;devel/php85-sysvsem&lt;span class="w"&gt; &lt;/span&gt;devel/php85-tokenizer&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;ftp/php85-curl&lt;span class="w"&gt; &lt;/span&gt;graphics/php85-exif&lt;span class="w"&gt; &lt;/span&gt;graphics/php85-gd&lt;span class="w"&gt; &lt;/span&gt;math/php85-bcmath&lt;span class="w"&gt; &lt;/span&gt;math/php85-gmp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;net/php85-ldap&lt;span class="w"&gt; &lt;/span&gt;security/php85-filter&lt;span class="w"&gt; &lt;/span&gt;security/php85-sodium&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;sysutils/php85-fileinfo&lt;span class="w"&gt; &lt;/span&gt;sysutils/php85-posix&lt;span class="w"&gt; &lt;/span&gt;textproc/php85-ctype&lt;span class="w"&gt; &lt;/span&gt;textproc/php85-simplexml&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;textproc/php85-xmlreader&lt;span class="w"&gt; &lt;/span&gt;textproc/php85-xmlwriter&lt;span class="w"&gt; &lt;/span&gt;www/php85-session&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;=== &lt;/span&gt;&lt;span class="nv"&gt;$o&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;missing&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;make&lt;span class="w"&gt; &lt;/span&gt;-C&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$P&lt;/span&gt;/&lt;span class="nv"&gt;$o&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;missing&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;deps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%s\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$missing&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;php85&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$deps&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$deps&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;make&lt;span class="w"&gt; &lt;/span&gt;-C&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$P&lt;/span&gt;/&lt;span class="nv"&gt;$o&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;clean&lt;span class="w"&gt; &lt;/span&gt;package&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;clean
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# build jail&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;pipefail
sh&lt;span class="w"&gt; &lt;/span&gt;/root/build-php.sh&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tee&lt;span class="w"&gt; &lt;/span&gt;/root/build.log
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;exit status: &lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Without &lt;code&gt;pipefail&lt;/code&gt;, the pipeline reports the exit status&amp;nbsp;of &lt;code&gt;tee&lt;/code&gt;, which is almost always 0, even if the script stopped halfway. The script and this invocation are slightly hardened compared to what I actually ran: my original version&amp;nbsp;piped &lt;code&gt;make missing&lt;/code&gt; straight&amp;nbsp;into &lt;code&gt;grep&lt;/code&gt;, which would have hidden a&amp;nbsp;failing &lt;code&gt;make missing&lt;/code&gt;, and I ran it&amp;nbsp;without &lt;code&gt;pipefail&lt;/code&gt;. Neither caused a problem this time, but both could have hidden&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;Three details are doing the heavy lifting&amp;nbsp;here:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;make missing&lt;/code&gt; plus &lt;code&gt;pkg install&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;make missing&lt;/code&gt; lists every dependency that is not installed yet. Instead of building those from source, the script installs them as binary packages from the quarterly repository. The only things compiled are &lt;span class="caps"&gt;PHP&lt;/span&gt; and its extensions. This is the shortcut that makes Poudriere unnecessary for this&amp;nbsp;job.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;grep -v php85&lt;/code&gt;.&lt;/strong&gt; Without&amp;nbsp;it, &lt;code&gt;pkg install&lt;/code&gt; would happily&amp;nbsp;pull &lt;code&gt;php85-pdo-8.5.11&lt;/code&gt; from the official repository as a dependency, together with the vulnerable interpreter. With the filter, a missing &lt;span class="caps"&gt;PHP&lt;/span&gt; dependency is not installed from the official repository. The build order makes sure there is none: by the time an extension is built, the &lt;span class="caps"&gt;PHP&lt;/span&gt; components it needs have already been built and installed by an earlier iteration of the loop. If the order were wrong, the ports framework would build the missing &lt;span class="caps"&gt;PHP&lt;/span&gt; dependency on its own, from the&amp;nbsp;same &lt;code&gt;2026Q4&lt;/code&gt; tree, as part of the dependent port&amp;#8217;s&amp;nbsp;build.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;install&lt;/code&gt; after &lt;code&gt;package&lt;/code&gt;.&lt;/strong&gt; The extensions are built&amp;nbsp;with &lt;code&gt;phpize&lt;/code&gt; against the installed &lt;span class="caps"&gt;PHP&lt;/span&gt; headers, so every freshly built package also gets installed in the build&amp;nbsp;jail.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Because the build jail is brand new, there is no old &lt;span class="caps"&gt;PHP&lt;/span&gt; to uninstall first. The kernel-error.de article listed in the references describes what happens when you&amp;nbsp;do &lt;code&gt;deinstall install&lt;/code&gt; on a live system: the port&amp;#8217;s packing list step&amp;nbsp;calls &lt;code&gt;php&lt;/code&gt; from&amp;nbsp;the &lt;code&gt;PATH&lt;/code&gt;, which you have just removed. A clean jail avoids that&amp;nbsp;entirely.&lt;/p&gt;
&lt;p&gt;On my arm64 host with 6 cores and 8 &lt;span class="caps"&gt;GB&lt;/span&gt; of &lt;span class="caps"&gt;RAM&lt;/span&gt;, the whole run took just under half an hour. The interpreter itself was done after five minutes, and the extensions took the&amp;nbsp;rest.&lt;/p&gt;
&lt;p&gt;At the&amp;nbsp;end:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# build jail&lt;/span&gt;
ls&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/packages/All/php85-*&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;wc&lt;span class="w"&gt; &lt;/span&gt;-l
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;      32
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="the-mistake-where-did-my-packages-go"&gt;The Mistake: Where Did My Packages&amp;nbsp;Go?&lt;/h3&gt;
&lt;p&gt;On my first run,&amp;nbsp;that &lt;code&gt;ls&lt;/code&gt; said &lt;code&gt;No such file or directory&lt;/code&gt;. The build log looked perfect: every port ended&amp;nbsp;with &lt;code&gt;Building packages for php85-...&lt;/code&gt; and &lt;code&gt;Installing for php85-...&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The reason is how the ports framework handles a missing packages directory, and I had simply not thought of&amp;nbsp;it: &lt;code&gt;make package&lt;/code&gt; only writes&amp;nbsp;to &lt;code&gt;/usr/ports/packages/All/&lt;/code&gt; if &lt;code&gt;/usr/ports/packages&lt;/code&gt; &lt;strong&gt;already exists&lt;/strong&gt;. Otherwise, the package ends up in the port&amp;#8217;s work directory. The&amp;nbsp;final &lt;code&gt;clean&lt;/code&gt; in my loop then deleted the work directory, together with the&amp;nbsp;package.&lt;/p&gt;
&lt;p&gt;The good news: every package was still installed in the build&amp;nbsp;jail. &lt;code&gt;pkg create&lt;/code&gt; can rebuild package files from installed&amp;nbsp;packages:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# build jail&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/packages/All
pkg&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/packages/All&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;php85*&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Thirty-two packages, no second build. Still,&amp;nbsp;the &lt;code&gt;mkdir&lt;/code&gt; at the top of the build section is the better&amp;nbsp;way.&lt;/p&gt;
&lt;h2 id="a-local-repository"&gt;A Local&amp;nbsp;Repository&lt;/h2&gt;
&lt;p&gt;Copying &lt;code&gt;.pkg&lt;/code&gt; files into a jail and&amp;nbsp;running &lt;code&gt;pkg add&lt;/code&gt; works, but it ignores repository priorities and makes it easy to end up with a half-upgraded set. A repository is one command. Back on the&amp;nbsp;host:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/root/php85-8.5.12
cp&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bastille/jails/phpbuild/root/usr/ports/packages/All/php85-*.pkg&lt;span class="w"&gt; &lt;/span&gt;/root/php85-8.5.12/
pkg&lt;span class="w"&gt; &lt;/span&gt;repo&lt;span class="w"&gt; &lt;/span&gt;/root/php85-8.5.12/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;pkg repo&lt;/code&gt; adds &lt;code&gt;meta.conf&lt;/code&gt;, &lt;code&gt;data.pkg&lt;/code&gt;, &lt;code&gt;packagesite.pkg&lt;/code&gt; and &lt;code&gt;meta&lt;/code&gt; next to the packages. That is the&amp;nbsp;catalogue &lt;code&gt;pkg&lt;/code&gt; reads.&lt;/p&gt;
&lt;h2 id="upgrading-the-target-jails"&gt;Upgrading the Target&amp;nbsp;Jails&lt;/h2&gt;
&lt;p&gt;For every target jail, the same procedure. I started with the looking glass, the smallest one with only four &lt;span class="caps"&gt;PHP&lt;/span&gt; packages, as a dry run for the others. Everything in this section runs on the&amp;nbsp;host.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
&lt;span class="nv"&gt;j&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;lg&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;J&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/usr/local/bastille/jails/&lt;span class="nv"&gt;$j&lt;/span&gt;/root
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Copy the repository into the jail and register it with a higher priority than the official&amp;nbsp;one:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$J&lt;/span&gt;/var/tmp/php85-local&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$J&lt;/span&gt;/usr/local/etc/pkg/repos
cp&lt;span class="w"&gt; &lt;/span&gt;/root/php85-8.5.12/*&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$J&lt;/span&gt;/var/tmp/php85-local/

cat&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$J&lt;/span&gt;/usr/local/etc/pkg/repos/php85-local.conf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;&amp;#39;EOF&amp;#39;&lt;/span&gt;
&lt;span class="s"&gt;php85-local: {&lt;/span&gt;
&lt;span class="s"&gt;  url: &amp;quot;file:///var/tmp/php85-local&amp;quot;,&lt;/span&gt;
&lt;span class="s"&gt;  signature_type: &amp;quot;none&amp;quot;,&lt;/span&gt;
&lt;span class="s"&gt;  priority: 10,&lt;/span&gt;
&lt;span class="s"&gt;  enabled: yes&lt;/span&gt;
&lt;span class="s"&gt;}&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;signature_type: "none"&lt;/code&gt; is acceptable here because the repository is a directory inside the jail, filled by root on the host. Do not copy this line for anything that travels over a&amp;nbsp;network.&lt;/p&gt;
&lt;p&gt;The jail has to be running&amp;nbsp;for &lt;code&gt;pkg&lt;/code&gt; to do its job properly, but it should not start the vulnerable &lt;span class="caps"&gt;PHP&lt;/span&gt; and expose it while I am upgrading it. So I disable the web services before starting the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$J&lt;/span&gt;/etc/rc.conf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;nginx_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;NO&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;php_fpm_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;NO
bastille&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then the upgrade, limited to &lt;span class="caps"&gt;PHP&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;update
bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;php85*&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Installed packages to be UPGRADED:
        php85: 8.5.11 -&amp;gt; 8.5.12_1 [php85-local]
        php85-filter: 8.5.11 -&amp;gt; 8.5.12_1 [php85-local]
        php85-mbstring: 8.5.11 -&amp;gt; 8.5.12_1 [php85-local]
        php85-session: 8.5.11 -&amp;gt; 8.5.12_1 [php85-local]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Read that list before you&amp;nbsp;type &lt;code&gt;y&lt;/code&gt;. Every &lt;span class="caps"&gt;PHP&lt;/span&gt; package should come&amp;nbsp;from &lt;code&gt;php85-local&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;At the&amp;nbsp;end, &lt;code&gt;pkg&lt;/code&gt; prints a note that you may need to&amp;nbsp;remove &lt;code&gt;/usr/local/etc/php-fpm.d/www.conf&lt;/code&gt; manually. &lt;code&gt;pkg&lt;/code&gt; keeps a configuration file that differs from its sample,&amp;nbsp;here &lt;code&gt;www.conf.default&lt;/code&gt;, instead of deleting it. Mine differs on purpose, so I kept it. It is still worth comparing it with the new sample to see whether the update changed anything&amp;nbsp;relevant:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;diff&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/php-fpm.d/www.conf.default&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/php-fpm.d/www.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Check the version, turn the services back on, restart the jail and re-enable&amp;nbsp;autostart:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;php&lt;span class="w"&gt; &lt;/span&gt;-v
sysrc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$J&lt;/span&gt;/etc/rc.conf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;nginx_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;php_fpm_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
bastille&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;boot&lt;span class="w"&gt; &lt;/span&gt;on
bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sh&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;php-fpm -t; nginx -t; service php_fpm status; service nginx status&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;PHP 8.5.12 (cli) (built: Oct 22 2026 20:24:02) (NTS)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then open the application in a browser and actually use&amp;nbsp;it. &lt;code&gt;php -m&lt;/code&gt; and &lt;code&gt;php-fpm -t&lt;/code&gt; being happy does not prove&amp;nbsp;much.&lt;/p&gt;
&lt;h3 id="surprise-guests-in-the-second-jail"&gt;Surprise Guests in the Second&amp;nbsp;Jail&lt;/h3&gt;
&lt;p&gt;The DokuWiki jail had 13 &lt;span class="caps"&gt;PHP&lt;/span&gt; packages to upgrade,&amp;nbsp;and &lt;code&gt;pkg&lt;/code&gt; wanted to touch&amp;nbsp;16:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Installed packages to be UPGRADED:
        fontconfig: 2.17.1,1 -&amp;gt; 2.18.3,1 [FreeBSD-ports]
        php85: 8.5.11 -&amp;gt; 8.5.12_1 [php85-local]
        ...
        png: 1.6.58 -&amp;gt; 1.6.59_1 [FreeBSD-ports]

Installed packages to be REINSTALLED:
        cyrus-sasl-2.1.28_6 [FreeBSD-ports] (option changed: LMDB0 (added))
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is the quarterly branch doing its job, not a problem. The build jail had installed the &lt;strong&gt;current&lt;/strong&gt; quarterly versions of libpng and fontconfig as build dependencies, so the&amp;nbsp;new &lt;code&gt;php85-gd&lt;/code&gt; depends on them. The wiki jail had not been updated in a while and still had the older ones,&amp;nbsp;so &lt;code&gt;pkg&lt;/code&gt; upgraded them from the official&amp;nbsp;repository. &lt;code&gt;cyrus-sasl&lt;/code&gt; (pulled in through the &lt;span class="caps"&gt;LDAP&lt;/span&gt; extension) is reinstalled because the official package has gained an option since. Everything that does not come&amp;nbsp;from &lt;code&gt;php85-local&lt;/code&gt; comes from the official repository, which is exactly what I&amp;nbsp;want.&lt;/p&gt;
&lt;h2 id="switching-back-to-official-packages"&gt;Switching Back to Official&amp;nbsp;Packages&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Not tested yet.&lt;/strong&gt; At the time of writing, the official arm64 repository still offers 8.5.11, so this section is my plan, not something I have&amp;nbsp;run.&lt;/p&gt;
&lt;p&gt;The local repository is a stopgap. Once the official builders have caught up for arm64, I want the jails back on regular packages. First, check that the official repository has every &lt;span class="caps"&gt;PHP&lt;/span&gt; package the jail uses, not just the&amp;nbsp;interpreter. &lt;code&gt;pkg -j&lt;/code&gt; runs against a running jail from the&amp;nbsp;host:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host, with j and J set per jail as above&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;rquery&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-ports&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%n %v&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;query&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;php85*&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;php85 8.5.11
php85-ctype 8.5.11
...
php85-zlib 8.5.11
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Once every line&amp;nbsp;says &lt;code&gt;8.5.12_1&lt;/code&gt; or newer, remove the local repository and do a dry run of the forced&amp;nbsp;reinstall:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
rm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$J&lt;/span&gt;/usr/local/etc/pkg/repos/php85-local.conf
rm&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$J&lt;/span&gt;/var/tmp/php85-local
bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;php85*&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every package in that list should be&amp;nbsp;marked &lt;code&gt;[FreeBSD-ports]&lt;/code&gt;. If it is, run the same command&amp;nbsp;without &lt;code&gt;-n&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$j&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;php85*&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-f&lt;/code&gt; matters. Same version, same options, so without&amp;nbsp;it &lt;code&gt;pkg&lt;/code&gt; would see nothing to do and keep my self-built&amp;nbsp;packages.&lt;/p&gt;
&lt;p&gt;And the build jail goes away&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt; &lt;/span&gt;phpbuild
bastille&lt;span class="w"&gt; &lt;/span&gt;destroy&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;phpbuild
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I am keeping mine for now, stopped. The third jail stays offline for a while for unrelated reasons, and the packages&amp;nbsp;in &lt;code&gt;/root/php85-8.5.12&lt;/code&gt; will still be there when it is&amp;nbsp;ready.&lt;/p&gt;
&lt;h2 id="pitfalls-the-checklist"&gt;Pitfalls: The&amp;nbsp;Checklist&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Match the branch.&lt;/strong&gt; Ports tree branch = package repository&amp;nbsp;branch. &lt;code&gt;quarterly&lt;/code&gt; means &lt;code&gt;20xxQy&lt;/code&gt;, &lt;code&gt;latest&lt;/code&gt; means &lt;code&gt;main&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;mkdir -p /usr/ports/packages&lt;/code&gt; before &lt;code&gt;make package&lt;/code&gt;.&lt;/strong&gt;&amp;nbsp;Otherwise &lt;code&gt;make clean&lt;/code&gt; throws away your&amp;nbsp;packages.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Install dependencies as binary packages&lt;/strong&gt;, but keep &lt;span class="caps"&gt;PHP&lt;/span&gt; itself out of that&amp;nbsp;list.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rebuild all core extensions&lt;/strong&gt;, not&amp;nbsp;just &lt;code&gt;lang/php85&lt;/code&gt;. Leave &lt;span class="caps"&gt;PECL&lt;/span&gt; extensions and applications&amp;nbsp;alone.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Build in a fresh jail&lt;/strong&gt;, not on a system where the old &lt;span class="caps"&gt;PHP&lt;/span&gt; is still&amp;nbsp;installed.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Check &lt;code&gt;/etc/make.conf&lt;/code&gt;.&lt;/strong&gt; The kernel-error.de article ran into a broken &lt;span class="caps"&gt;CA&lt;/span&gt; bundle because&amp;nbsp;their &lt;code&gt;make.conf&lt;/code&gt; built &lt;span class="caps"&gt;PHP&lt;/span&gt; against OpenSSL from ports. Mine has&amp;nbsp;no &lt;code&gt;make.conf&lt;/code&gt; at all, so the port defaults match the official&amp;nbsp;packages.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Do not start the vulnerable service while you upgrade it.&lt;/strong&gt; Disable it&amp;nbsp;in &lt;code&gt;rc.conf&lt;/code&gt;, start the jail, upgrade, enable,&amp;nbsp;restart.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Read&amp;nbsp;the &lt;code&gt;pkg upgrade&lt;/code&gt; list&lt;/strong&gt; before confirming. Every &lt;span class="caps"&gt;PHP&lt;/span&gt; package must come from your local&amp;nbsp;repository.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Test the application&lt;/strong&gt;, not just the &lt;span class="caps"&gt;PHP&lt;/span&gt;&amp;nbsp;binary.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Poudriere is the right answer when building packages is part of your routine. For a single security fix on an architecture the official builders have not reached yet, the ports framework, a throwaway jail&amp;nbsp;and &lt;code&gt;pkg repo&lt;/code&gt; are enough. Once the build jail is destroyed, the only thing left on the host is a directory with 32&amp;nbsp;packages.&lt;/p&gt;
&lt;p&gt;What I like most about this approach is that nothing about it is clever. The jail is a normal Bastille jail, the packages are normal packages, and the repository is a normal pkg repository. When the official packages arrive,&amp;nbsp;two &lt;code&gt;rm&lt;/code&gt; commands and a forced upgrade return everything to its usual&amp;nbsp;state.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.kernel-error.de/2026/10/21/php2shell-php-freebsd-ports-gepatcht/"&gt;php2shell: &lt;span class="caps"&gt;PHP&lt;/span&gt; aus den FreeBSD-Ports gepatcht (kernel-error.de, German)&lt;/a&gt;, the article that got me started, including&amp;nbsp;the &lt;code&gt;deinstall install&lt;/code&gt; and &lt;span class="caps"&gt;CA&lt;/span&gt; bundle&amp;nbsp;pitfalls&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.heise.de/news/php2shell-Kritische-Sicherheitsluecken-erlauben-Codeeinschleusung-in-PHP-11492938.html"&gt;heise online on php2shell&amp;nbsp;(German)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://cgit.freebsd.org/ports/log/lang/php85?h=2026Q4"&gt;lang/php85 history on the 2026Q4&amp;nbsp;branch&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.php.net/releases/index.php?json&amp;amp;version=8.5.12"&gt;&lt;span class="caps"&gt;PHP&lt;/span&gt; 8.5.12 release&amp;nbsp;data&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pkg-repo&amp;amp;sektion=8"&gt;pkg-repo(8)&lt;/a&gt;, &lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pkg-create&amp;amp;sektion=8"&gt;pkg-create(8)&lt;/a&gt;, &lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pkg.conf&amp;amp;sektion=5"&gt;pkg.conf(5)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/porters-handbook/"&gt;FreeBSD Porter&amp;#8217;s&amp;nbsp;Handbook&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastille.readthedocs.io/"&gt;Bastille&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="php"/><category term="php2shell"/><category term="security"/><category term="ports"/><category term="pkg"/><category term="bastille"/><category term="jails"/><category term="arm64"/><category term="aarch64"/></entry><entry><title>The T480 Is Back: My FreeBSD 15.1 Laptop, From Boot to Browser</title><link href="https://hofstede.it/the-t480-is-back-my-freebsd-151-laptop-from-boot-to-browser/" rel="alternate"/><published>2026-10-15T00:00:00+02:00</published><updated>2026-10-15T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-10-15:/the-t480-is-back-my-freebsd-151-laptop-from-boot-to-browser/</id><summary type="html">&lt;p&gt;My ThinkPad T480 now runs FreeBSD 15.1 as a complete desktop: Xfce 4.20, 5 GHz Wi-Fi, verified &lt;span class="caps"&gt;VA&lt;/span&gt;-&lt;span class="caps"&gt;API&lt;/span&gt; video decoding in Firefox and Chromium, Bluetooth audio, a webcam that handles Zoom calls in Firefox, a smart-card-backed &lt;span class="caps"&gt;SSH&lt;/span&gt; and Git workflow, and coding agents. Here are the packages, configuration, tests and machine-specific traps behind a laptop I can simply&amp;nbsp;use.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Xfce on the ThinkPad T480, with powermon, Firefox, Neovim and fastfetch visible" src="https://hofstede.it/images/2026-10-15-t480-freebsd-daily-laptop.webp" title="The T480 at work. powermon shows CPU package power, not whole-laptop draw."&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-machine-today"&gt;The Machine&amp;nbsp;Today&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#graphics-and-the-desktop"&gt;Graphics and the&amp;nbsp;Desktop&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#wi-fi-what-5-ghz-works-means"&gt;Wi-Fi: What &amp;#8220;5 GHz Works&amp;#8221;&amp;nbsp;Means&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#video-decoding-i-checked-the-decoder"&gt;Video Decoding: I Checked the&amp;nbsp;Decoder&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#bluetooth-audio-to-the-muvo"&gt;Bluetooth Audio to the &lt;span class="caps"&gt;MUVO&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-webcam-one-zoom-call-no-drama"&gt;The Webcam: One Zoom Call, No&amp;nbsp;Drama&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#power-about-3-w-at-idle"&gt;Power: About 3 W at&amp;nbsp;Idle&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#zfs-pf-and-a-private-vm-network"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt;, &lt;span class="caps"&gt;PF&lt;/span&gt; and a Private &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;nbsp;Network&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#chezmoi-familiar-dotfiles-freebsd-specific-behavior"&gt;Chezmoi: Familiar Dotfiles, FreeBSD-Specific&amp;nbsp;Behavior&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-rust-openpgp-card-stack-for-ssh-and-git"&gt;A Rust OpenPGP Card Stack for &lt;span class="caps"&gt;SSH&lt;/span&gt; and&amp;nbsp;Git&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#agents-on-the-laptop"&gt;Agents on the&amp;nbsp;Laptop&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#if-you-are-setting-up-your-own-t480"&gt;If You Are Setting Up Your Own&amp;nbsp;T480&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;I wrote about &lt;a href="https://hofstede.it/freebsd-150-on-the-thinkpad-t480-efficient-stable-and-8-hours-on-battery/"&gt;power tuning my ThinkPad T480 on FreeBSD&lt;/a&gt; last year. That was the article you write when a laptop is still a project: get the graphics driver loaded, persuade the wireless card to connect, measure battery drain, repeat. My T14s later became my main writing machine, but the T480 stayed on my desk. Recently I brought it back into regular&amp;nbsp;use.&lt;/p&gt;
&lt;p&gt;The interesting thing now is how uninteresting it feels. I log in to Xfce, join my 5 GHz network, watch a video in Firefox or Chromium, sit through a Zoom call, send audio to my Bluetooth speaker, sign a Git commit with my card and work with coding agents. It all runs on FreeBSD. There are machine-specific details behind that sentence, so this is both a status report and a set of notes for another T480&amp;nbsp;owner.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The short version:&lt;/strong&gt; FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;-p4, Xfce 4.20, Intel graphics, an &lt;span class="caps"&gt;AX210&lt;/span&gt; Wi-Fi card, and an idle &lt;span class="caps"&gt;CPU&lt;/span&gt;-package reading around &lt;strong&gt;3 W&amp;nbsp;in &lt;code&gt;powermon&lt;/code&gt;&lt;/strong&gt;. It is a perfectly usable laptop for what I do, and it has been crazily&amp;nbsp;stable.&lt;/p&gt;
&lt;h2 id="the-machine-today"&gt;The Machine&amp;nbsp;Today&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;What is in this T480&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;Intel Core i7-8550U, four cores and eight threads&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Memory&lt;/td&gt;
&lt;td&gt;32 GiB&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Graphics in use&lt;/td&gt;
&lt;td&gt;Intel &lt;span class="caps"&gt;UHD&lt;/span&gt; Graphics 620&amp;nbsp;with &lt;code&gt;i915kms&lt;/code&gt; and Xorg&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Other &lt;span class="caps"&gt;GPU&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;NVIDIA&lt;/span&gt; GeForce &lt;span class="caps"&gt;MX150&lt;/span&gt;, present but not used by this desktop&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wireless&lt;/td&gt;
&lt;td&gt;Intel &lt;span class="caps"&gt;AX210&lt;/span&gt;&amp;nbsp;with &lt;code&gt;iwlwifi&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Storage&lt;/td&gt;
&lt;td&gt;NVMe &lt;span class="caps"&gt;SSD&lt;/span&gt;, &lt;span class="caps"&gt;ZFS&lt;/span&gt; root&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sound&lt;/td&gt;
&lt;td&gt;FreeBSD &lt;span class="caps"&gt;OSS&lt;/span&gt;, PulseAudio&amp;nbsp;and &lt;code&gt;virtual_oss&lt;/code&gt; for Bluetooth&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Desktop&lt;/td&gt;
&lt;td&gt;Xfce 4.20 on X11, started by &lt;span class="caps"&gt;SDDM&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;System&lt;/td&gt;
&lt;td&gt;FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;-p4&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;That is not new hardware. It does not need to be. The keyboard, ports and repairability are still good reasons to keep a T480 around, and the four-core Intel &lt;span class="caps"&gt;CPU&lt;/span&gt; remains plenty for browsers, terminals, writing and&amp;nbsp;development.&lt;/p&gt;
&lt;p&gt;These are the relevant installed packages, grouped by job rather than copied from the machine&amp;#8217;s&amp;nbsp;several-hundred-line &lt;code&gt;pkg info&lt;/code&gt; output:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;Packages installed here&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Desktop and graphics&lt;/td&gt;
&lt;td&gt;&lt;code&gt;xfce&lt;/code&gt;, &lt;code&gt;sddm&lt;/code&gt;, &lt;code&gt;xorg-server&lt;/code&gt;, &lt;code&gt;drm-latest-kmod&lt;/code&gt;, &lt;code&gt;gpu-firmware-intel-kmod-kabylake&lt;/code&gt;, &lt;code&gt;xf86-input-libinput&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wireless&lt;/td&gt;
&lt;td&gt;&lt;code&gt;wifi-firmware-iwlwifi-kmod-ax210&lt;/code&gt;, &lt;code&gt;iwmbt-firmware&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Browsers and video&lt;/td&gt;
&lt;td&gt;&lt;code&gt;firefox&lt;/code&gt;, &lt;code&gt;chromium&lt;/code&gt;, &lt;code&gt;libva-intel-media-driver&lt;/code&gt;, &lt;code&gt;libva-utils&lt;/code&gt;, &lt;code&gt;mesa-dri&lt;/code&gt;, &lt;code&gt;mpv&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Webcam&lt;/td&gt;
&lt;td&gt;&lt;code&gt;webcamd&lt;/code&gt;, &lt;code&gt;v4l-utils&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bluetooth and sound&lt;/td&gt;
&lt;td&gt;&lt;code&gt;virtual_oss_bluetooth&lt;/code&gt;, &lt;code&gt;pulseaudio&lt;/code&gt;, &lt;code&gt;pavucontrol&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Card, Git and dotfiles&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pcsc-lite&lt;/code&gt;, &lt;code&gt;ccid&lt;/code&gt;, &lt;code&gt;pinentry-gtk2&lt;/code&gt;, &lt;code&gt;git&lt;/code&gt;, &lt;code&gt;chezmoi&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Work&lt;/td&gt;
&lt;td&gt;&lt;code&gt;codex&lt;/code&gt;, &lt;code&gt;crush&lt;/code&gt;, &lt;code&gt;fish&lt;/code&gt;, &lt;code&gt;neovim&lt;/code&gt;, &lt;code&gt;tmux&lt;/code&gt;, &lt;code&gt;rust&lt;/code&gt;, &lt;code&gt;go&lt;/code&gt;, &lt;code&gt;powermon&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The OpenPGP Card tools are not in that list: on this machine they live&amp;nbsp;under &lt;code&gt;~/.cargo/bin&lt;/code&gt;. More on those&amp;nbsp;below.&lt;/p&gt;
&lt;h2 id="graphics-and-the-desktop"&gt;Graphics and the&amp;nbsp;Desktop&lt;/h2&gt;
&lt;p&gt;The Intel &lt;span class="caps"&gt;GPU&lt;/span&gt; is the useful one here. The &lt;span class="caps"&gt;MX150&lt;/span&gt; appears on the &lt;span class="caps"&gt;PCI&lt;/span&gt; bus, but my Xfce session runs on the &lt;span class="caps"&gt;UHD&lt;/span&gt; 620&amp;nbsp;with &lt;code&gt;i915kms&lt;/code&gt;. The&amp;nbsp;installed &lt;code&gt;drm-latest-kmod&lt;/code&gt; and matching Intel Kaby Lake firmware provide the kernel side. The relevant part&amp;nbsp;of &lt;code&gt;/etc/rc.conf&lt;/code&gt; is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;kld_list=&amp;quot;i915kms coretemp cpuctl cuse nullfs&amp;quot;
dbus_enable=&amp;quot;YES&amp;quot;
sddm_enable=&amp;quot;YES&amp;quot;
seatd_enable=&amp;quot;YES&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;seatd&lt;/code&gt; is installed and enabled because I also keep a Wayland session available. This Xfce desktop is X11. Xorg has a short Intel device file&amp;nbsp;under &lt;code&gt;/usr/local/etc/X11/xorg.conf.d/20-modesetting.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Section &amp;quot;Device&amp;quot;
    Identifier &amp;quot;Intel Modesetting&amp;quot;
    Driver &amp;quot;modesetting&amp;quot;
    Option &amp;quot;AccelMethod&amp;quot; &amp;quot;glamor&amp;quot;
    Option &amp;quot;DRI&amp;quot; &amp;quot;3&amp;quot;
EndSection
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There is a&amp;nbsp;separate &lt;code&gt;libinput&lt;/code&gt; file for touchpad tapping and natural scrolling. Those are preferences, not prerequisites for FreeBSD graphics. The &lt;a href="https://docs.freebsd.org/en/books/handbook/x11/"&gt;FreeBSD Handbook&amp;#8217;s Xorg chapter&lt;/a&gt; is the better starting point for a fresh installation. Use the driver and kernel-module packages that match your release. In particular, do not paste my&amp;nbsp;entire &lt;code&gt;kld_list&lt;/code&gt; into a new&amp;nbsp;system: &lt;code&gt;nullfs&lt;/code&gt;, &lt;code&gt;cuse&lt;/code&gt; and the &lt;span class="caps"&gt;CPU&lt;/span&gt; modules have jobs specific to my&amp;nbsp;setup.&lt;/p&gt;
&lt;p&gt;One boot setting deserves its own warning.&amp;nbsp;My &lt;code&gt;/boot/loader.conf&lt;/code&gt; contains:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;zfs_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;compat&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;linuxkpi&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;i915_disable_power_well&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;hw&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pci&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;do_power_nodriver&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;3&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;hw&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pci&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;do_power_suspend&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The Intel display power-well value is retained because it has been necessary for reliable display and suspend behavior &lt;strong&gt;on this laptop&lt;/strong&gt;. The &lt;span class="caps"&gt;PCI&lt;/span&gt; settings are an established, fairly aggressive policy for unclaimed devices, not general T480 defaults. I would test one change at a time on another machine and keep a boot environment before experimenting with loader settings. Notice what is absent: I do not&amp;nbsp;load &lt;code&gt;i915kms&lt;/code&gt; twice from both loader&amp;nbsp;and &lt;code&gt;rc.conf&lt;/code&gt;, and NVMe is already in this &lt;span class="caps"&gt;GENERIC&lt;/span&gt;&amp;nbsp;kernel.&lt;/p&gt;
&lt;h2 id="wi-fi-what-5-ghz-works-means"&gt;Wi-Fi: What &amp;#8220;5 GHz Works&amp;#8221;&amp;nbsp;Means&lt;/h2&gt;
&lt;p&gt;The wireless card is an &lt;span class="caps"&gt;AX210&lt;/span&gt;. Its firmware comes&amp;nbsp;from &lt;code&gt;wifi-firmware-iwlwifi-kmod-ax210&lt;/code&gt;, and the package also owns a small file&amp;nbsp;under &lt;code&gt;/boot/loader.conf.d/&lt;/code&gt;. I leave that package-owned file alone. My network setup&amp;nbsp;in &lt;code&gt;/etc/rc.conf&lt;/code&gt; is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;wlans_iwlwifi0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;wlan0&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;create_args_wlan0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;wlanmode sta regdomain ETSI country DE&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;ifconfig_wlan0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;WPA DHCP&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;ifconfig_wlan0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;inet6 accept_rtadv&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I am in Germany, hence the regulatory domain and country. Use the values appropriate to your location. The network credentials live in the wireless supplicant configuration, not in this&amp;nbsp;article.&lt;/p&gt;
&lt;p&gt;At the time I&amp;nbsp;checked, &lt;code&gt;ifconfig wlan0&lt;/code&gt; showed an associated connection on &lt;strong&gt;channel 36, 5180 MHz, &lt;span class="caps"&gt;VHT&lt;/span&gt;/80, 802.11ac&lt;/strong&gt;. That is a working 5 GHz connection. It does not establish 6 GHz or Wi-Fi 6 operation merely because the card&amp;#8217;s product name says Wi-Fi 6E. When diagnosing this, look at the negotiated channel and media line, not the card&amp;#8217;s marketing&amp;nbsp;name:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ifconfig&lt;span class="w"&gt; &lt;/span&gt;wlan0
sysctl&lt;span class="w"&gt; &lt;/span&gt;net.wlan.devices
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I have resisted adding&amp;nbsp;speculative &lt;code&gt;iwlwifi&lt;/code&gt; power-save knobs. If wireless or idle power goes odd, I first check the actual connection, route, &lt;span class="caps"&gt;CPU&lt;/span&gt; idle state and busy processes. The &lt;a href="https://docs.freebsd.org/en/books/handbook/network/"&gt;FreeBSD networking chapter&lt;/a&gt; covers the&amp;nbsp;normal &lt;code&gt;wlan&lt;/code&gt;/&lt;span class="caps"&gt;WPA&lt;/span&gt;/&lt;span class="caps"&gt;DHCP&lt;/span&gt; setup in more&amp;nbsp;detail.&lt;/p&gt;
&lt;h2 id="video-decoding-i-checked-the-decoder"&gt;Video Decoding: I Checked the&amp;nbsp;Decoder&lt;/h2&gt;
&lt;p&gt;Video playback was the last browser detail I wanted to settle properly.&amp;nbsp;Intel&amp;#8217;s &lt;code&gt;i915kms&lt;/code&gt; gives&amp;nbsp;me &lt;code&gt;/dev/dri/renderD128&lt;/code&gt;, and the&amp;nbsp;installed &lt;code&gt;libva-intel-media-driver&lt;/code&gt; exposes H.264, &lt;span class="caps"&gt;HEVC&lt;/span&gt; and &lt;span class="caps"&gt;VP9&lt;/span&gt; decoding through &lt;span class="caps"&gt;VA&lt;/span&gt;-&lt;span class="caps"&gt;API&lt;/span&gt;. It does not expose &lt;span class="caps"&gt;AV1&lt;/span&gt; decoding on this &lt;span class="caps"&gt;GPU&lt;/span&gt;. A first check&amp;nbsp;is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vainfo&lt;span class="w"&gt; &lt;/span&gt;--display&lt;span class="w"&gt; &lt;/span&gt;drm&lt;span class="w"&gt; &lt;/span&gt;--device&lt;span class="w"&gt; &lt;/span&gt;/dev/dri/renderD128
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That only says what the driver &lt;em&gt;can&lt;/em&gt; decode. It does not say what a browser &lt;em&gt;did&lt;/em&gt; decode. I learned this with&amp;nbsp;Chromium: &lt;code&gt;chrome://gpu&lt;/code&gt; looked happy, while an H.264 test video still&amp;nbsp;used &lt;code&gt;FFmpegVideoDecoder&lt;/code&gt; in software. On this installation, Chromium&amp;#8217;s &lt;span class="caps"&gt;VA&lt;/span&gt;-&lt;span class="caps"&gt;API&lt;/span&gt; setup did not choose the working render node until I started it&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chrome --render-node-override=/dev/dri/renderD128
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For normal use I put that flag in a &lt;strong&gt;per-user&lt;/strong&gt; &lt;code&gt;chromium-browser.desktop&lt;/code&gt; override&amp;nbsp;under &lt;code&gt;~/.local/share/applications&lt;/code&gt;, leaving the package&amp;#8217;s launcher untouched. Its relevant line&amp;nbsp;is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;Exec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/bin/chrome --render-node-override=/dev/dri/renderD128 %U&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After fully quitting and relaunching Chromium from that menu&amp;nbsp;entry, &lt;code&gt;chrome://media-internals&lt;/code&gt; reported &lt;code&gt;VaapiVideoDecoder&lt;/code&gt; and &lt;code&gt;kIsPlatformVideoDecoder: true&lt;/code&gt; for a controlled 720p H.264 clip. Sending a &lt;span class="caps"&gt;URL&lt;/span&gt; to an already-running Chromium process would not prove that the new flag took effect. If the &lt;span class="caps"&gt;DRM&lt;/span&gt; device number changes, the launcher needs another&amp;nbsp;look.&lt;/p&gt;
&lt;p&gt;Firefox has its own set of preferences in my normal profile: &lt;span class="caps"&gt;VA&lt;/span&gt;-&lt;span class="caps"&gt;API&lt;/span&gt; enabled, &lt;span class="caps"&gt;AV1&lt;/span&gt; disabled, and hardware video decoding and WebRender forced&amp;nbsp;on. &lt;code&gt;about:support&lt;/code&gt; reports &lt;span class="caps"&gt;EGL&lt;/span&gt;, &lt;span class="caps"&gt;DMABUF&lt;/span&gt; and the expected decode capabilities. More decisively, a disposable Firefox profile using those video settings played the same local H.264 clip while its decoder log reported successful &lt;span class="caps"&gt;VA&lt;/span&gt;-&lt;span class="caps"&gt;API&lt;/span&gt; initialization and decoded frames. That is the difference between &amp;#8220;supported&amp;#8221; and&amp;nbsp;&amp;#8220;used&amp;#8221;.&lt;/p&gt;
&lt;p&gt;So both browsers decode H.264 in hardware. One caveat matters in practice: a site can choose &lt;span class="caps"&gt;AV1&lt;/span&gt;, which this &lt;span class="caps"&gt;UHD&lt;/span&gt; 620 cannot decode in hardware. The codec, not just the browser, determines the&amp;nbsp;result.&lt;/p&gt;
&lt;h2 id="bluetooth-audio-to-the-muvo"&gt;Bluetooth Audio to the &lt;span class="caps"&gt;MUVO&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;My Creative &lt;span class="caps"&gt;MUVO&lt;/span&gt; 2c works as a Bluetooth speaker, but its route to Xfce&amp;#8217;s volume control is longer than the route on many Linux&amp;nbsp;desktops:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Bluetooth controller → paired MUVO → FreeBSD Bluetooth audio
    → virtual_oss (/dev/dsp9) → PulseAudio OSS sink → Xfce applications
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The paired device has a local name&amp;nbsp;in &lt;code&gt;/etc/bluetooth/hosts&lt;/code&gt;. I am leaving hardware addresses and pairing material out of the article, since pairing is specific to your own speaker. On this&amp;nbsp;machine, &lt;code&gt;/etc/rc.conf&lt;/code&gt; contains:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;hcsecd_enable=&amp;quot;YES&amp;quot;
virtual_oss_configs=&amp;quot;muvo&amp;quot;
virtual_oss_muvo=&amp;quot;-S -C 2 -c 2 -r 48000 -b 16 -s 16ms -R /dev/null -P /dev/bluetooth/muvo -d dsp9&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;virtual_oss&lt;/code&gt; itself is part of the base system on FreeBSD 15.1, including its rc.d script.&amp;nbsp;The &lt;code&gt;virtual_oss_bluetooth&lt;/code&gt; package only adds the Bluetooth&amp;nbsp;backend. &lt;code&gt;virtual_oss_enable&lt;/code&gt; stays at its default&amp;nbsp;of &lt;code&gt;NO&lt;/code&gt;, so nothing starts at boot. When I want the speaker, I&amp;nbsp;run &lt;code&gt;muvo-connect.sh&lt;/code&gt;, a crude little script that does the trick. It starts&amp;nbsp;Bluetooth, &lt;code&gt;hcsecd&lt;/code&gt; and the&amp;nbsp;named &lt;code&gt;virtual_oss&lt;/code&gt; instance, then builds the PulseAudio&amp;nbsp;side:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pactl&lt;span class="w"&gt; &lt;/span&gt;load-module&lt;span class="w"&gt; &lt;/span&gt;module-oss&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;device&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/dev/dsp9&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sink_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;muvo
pactl&lt;span class="w"&gt; &lt;/span&gt;load-module&lt;span class="w"&gt; &lt;/span&gt;module-remap-sink&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sink_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;muvo_volume&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;master&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;muvo
pactl&lt;span class="w"&gt; &lt;/span&gt;set-sink-volume&lt;span class="w"&gt; &lt;/span&gt;muvo_volume&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;70&lt;/span&gt;%
pactl&lt;span class="w"&gt; &lt;/span&gt;set-default-sink&lt;span class="w"&gt; &lt;/span&gt;muvo_volume
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The&amp;nbsp;extra &lt;code&gt;module-remap-sink&lt;/code&gt; layer gives the speaker a PulseAudio volume of its own, and that is what the Xfce volume control talks to. The script also moves any stream that landed on the&amp;nbsp;raw &lt;code&gt;muvo&lt;/code&gt; sink over&amp;nbsp;to &lt;code&gt;muvo_volume&lt;/code&gt;. Streams already playing on the laptop speakers stay where they are, so a running video needs a restart or a quick move&amp;nbsp;in &lt;code&gt;pavucontrol&lt;/code&gt;. &lt;code&gt;muvo-connect.sh stop&lt;/code&gt; points the default back&amp;nbsp;at &lt;code&gt;dsp0&lt;/code&gt;, unloads both modules and stops&amp;nbsp;the &lt;code&gt;virtual_oss&lt;/code&gt; instance. Bluetooth itself keeps&amp;nbsp;running.&lt;/p&gt;
&lt;p&gt;This is a working arrangement, but&amp;nbsp;those &lt;code&gt;dsp9&lt;/code&gt; and device-name choices describe &lt;strong&gt;my&lt;/strong&gt; host, not a magic universal number. If you build the same path,&amp;nbsp;inspect &lt;code&gt;/dev/sndstat&lt;/code&gt; and &lt;code&gt;pactl list short sinks&lt;/code&gt; at each step. Bluetooth audio is something I switch on when I want it, not something I keep active all&amp;nbsp;day.&lt;/p&gt;
&lt;h2 id="the-webcam-one-zoom-call-no-drama"&gt;The Webcam: One Zoom Call, No&amp;nbsp;Drama&lt;/h2&gt;
&lt;p&gt;The built-in camera is the easiest part of this article.&amp;nbsp;The &lt;code&gt;webcamd&lt;/code&gt; package runs the Linux &lt;span class="caps"&gt;UVC&lt;/span&gt; driver in userspace and exposes the camera as a &lt;span class="caps"&gt;V4L2&lt;/span&gt; device. Two pieces of configuration are&amp;nbsp;needed:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;webcamd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
pw&lt;span class="w"&gt; &lt;/span&gt;groupmod&lt;span class="w"&gt; &lt;/span&gt;webcamd&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;chofstede
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The package&amp;#8217;s devd rules&amp;nbsp;start &lt;code&gt;webcamd&lt;/code&gt; when the camera appears, and it&amp;nbsp;creates &lt;code&gt;/dev/video0&lt;/code&gt; and &lt;code&gt;/dev/video1&lt;/code&gt;, owned by&amp;nbsp;the &lt;code&gt;webcamd&lt;/code&gt; group. The second node carries metadata, not a picture, so applications should&amp;nbsp;use &lt;code&gt;video0&lt;/code&gt;. Log out and back in after the group&amp;nbsp;change. &lt;code&gt;v4l2-ctl --list-devices&lt;/code&gt; from &lt;code&gt;v4l-utils&lt;/code&gt; is the quick check that the camera is&amp;nbsp;there.&lt;/p&gt;
&lt;p&gt;The real test was a 45-minute Zoom call in Firefox this morning. The camera worked for the whole call, and I didn&amp;#8217;t have to think about it once. That is how it should&amp;nbsp;be.&lt;/p&gt;
&lt;h2 id="power-about-3-w-at-idle"&gt;Power: About 3 W at&amp;nbsp;Idle&lt;/h2&gt;
&lt;p&gt;At&amp;nbsp;idle, &lt;code&gt;powermon&lt;/code&gt; shows me &lt;strong&gt;roughly 3 W for the &lt;span class="caps"&gt;CPU&lt;/span&gt; package&lt;/strong&gt;, and the screenshot above catches an even lower reading. That is the &lt;span class="caps"&gt;CPU&lt;/span&gt; package only, not the whole laptop: the display, radios and everything else come on top. My older article measured at a different point in the configuration, so the two numbers are not a before-and-after comparison. The configuration also retains an Intel display workaround for reliability. I am not going to trade a working resume for a prettier idle&amp;nbsp;number.&lt;/p&gt;
&lt;p&gt;This machine currently uses Intel Speed Shift with an &lt;span class="caps"&gt;EPP&lt;/span&gt; of &lt;strong&gt;50&lt;/strong&gt; and&amp;nbsp;the &lt;code&gt;Cmax&lt;/code&gt; idle policy. The resolved &lt;span class="caps"&gt;CPU&lt;/span&gt; idle level is C8. I disabled the&amp;nbsp;older &lt;code&gt;powerd&lt;/code&gt; service after checking that Speed Shift was already controlling&amp;nbsp;frequency:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;#&lt;/span&gt; /etc/rc.conf
powerd_enable=&amp;quot;NO&amp;quot;
performance_cx_lowest=&amp;quot;Cmax&amp;quot;
economy_cx_lowest=&amp;quot;Cmax&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;To see the &lt;em&gt;current&lt;/em&gt; state rather than assume those lines had the intended&amp;nbsp;effect:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;dev.hwpstate_intel.0.epp&lt;span class="w"&gt; &lt;/span&gt;hw.acpi.cpu.cx_lowest
sysctl&lt;span class="w"&gt; &lt;/span&gt;hw.acpi.acline&lt;span class="w"&gt; &lt;/span&gt;hw.acpi.battery.rate
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;hw.acpi.battery.rate&lt;/code&gt; only means something on battery, and only after the workload has&amp;nbsp;settled.&lt;/p&gt;
&lt;h2 id="zfs-pf-and-a-private-vm-network"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt;, &lt;span class="caps"&gt;PF&lt;/span&gt; and a Private &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;nbsp;Network&lt;/h2&gt;
&lt;p&gt;The T480 boots from an NVMe &lt;span class="caps"&gt;SSD&lt;/span&gt; into &lt;span class="caps"&gt;ZFS&lt;/span&gt;. &lt;code&gt;zroot&lt;/code&gt; currently&amp;nbsp;uses &lt;code&gt;lz4&lt;/code&gt; compression and&amp;nbsp;has &lt;code&gt;atime&lt;/code&gt; off. FreeBSD&amp;#8217;s stock daily scrub check is off by default.&amp;nbsp;My &lt;code&gt;/etc/periodic.conf&lt;/code&gt; turns it on&amp;nbsp;for &lt;code&gt;zroot&lt;/code&gt; with a 35-day threshold, but only&amp;nbsp;when &lt;code&gt;hw.acpi.acline&lt;/code&gt; says the laptop is on &lt;span class="caps"&gt;AC&lt;/span&gt; power.&amp;nbsp;Since &lt;code&gt;periodic.conf&lt;/code&gt; is just shell, that is a&amp;nbsp;three-line &lt;code&gt;if&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;daily_scrub_zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;/sbin/sysctl&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;hw.acpi.acline&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;daily_scrub_zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A scrub that has already started keeps running if I unplug the laptop. A sleeping or powered-off laptop can miss that nightly cron run, so I still&amp;nbsp;check &lt;code&gt;zpool status&lt;/code&gt; myself.&amp;nbsp;My &lt;code&gt;/etc/sysctl.conf&lt;/code&gt; also requests 4 KiB minimum sectors for &lt;em&gt;future&lt;/em&gt; &lt;span class="caps"&gt;ZFS&lt;/span&gt; vdev creation. That knob does not change the geometry of the existing&amp;nbsp;pool.&lt;/p&gt;
&lt;p&gt;The rest of that sysctl file is deliberately small: S3 for lid and sleep button, process visibility limited across user IDs and groups, randomized PIDs, and bridge-filter settings below. Those privacy settings can make an unprivileged service check look incomplete, so I&amp;nbsp;use &lt;code&gt;sudo&lt;/code&gt; when I need a full process view. I did not add a page of &lt;span class="caps"&gt;TCP&lt;/span&gt; or scheduler &amp;#8220;optimizations&amp;#8221; to achieve the idle&amp;nbsp;reading.&lt;/p&gt;
&lt;p&gt;I also keep a&amp;nbsp;private &lt;code&gt;bhyve&lt;/code&gt; network&amp;nbsp;on &lt;code&gt;bridge0&lt;/code&gt;,&amp;nbsp;with &lt;code&gt;dnsmasq&lt;/code&gt; serving guests and &lt;span class="caps"&gt;PF&lt;/span&gt; providing &lt;span class="caps"&gt;NAT&lt;/span&gt; through either Wi-Fi or Ethernet. That is why this workstation&amp;#8217;s firewall is more involved than a laptop that only browses the&amp;nbsp;web:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;#&lt;/span&gt; /etc/rc.conf, abbreviated
cloned_interfaces=&amp;quot;bridge0&amp;quot;
ifconfig_bridge0=&amp;quot;inet 192.168.87.1 netmask 255.255.255.0&amp;quot;
gateway_enable=&amp;quot;YES&amp;quot;
vm_enable=&amp;quot;YES&amp;quot;
vm_dir=&amp;quot;zfs:zroot/bhyve&amp;quot;
dnsmasq_enable=&amp;quot;YES&amp;quot;
pf_enable=&amp;quot;YES&amp;quot;
pflog_enable=&amp;quot;YES&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;dnsmasq&lt;/code&gt; binds&amp;nbsp;to &lt;code&gt;bridge0&lt;/code&gt; and hands out addresses&amp;nbsp;in &lt;code&gt;192.168.87.100&lt;/code&gt; to &lt;code&gt;.150&lt;/code&gt;,&amp;nbsp;advertising &lt;code&gt;192.168.87.1&lt;/code&gt; as router and &lt;span class="caps"&gt;DNS&lt;/span&gt; server. The key shape&amp;nbsp;of &lt;code&gt;/etc/pf.conf&lt;/code&gt; is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;uplinks&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{ wlan0 em0 }&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;vm_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;192.168.87.0/24&amp;quot;&lt;/span&gt;
&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;skip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lo0&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;wlan0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;vm_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;wlan0&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;em0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;vm_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;em0&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;all&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;uplinks&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;vm_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bridge0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;uplinks&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;67&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;68&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;uplinks&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;547&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;546&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;all&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;all&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; blocks unsolicited inbound &lt;span class="caps"&gt;TCP&lt;/span&gt; and &lt;span class="caps"&gt;UDP&lt;/span&gt; connections on the physical interfaces, permits outbound traffic with state, trusts the &lt;span class="caps"&gt;VM&lt;/span&gt; bridge, and NATs that subnet on&amp;nbsp;either &lt;code&gt;wlan0&lt;/code&gt; or &lt;code&gt;em0&lt;/code&gt;. The &lt;span class="caps"&gt;DHCP&lt;/span&gt; and &lt;span class="caps"&gt;ICMP&lt;/span&gt; exceptions allow normal client networking, IPv6 neighbor discovery and path &lt;span class="caps"&gt;MTU&lt;/span&gt; discovery.&amp;nbsp;The &lt;code&gt;quick&lt;/code&gt; rule drops packets arriving from a physical uplink while claiming to originate in my private &lt;span class="caps"&gt;VM&lt;/span&gt; subnet.&amp;nbsp;In &lt;code&gt;/etc/sysctl.conf&lt;/code&gt;, bridge filtering is done&amp;nbsp;on &lt;code&gt;bridge0&lt;/code&gt; rather than on every member as&amp;nbsp;well:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;net.link.bridge.pfil_bridge=1
net.link.bridge.pfil_member=0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This arrangement has&amp;nbsp;consequences. &lt;code&gt;sshd&lt;/code&gt; is enabled, but a new &lt;span class="caps"&gt;SSH&lt;/span&gt; connection from the physical &lt;span class="caps"&gt;LAN&lt;/span&gt; is blocked by &lt;span class="caps"&gt;PF&lt;/span&gt; unless I add a rule. The &lt;span class="caps"&gt;VM&lt;/span&gt; bridge is trusted, so guests can reach host services. It is not a guest-isolation design. If you do not run VMs, you do not need to copy this bridge, forwarding, &lt;span class="caps"&gt;DNS&lt;/span&gt; or &lt;span class="caps"&gt;NAT&lt;/span&gt; setup. If you do, read and validate your complete &lt;span class="caps"&gt;PF&lt;/span&gt; ruleset before loading it, especially while connected&amp;nbsp;remotely.&lt;/p&gt;
&lt;h2 id="chezmoi-familiar-dotfiles-freebsd-specific-behavior"&gt;Chezmoi: Familiar Dotfiles, FreeBSD-Specific&amp;nbsp;Behavior&lt;/h2&gt;
&lt;p&gt;My &lt;a href="https://hofstede.it/chezmoi-and-git-dotfiles-without-pretending-every-machine-is-the-same/"&gt;chezmoi and Git article&lt;/a&gt; explains why I keep source state and machine state separate. On this laptop, chezmoi points to&amp;nbsp;the &lt;code&gt;freebsd-workstation&lt;/code&gt; branch of my dotfiles repository. It manages fish, Neovim, Git, &lt;span class="caps"&gt;SSH&lt;/span&gt; configuration, Atuin, the OpenPGP card-state file and other small parts of my home&amp;nbsp;directory.&lt;/p&gt;
&lt;p&gt;The branch is not a blind copy of my Linux&amp;nbsp;setup. &lt;code&gt;.chezmoiignore&lt;/code&gt; excludes &lt;code&gt;systemd&lt;/code&gt; files, Linux shell startup files, Konsole state and other inappropriate paths&amp;nbsp;when &lt;code&gt;.chezmoi.os&lt;/code&gt; is &lt;code&gt;freebsd&lt;/code&gt;. Templates&amp;nbsp;select &lt;code&gt;/usr/local&lt;/code&gt; program paths and the&amp;nbsp;FreeBSD &lt;code&gt;oct-git&lt;/code&gt; wrapper. The &lt;span class="caps"&gt;SSH&lt;/span&gt; template points my Git host at the card agent&amp;#8217;s socket. The card-state template&amp;nbsp;uses &lt;code&gt;Pinentry&lt;/code&gt; on FreeBSD, rather than assuming a Linux keyring is&amp;nbsp;present.&lt;/p&gt;
&lt;p&gt;That is the practical value of a FreeBSD branch: I can share the preferences that make my shell and editor feel like mine without asking a FreeBSD machine to behave like&amp;nbsp;Fedora. &lt;code&gt;chezmoi diff&lt;/code&gt;, &lt;code&gt;chezmoi status&lt;/code&gt; and &lt;code&gt;chezmoi ignored&lt;/code&gt; are the useful commands when bringing that branch onto another&amp;nbsp;installation.&lt;/p&gt;
&lt;h2 id="a-rust-openpgp-card-stack-for-ssh-and-git"&gt;A Rust OpenPGP Card Stack for &lt;span class="caps"&gt;SSH&lt;/span&gt; and&amp;nbsp;Git&lt;/h2&gt;
&lt;p&gt;For card-backed operations I use three Rust tools from the &lt;a href="https://codeberg.org/openpgp-card"&gt;OpenPGP Card project&lt;/a&gt;: &lt;code&gt;oct&lt;/code&gt; to inspect and administer&amp;nbsp;cards, &lt;code&gt;oct-git&lt;/code&gt; to sign Git data,&amp;nbsp;and &lt;code&gt;openpgp-card-ssh-agent&lt;/code&gt; for &lt;span class="caps"&gt;SSH&lt;/span&gt; authentication. I wrote &lt;a href="https://codeberg.org/Larvitz/freebsd-openpgp-card-ports"&gt;FreeBSD ports for those tools&lt;/a&gt;, but this laptop currently runs my existing FreeBSD binaries&amp;nbsp;from &lt;code&gt;~/.cargo/bin&lt;/code&gt;. &lt;code&gt;pcsc-lite&lt;/code&gt; and &lt;code&gt;ccid&lt;/code&gt; provide the smart-card plumbing,&amp;nbsp;with &lt;code&gt;pcscd&lt;/code&gt; enabled at boot. My&amp;nbsp;local &lt;code&gt;pcscd&lt;/code&gt; runs&amp;nbsp;with &lt;code&gt;--disable-polkit&lt;/code&gt;. Think about that one before copying&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The agent starts at Xfce login through a desktop entry.&amp;nbsp;Its &lt;code&gt;Exec&lt;/code&gt; points to a small wrapper,&amp;nbsp;because &lt;code&gt;.desktop&lt;/code&gt; files do not expand shell variables&amp;nbsp;like &lt;code&gt;$XDG_RUNTIME_DIR&lt;/code&gt; in that field. The wrapper creates a private directory, puts the pinentry helper&amp;nbsp;on &lt;code&gt;PATH&lt;/code&gt; and starts the agent on a Unix socket, unless this login already has&amp;nbsp;one:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-eu

:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="p"&gt;:?XDG_RUNTIME_DIR must be set by the desktop session&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;socket_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="s2"&gt;/openpgp-card&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;socket&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket_dir&lt;/span&gt;&lt;span class="s2"&gt;/ssh-agent.sock&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;umask&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;077&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket_dir&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;700&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket_dir&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# The Rust card tools look for an executable named pinentry on PATH.&lt;/span&gt;
&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.local/libexec/openpgp-card:&lt;/span&gt;&lt;span class="nv"&gt;$PATH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;PATH

&lt;span class="c1"&gt;# An existing socket means this login already runs an agent.&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Refusing to replace unexpected path: &lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.cargo/bin/openpgp-card-ssh-agent&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;unix://&lt;/span&gt;&lt;span class="nv"&gt;$socket&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The runtime directory is fresh on every login, so an existing socket means the agent is already running in this session. The check stops a second launch from replacing&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;My &lt;span class="caps"&gt;SSH&lt;/span&gt; configuration selects that socket&amp;nbsp;with &lt;code&gt;IdentityAgent&lt;/code&gt; for the Git host.&amp;nbsp;Git&amp;#8217;s &lt;code&gt;gpg.program&lt;/code&gt; points to a wrapper&amp;nbsp;around &lt;code&gt;oct-git&lt;/code&gt; that puts my &lt;span class="caps"&gt;GTK&lt;/span&gt; pinentry&amp;nbsp;on &lt;code&gt;PATH&lt;/code&gt;. The card-state configuration asks for the &lt;span class="caps"&gt;PIN&lt;/span&gt;&amp;nbsp;through &lt;code&gt;Pinentry&lt;/code&gt;. &lt;span class="caps"&gt;SSH&lt;/span&gt; to my Forgejo instance authenticates with the card, and signed commits carry a real &lt;span class="caps"&gt;PGP&lt;/span&gt; signature.&amp;nbsp;Global &lt;code&gt;commit.gpgsign&lt;/code&gt; is &lt;strong&gt;off&lt;/strong&gt;: I sign when I ask for&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;One trap:&amp;nbsp;GnuPG&amp;#8217;s &lt;code&gt;scdaemon&lt;/code&gt; can take exclusive control of the card. On this&amp;nbsp;laptop, &lt;code&gt;oct list&lt;/code&gt; found no card&amp;nbsp;while &lt;code&gt;scdaemon&lt;/code&gt; held it, then found it immediately&amp;nbsp;after &lt;code&gt;gpgconf --kill scdaemon&lt;/code&gt;. I keep GnuPG installed for other OpenPGP work, but use the Rust tools as the normal card &lt;span class="caps"&gt;SSH&lt;/span&gt; and Git path. If your own card seems to vanish, check the competing card daemon before reinstalling three Rust&amp;nbsp;tools.&lt;/p&gt;
&lt;h2 id="agents-on-the-laptop"&gt;Agents on the&amp;nbsp;Laptop&lt;/h2&gt;
&lt;p&gt;I do agentic development on this machine as well. I use &lt;strong&gt;Codex&lt;/strong&gt; and &lt;strong&gt;Crush&lt;/strong&gt; here, the latter with models through OpenRouter. The models run elsewhere. The laptop is my terminal, browser, editor and build environment for that&amp;nbsp;work.&lt;/p&gt;
&lt;p&gt;That workload is a useful test of the whole desktop. It touches &lt;span class="caps"&gt;SSH&lt;/span&gt; authentication, Git signing, browsers, editor configuration, network access and ordinary package tooling. All of those pieces work together here. The machine has stopped feeling like a FreeBSD demonstration and started feeling like a laptop&amp;nbsp;again.&lt;/p&gt;
&lt;h2 id="if-you-are-setting-up-your-own-t480"&gt;If You Are Setting Up Your Own&amp;nbsp;T480&lt;/h2&gt;
&lt;p&gt;Use this as a map, not a file to paste wholesale. The order I would follow&amp;nbsp;is:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Get the Intel &lt;span class="caps"&gt;DRM&lt;/span&gt; driver and Xorg/Xfce session working, then confirm&amp;nbsp;that &lt;code&gt;/dev/dri/renderD128&lt;/code&gt; exists.&lt;/li&gt;
&lt;li&gt;Install firmware for the wireless adapter actually fitted to &lt;em&gt;your&lt;/em&gt; T480, then check the negotiated band and mode&amp;nbsp;with &lt;code&gt;ifconfig&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Check &lt;code&gt;vainfo&lt;/code&gt;, then check a real browser decoder. A capabilities page alone is&amp;nbsp;insufficient.&lt;/li&gt;
&lt;li&gt;Add Bluetooth audio only if you need it, and trace each hand-off from the paired device&amp;nbsp;to &lt;code&gt;/dev/dsp*&lt;/code&gt; to the PulseAudio&amp;nbsp;sink.&lt;/li&gt;
&lt;li&gt;Measure power under repeatable conditions before copying loader or &lt;span class="caps"&gt;CPU&lt;/span&gt; settings from somebody else&amp;#8217;s&amp;nbsp;machine.&lt;/li&gt;
&lt;li&gt;Bring in dotfiles and card tooling after the basic desktop and &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; service are&amp;nbsp;stable.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;My old post described the work required to make this T480 viable. This one describes the result: a FreeBSD laptop with Wi-Fi, browsers, a webcam, a speaker, smart-card authentication, development tools and low idle &lt;em&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt;-package&lt;/em&gt; power, where I can open the lid and get on with the day. That is exactly what I wanted from&amp;nbsp;it.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="thinkpad"/><category term="t480"/><category term="xfce"/><category term="wifi"/><category term="vaapi"/><category term="bluetooth"/><category term="power"/><category term="zfs"/><category term="pf"/><category term="chezmoi"/><category term="openpgp"/><category term="codex"/><category term="crush"/></entry><entry><title>The Ansible Guy Finally Automates His Own Patching: FreeBSD, Jails, Proxmox and RHEL</title><link href="https://hofstede.it/the-ansible-guy-finally-automates-his-own-patching-freebsd-jails-proxmox-and-rhel/" rel="alternate"/><published>2026-10-11T00:00:00+02:00</published><updated>2026-10-11T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-10-11:/the-ansible-guy-finally-automates-his-own-patching-freebsd-jails-proxmox-and-rhel/</id><summary type="html">&lt;p&gt;I write Ansible for a living, and I still patched most of my own servers by hand every week. This is how that ended: a Git-based inventory, defensive patch playbooks for FreeBSD (freebsd-update and pkgbase), Bastille thin jails, classic thick jails via my jailexec connection plugin, a Proxmox cluster and &lt;span class="caps"&gt;RHEL&lt;/span&gt;. Ansible Automation Platform runs it for me, but nothing here requires it. Plus the mistakes, including the reboot policy I should have checked before the first real&amp;nbsp;run.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Ansible Automation Platform job output of the Patch FreeBSD Systems job template, showing the jail tasks and a summary for a thick jail" src="https://hofstede.it/images/2026-10-11-patching-a-mixed-fleet-with-ansible.webp" title="Sunday evening, without the SSH sessions."&gt;&lt;/p&gt;
&lt;p&gt;There is a special kind of embarrassment in being the person who explains Ansible to other people during the week and then spends Sunday evening&amp;nbsp;typing &lt;code&gt;freebsd-update fetch install&lt;/code&gt; into a dozen &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;sessions.&lt;/p&gt;
&lt;p&gt;My &lt;span class="caps"&gt;RHEL&lt;/span&gt; machines were fine. They have been patched from Satellite through Ansible for a long time. Everything else was not: FreeBSD routers, jail hosts with Bastille and with&amp;nbsp;classic &lt;code&gt;jail.conf&lt;/code&gt; jails, a mail server, a Proxmox cluster, a Proxmox Backup Server. Each one had its own little ritual, and the rituals lived in my head and in my shell&amp;nbsp;history.&lt;/p&gt;
&lt;p&gt;This article is about replacing those rituals with playbooks. Not with one clever generic &amp;#8220;update everything&amp;#8221; task, but with patching logic that knows the difference&amp;nbsp;between &lt;code&gt;freebsd-update&lt;/code&gt; and pkgbase, between a thin and a thick jail, between a router that may reboot and a hypervisor that must&amp;nbsp;not.&lt;/p&gt;
&lt;p&gt;All host names and addresses in this article are placeholders&amp;nbsp;(&lt;code&gt;example.com&lt;/code&gt;, &lt;span class="caps"&gt;RFC&lt;/span&gt; 5737 and &lt;span class="caps"&gt;RFC&lt;/span&gt; 3849 documentation ranges). The structure and the code are what I actually&amp;nbsp;run.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-fleet"&gt;The&amp;nbsp;Fleet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#do-you-need-ansible-automation-platform"&gt;Do You Need Ansible Automation&amp;nbsp;Platform?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#inventory-as-code"&gt;Inventory as Code&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#address-hosts-the-way-you-can-reach-them-during-an-outage"&gt;Address Hosts the Way You Can Reach Them During an&amp;nbsp;Outage&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#one-service-user-bootstrapped-once"&gt;One Service User, Bootstrapped Once&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#firewalls-with-comments-that-must-survive"&gt;Firewalls With Comments That Must&amp;nbsp;Survive&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#rhel-the-boring-part"&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt;: The Boring&amp;nbsp;Part&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#debian-and-proxmox-patch-never-reboot"&gt;Debian and Proxmox: Patch, Never&amp;nbsp;Reboot&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#freebsd-two-base-systems-one-playbook"&gt;FreeBSD: Two Base Systems, One Playbook&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-freebsd-update-path"&gt;The freebsd-update&amp;nbsp;Path&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-pkgbase-path"&gt;The pkgbase&amp;nbsp;Path&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rebooting-carefully"&gt;Rebooting,&amp;nbsp;Carefully&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-a-patch-doesnt-restart"&gt;What a Patch Doesn&amp;#8217;t&amp;nbsp;Restart&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#jails-two-kinds-two-strategies"&gt;Jails: Two Kinds, Two Strategies&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#thin-bastille-jails"&gt;Thin Bastille&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#thick-jails-with-jailexec"&gt;Thick Jails With&amp;nbsp;jailexec&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#how-a-connection-plugin-ends-up-in-a-container"&gt;How a Connection Plugin Ends Up in a&amp;nbsp;Container&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#defensive-practices"&gt;Defensive&amp;nbsp;Practices&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-changed"&gt;What&amp;nbsp;Changed&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-fleet"&gt;The&amp;nbsp;Fleet&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Host (placeholder)&lt;/th&gt;
&lt;th&gt;&lt;span class="caps"&gt;OS&lt;/span&gt;&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;edge1&lt;/code&gt; … &lt;code&gt;edge5.example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;FreeBSD 15.1, &lt;span class="caps"&gt;FRR&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;BGP&lt;/span&gt;&amp;nbsp;routers, &lt;code&gt;doas&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;mail.example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;FreeBSD 15.1&lt;/td&gt;
&lt;td&gt;four thick jails&amp;nbsp;in &lt;code&gt;jail.conf&lt;/code&gt;, encrypted dataset&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;web.example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;FreeBSD 15.1, &lt;strong&gt;pkgbase&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;nine thin Bastille jails&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;social.example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;FreeBSD 15.1&lt;/td&gt;
&lt;td&gt;four thin Bastille jails (Mastodon)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;pve1&lt;/code&gt;, &lt;code&gt;pve2.example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; 9 (Debian 13)&lt;/td&gt;
&lt;td&gt;two-node cluster with QDevice&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;pbs.example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Proxmox Backup Server 4&lt;/td&gt;
&lt;td&gt;Debian 13&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rhel*.example.com&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; 9/10&lt;/td&gt;
&lt;td&gt;Satellite-managed, already automated&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Three operating system families, four ways to update a base system&amp;nbsp;(&lt;code&gt;freebsd-update&lt;/code&gt;,&amp;nbsp;pkgbase, &lt;code&gt;apt&lt;/code&gt; and &lt;code&gt;dnf&lt;/code&gt;), two jail managers, and one hypervisor cluster that happens to run the automation controller itself. That last detail will matter&amp;nbsp;later.&lt;/p&gt;
&lt;p&gt;So far, real runs cover the mail server, its jails, and both Bastille hosts. The router and Proxmox paths are written and reviewed, but have not had their first real run yet. I&amp;#8217;ll say so where it&amp;nbsp;matters.&lt;/p&gt;
&lt;h2 id="do-you-need-ansible-automation-platform"&gt;Do You Need Ansible Automation&amp;nbsp;Platform?&lt;/h2&gt;
&lt;p&gt;No.&lt;/p&gt;
&lt;p&gt;I run this through Ansible Automation Platform (&lt;span class="caps"&gt;AAP&lt;/span&gt;), because I already have it and because a web &lt;span class="caps"&gt;UI&lt;/span&gt; with job history, credential storage, &lt;span class="caps"&gt;RBAC&lt;/span&gt;, and a &amp;#8220;launch&amp;#8221; button is genuinely nice for something you do every&amp;nbsp;week.&lt;/p&gt;
&lt;p&gt;But every playbook in this article is a plain&amp;nbsp;playbook. &lt;code&gt;ansible-playbook -i inventory/hosts patch_freebsd.yml&lt;/code&gt; from a laptop works the same, as long as it has the same collections, a compatible ansible-core, the same credentials, and the same inventory checkout. A systemd timer, a cron job, Semaphore, or a &lt;span class="caps"&gt;CI&lt;/span&gt; pipeline would all be fine schedulers. What &lt;span class="caps"&gt;AAP&lt;/span&gt; added for me was operational, not&amp;nbsp;functional:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Credentials stay in one place.&lt;/strong&gt; The private key of the service user lives in &lt;span class="caps"&gt;AAP&lt;/span&gt;, encrypted, and never on my&amp;nbsp;laptop.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Job templates with prompts.&lt;/strong&gt; One template, launched with&amp;nbsp;a &lt;code&gt;limit&lt;/code&gt; and a job type&amp;nbsp;of &lt;code&gt;check&lt;/code&gt; or &lt;code&gt;run&lt;/code&gt;, covers both the dry run and the real&amp;nbsp;thing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Collections only from my own Private Automation Hub.&lt;/strong&gt; More on that in the defensive practices&amp;nbsp;section.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you don&amp;#8217;t have &lt;span class="caps"&gt;AAP&lt;/span&gt;, skip the parts that mention it. You lose convenience, not&amp;nbsp;capability.&lt;/p&gt;
&lt;h2 id="inventory-as-code"&gt;Inventory as&amp;nbsp;Code&lt;/h2&gt;
&lt;p&gt;The inventory lives in its own Git repository, separate from the playbooks. &lt;span class="caps"&gt;AAP&lt;/span&gt; syncs it as an inventory source, and the same checkout works&amp;nbsp;locally.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;external_inventory/
├── README.md
└── inventory/
    ├── hosts                  # INI: hosts and groups only, no inline variables
    ├── group_vars/
    │   ├── all.yml
    │   ├── freebsd.yml
    │   ├── mail_jails.yml
    │   └── proxmox.yml
    └── host_vars/
        ├── edge1.example.com.yml
        ├── mail.example.com.yml
        └── ...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I like &lt;span class="caps"&gt;INI&lt;/span&gt; for the host list because it reads like a table of contents, and &lt;span class="caps"&gt;YAML&lt;/span&gt; for everything else because variables deserve structure. The one rule: &lt;strong&gt;no variables&amp;nbsp;in &lt;code&gt;hosts&lt;/code&gt;&lt;/strong&gt;. Group membership lives in one file, and host-specific overrides have exactly one predictable place: that host&amp;#8217;s file&amp;nbsp;in &lt;code&gt;host_vars/&lt;/code&gt;.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[freebsd_routers]&lt;/span&gt;
&lt;span class="na"&gt;edge1.example.com&lt;/span&gt;
&lt;span class="na"&gt;edge2.example.com&lt;/span&gt;
&lt;span class="c1"&gt;# ... edge3 to edge5&lt;/span&gt;

&lt;span class="k"&gt;[freebsd_mail]&lt;/span&gt;
&lt;span class="na"&gt;mail.example.com&lt;/span&gt;

&lt;span class="k"&gt;[bastille_hosts]&lt;/span&gt;
&lt;span class="na"&gt;web.example.com&lt;/span&gt;
&lt;span class="na"&gt;social.example.com&lt;/span&gt;

&lt;span class="c1"&gt;# base system updated via pkg, not freebsd-update&lt;/span&gt;
&lt;span class="k"&gt;[freebsd_pkgbase]&lt;/span&gt;
&lt;span class="na"&gt;web.example.com&lt;/span&gt;

&lt;span class="k"&gt;[freebsd:children]&lt;/span&gt;
&lt;span class="na"&gt;freebsd_routers&lt;/span&gt;
&lt;span class="na"&gt;freebsd_mail&lt;/span&gt;
&lt;span class="na"&gt;bastille_hosts&lt;/span&gt;

&lt;span class="c1"&gt;# thick jails on mail.example.com, reached through the jail host&lt;/span&gt;
&lt;span class="k"&gt;[mail_jails]&lt;/span&gt;
&lt;span class="na"&gt;mail-mta&lt;/span&gt;
&lt;span class="na"&gt;mail-webmail&lt;/span&gt;
&lt;span class="na"&gt;mail-client&lt;/span&gt;
&lt;span class="c1"&gt;# ... (excerpt)&lt;/span&gt;

&lt;span class="c1"&gt;# Proxmox VE nodes and Proxmox Backup Server: patched, never rebooted&lt;/span&gt;
&lt;span class="k"&gt;[proxmox]&lt;/span&gt;
&lt;span class="na"&gt;pve1.example.com&lt;/span&gt;
&lt;span class="na"&gt;pve2.example.com&lt;/span&gt;
&lt;span class="na"&gt;pbs.example.com&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The groups describe &lt;strong&gt;behaviour&lt;/strong&gt;, not just&amp;nbsp;location. &lt;code&gt;freebsd_pkgbase&lt;/code&gt; switches the base system update&amp;nbsp;method. &lt;code&gt;bastille_hosts&lt;/code&gt; adds release and jail&amp;nbsp;handling. &lt;code&gt;mail_jails&lt;/code&gt; is deliberately &lt;strong&gt;not&lt;/strong&gt; a child&amp;nbsp;of &lt;code&gt;freebsd&lt;/code&gt;, so the host play never accidentally runs against a&amp;nbsp;jail.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# group_vars/all.yml&lt;/span&gt;
&lt;span class="nt"&gt;ansible_user&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ansible&lt;/span&gt;
&lt;span class="nt"&gt;ansible_port&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2222&lt;/span&gt;
&lt;span class="nt"&gt;ansible_python_interpreter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;auto_silent&lt;/span&gt;

&lt;span class="c1"&gt;# group_vars/freebsd.yml&lt;/span&gt;
&lt;span class="nt"&gt;ansible_become_method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;community.general.doas&lt;/span&gt;

&lt;span class="c1"&gt;# group_vars/proxmox.yml&lt;/span&gt;
&lt;span class="nt"&gt;ansible_become_method&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;sudo&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="address-hosts-the-way-you-can-reach-them-during-an-outage"&gt;Address Hosts the Way You Can Reach Them During an&amp;nbsp;Outage&lt;/h3&gt;
&lt;p&gt;One &lt;code&gt;host_vars&lt;/code&gt; entry per host turned out to be&amp;nbsp;non-negotiable:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host_vars/edge1.example.com.yml&lt;/span&gt;
&lt;span class="c1"&gt;# Provider address: SSH must not depend on the network this host is part of&lt;/span&gt;
&lt;span class="nt"&gt;ansible_host&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;192.0.2.10&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;My routers&amp;#8217; &lt;span class="caps"&gt;DNS&lt;/span&gt; names point to addresses inside the network they route. That is perfectly fine until a playbook reboots the router that carries the active path: the &lt;span class="caps"&gt;SSH&lt;/span&gt; session to that address would disappear together with the route,&amp;nbsp;the &lt;code&gt;reboot&lt;/code&gt; module would time out, and the play would stop halfway through. Pinning every host to its &lt;strong&gt;provider-assigned&lt;/strong&gt; address removes the dependency on the routes being patched. It doesn&amp;#8217;t remove every dependency: if your automation controller&amp;#8217;s own egress runs through that same network, the controller can still lose its path, so check where its traffic actually&amp;nbsp;leaves.&lt;/p&gt;
&lt;h2 id="one-service-user-bootstrapped-once"&gt;One Service User, Bootstrapped&amp;nbsp;Once&lt;/h2&gt;
&lt;p&gt;Every host gets the same service user, the same public key, and passwordless privilege&amp;nbsp;escalation: &lt;code&gt;doas&lt;/code&gt; on&amp;nbsp;FreeBSD, &lt;code&gt;sudo&lt;/code&gt; on Debian. Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; and Proxmox Backup Server don&amp;#8217;t&amp;nbsp;install &lt;code&gt;sudo&lt;/code&gt; by default, so it has to be there before the bootstrap runs. A one-time bootstrap playbook, run as root from my workstation, sets it&amp;nbsp;up:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Ensure python is available&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;freebsd&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;gather_facts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Install python312 if no python3 is present&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.raw&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ls /usr/local/bin/python3.* &amp;gt;/dev/null 2&amp;gt;&amp;amp;1 || pkg install -y python312&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;python_install&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;#39;Installing&amp;#39;&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;in&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;python_install.stdout&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;raw&lt;/code&gt; first, because one of my routers had no Python at all. The same check, wrapped&amp;nbsp;in &lt;code&gt;jexec&lt;/code&gt;, puts Python into the thick mail jails, which jailexec needs because modules run inside the jail. After that, ordinary modules create the user, install the key, and add exactly one line&amp;nbsp;to &lt;code&gt;doas.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Check for existing doas.conf&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.stat&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/usr/local/etc/doas.conf&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;doas_conf&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Existing files keep their mode; only a newly created file gets one&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Allow the service user passwordless doas&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.lineinfile&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/usr/local/etc/doas.conf&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;line&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;permit nopass ansible as root&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;create&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;omit&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;if&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;doas_conf.stat.exists&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;else&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;0640&amp;#39;&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;validate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;doas -C %s&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;stat&lt;/code&gt; dance exists because the first dry run showed me that a&amp;nbsp;plain &lt;code&gt;mode: "0640"&lt;/code&gt; would silently change the permissions of every&amp;nbsp;existing &lt;code&gt;doas.conf&lt;/code&gt;. That is exactly the kind of&amp;nbsp;thing &lt;code&gt;--check --diff&lt;/code&gt; is&amp;nbsp;for.&lt;/p&gt;
&lt;h3 id="firewalls-with-comments-that-must-survive"&gt;Firewalls With Comments That Must&amp;nbsp;Survive&lt;/h3&gt;
&lt;p&gt;All of my hosts filter &lt;span class="caps"&gt;SSH&lt;/span&gt; to a small list of trusted addresses, and the controller&amp;#8217;s egress address had to be added. My firewall configs document those lists with a comment line&amp;nbsp;underneath:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;trusted_ipv4 = &amp;quot;{ 198.51.100.7, 203.0.113.42 }&amp;quot;
#                   ^VPN          ^OFFICE
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A &lt;code&gt;replace&lt;/code&gt; with a regex would have appended the new address and left the comment line out of sync. So the bootstrap uses a small custom module from the playbook&amp;nbsp;repository&amp;#8217;s &lt;code&gt;library/&lt;/code&gt; directory: it appends the address to the first trusted list it finds, extends the annotation line&amp;nbsp;with &lt;code&gt;^AAP&lt;/code&gt;, leaves every other byte alone, validates&amp;nbsp;with &lt;code&gt;pfctl -nf&lt;/code&gt; or &lt;code&gt;sh -n&lt;/code&gt;, and supports check and diff&amp;nbsp;mode:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;-trusted_ipv4 = &amp;quot;{ 198.51.100.7, 203.0.113.42 }&amp;quot;
-#                   ^VPN          ^OFFICE
+trusted_ipv4 = &amp;quot;{ 198.51.100.7, 203.0.113.42, 203.0.113.10 }&amp;quot;
+#                   ^VPN          ^OFFICE       ^AAP
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Sixty lines of Python, and a configuration file that is still documentation&amp;nbsp;afterwards.&lt;/p&gt;
&lt;h2 id="rhel-the-boring-part"&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt;: The Boring&amp;nbsp;Part&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; was already automated, and it is the least interesting case on purpose. Satellite promotes content views, and the playbook just applies whatever the host is allowed to&amp;nbsp;see:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Patch all systems and reboot if required&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;host&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;become&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Ensure all updates are applied&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.package&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;*&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;latest&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;update_cache&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;update_only&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Check whether a reboot is required&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;dnf needs-restarting -r&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;result&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;check_mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;failed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;result.rc not in [0, 1]&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Reboot if needed&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.reboot&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;result.rc == 1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;check_mode: false&lt;/code&gt; on the probe matters: without it, a check run skips the&amp;nbsp;command, &lt;code&gt;result&lt;/code&gt; has&amp;nbsp;no &lt;code&gt;rc&lt;/code&gt;, and the reboot condition fails. Even with it, a check run reports whether the host needs a reboot &lt;strong&gt;now&lt;/strong&gt;, not whether it would after the updates it didn&amp;#8217;t&amp;nbsp;install.&lt;/p&gt;
&lt;p&gt;The control is in Satellite, not in the playbook: what reaches production is decided when a content view is promoted. If you don&amp;#8217;t run Satellite, the same pattern works with plain repositories, you just lose the&amp;nbsp;gate.&lt;/p&gt;
&lt;h2 id="debian-and-proxmox-patch-never-reboot"&gt;Debian and Proxmox: Patch, Never&amp;nbsp;Reboot&lt;/h2&gt;
&lt;p&gt;Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; is Debian with opinions. The most important one: &lt;strong&gt;always &lt;code&gt;dist-upgrade&lt;/code&gt; (or &lt;code&gt;apt full-upgrade&lt;/code&gt;), never a plain upgrade&lt;/strong&gt;. Proxmox updates regularly change dependencies: new packages get pulled in, old ones are replaced or&amp;nbsp;removed. &lt;code&gt;apt-get upgrade&lt;/code&gt; does neither,&amp;nbsp;and &lt;code&gt;apt upgrade&lt;/code&gt; installs new packages but never removes any. Either can leave Proxmox packages held back or stuck halfway through a&amp;nbsp;transition.&lt;/p&gt;
&lt;p&gt;This playbook is one of the two that haven&amp;#8217;t had a real run&amp;nbsp;yet.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Patch Proxmox VE and Backup Server hosts&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;proxmox&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;serial&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;1&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;max_fail_percentage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;become&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;gather_facts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Detect Proxmox VE cluster node&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.stat&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/usr/bin/pvecm&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;pvecm_bin&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Check cluster quorum before upgrade&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;/usr/bin/pvecm&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;quorum_before&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;check_mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;failed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;quorum_before.stdout is not search(&amp;#39;Quorate:\s+Yes&amp;#39;)&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;pvecm_bin.stat.exists&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Apply all updates (Proxmox requires dist-upgrade)&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.apt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;update_cache&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;upgrade&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;dist&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;dpkg_options&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;force-confdef,force-confold&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;DEBIAN_FRONTEND&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;noninteractive&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;async&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;if&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;ansible_check_mode&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;else&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;3600&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;poll&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;15&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Check cluster quorum after upgrade&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;/usr/bin/pvecm&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;quorum_after&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;check_mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;retries&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;12&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;until&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;quorum_after.stdout is search(&amp;#39;Quorate:\s+Yes&amp;#39;)&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;pvecm_bin.stat.exists&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Three details are worth&amp;nbsp;explaining:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Quorum before and after.&lt;/strong&gt; A node is only touched if the cluster is quorate, and the next node only if it still is afterwards. On the Backup Server the check is skipped,&amp;nbsp;because &lt;code&gt;pvecm&lt;/code&gt; doesn&amp;#8217;t exist&amp;nbsp;there.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;async&lt;/code&gt; for apt, except in check mode.&lt;/strong&gt; The automation controller runs as a &lt;span class="caps"&gt;VM&lt;/span&gt; on this cluster. If a package upgrade restarts networking on the node, the &lt;span class="caps"&gt;SSH&lt;/span&gt; session can drop.&amp;nbsp;With &lt;code&gt;async&lt;/code&gt;, &lt;code&gt;dpkg&lt;/code&gt; keeps running on the node as a detached job instead of dying with the session. That covers exactly one failure mode: a package script can still fail, a node can still crash, and a job that outlives the async timeout gets killed. Ansible&amp;#8217;s async mode doesn&amp;#8217;t support check mode, so during&amp;nbsp;a &lt;code&gt;--check&lt;/code&gt; run the task runs synchronously&amp;nbsp;(&lt;code&gt;async: 0&lt;/code&gt;),&amp;nbsp;where &lt;code&gt;apt&lt;/code&gt; only simulates&amp;nbsp;anyway.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No reboot.&lt;/strong&gt; Hypervisors and the backup server are never rebooted by automation. The playbook compares the running kernel with the newest installed one,&amp;nbsp;checks &lt;code&gt;/var/run/reboot-required&lt;/code&gt;, prints the result with&amp;nbsp;a &lt;code&gt;debug&lt;/code&gt; task, and publishes it&amp;nbsp;through &lt;code&gt;set_stats&lt;/code&gt;. In &lt;span class="caps"&gt;AAP&lt;/span&gt;, that lands in the job&amp;#8217;s artifacts and is passed on to later workflow nodes. On the command line, custom stats only appear in the recap&amp;nbsp;with &lt;code&gt;show_custom_stats = True&lt;/code&gt;:&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Record reboot status&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.set_fact&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;reboot_required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;reboot_flag.stat.exists&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;or&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;newest_kernel.stdout&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;!=&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;running_kernel.stdout&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Publish reboot status&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.set_stats&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;proxmox_reboot_required&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{inventory_hostname:&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;reboot_required&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;bool}&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;aggregate&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I&amp;#8217;ll do the reboot myself, migrating VMs first, on an evening I&amp;nbsp;choose.&lt;/p&gt;
&lt;h2 id="freebsd-two-base-systems-one-playbook"&gt;FreeBSD: Two Base Systems, One&amp;nbsp;Playbook&lt;/h2&gt;
&lt;p&gt;The FreeBSD playbook started as a quick sketch and ended up based on a much more paranoid version I had written earlier and never tested. The core idea: &lt;strong&gt;never trust a single return code&lt;/strong&gt;.&lt;/p&gt;
&lt;h3 id="the-freebsd-update-path"&gt;The freebsd-update&amp;nbsp;Path&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Host | freebsd-update fetch&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;/usr/sbin/freebsd-update&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;--not-running-from-cron&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;fetch&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;PAGER&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/bin/cat&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_host_fetch&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;check_mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;retries&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;3&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;delay&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;30&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;until&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_host_fetch is succeeded&lt;/span&gt;

&lt;span class="c1"&gt;# updatesready: rc 0 = updates staged, rc 2 = nothing to do&lt;/span&gt;
&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Host | Check for staged updates&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;/usr/sbin/freebsd-update&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;updatesready&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_host_ready&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;failed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_host_ready.rc not in [0, 2]&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;check_mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;

&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Host | freebsd-update install&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;/usr/sbin/freebsd-update&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;--not-running-from-cron&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;install&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;PAGER&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/bin/cat&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_host_install&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;#39;Installing&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;updates&amp;#39;&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;in&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;_fu_host_install.stdout&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_host_ready.rc == 0&lt;/span&gt;

&lt;span class="c1"&gt;# Patch-level updates: the only accepted leftover is a two-stage install&lt;/span&gt;
&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Host | Verify the install is complete&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;/usr/sbin/freebsd-update&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;updatesready&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_host_ready_after&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;failed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;&amp;gt;-&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;_fu_host_ready_after.rc not in [0, 2]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;or (_fu_host_ready_after.rc == 0&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;and &amp;#39;Please reboot and run&amp;#39; not in _fu_host_install.stdout | default(&amp;#39;&amp;#39;))&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_host_install is not skipped&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;fetch&lt;/code&gt; only stages files&amp;nbsp;into &lt;code&gt;/var/db/freebsd-update&lt;/code&gt;, exactly like the&amp;nbsp;daily &lt;code&gt;freebsd-update cron&lt;/code&gt; would. That is why it runs even in check mode:&amp;nbsp;a &lt;code&gt;--check&lt;/code&gt; run tells me which hosts actually have pending updates, without installing&amp;nbsp;anything. &lt;code&gt;updatesready&lt;/code&gt; turns &amp;#8220;did anything happen?&amp;#8221; into an explicit answer instead of output parsing. The verification step catches an install that silently left something&amp;nbsp;behind.&lt;/p&gt;
&lt;p&gt;For patch-level&amp;nbsp;updates, &lt;code&gt;install&lt;/code&gt; normally finishes in one pass. The two-stage case (kernel first, reboot, install again) is mostly what release upgrades do, and those add another pass to remove old shared libraries, which this playbook doesn&amp;#8217;t support. If a patch install does ask for a reboot, the reboot block runs the&amp;nbsp;second &lt;code&gt;install&lt;/code&gt; pass once the host is back. A&amp;nbsp;final &lt;code&gt;updatesready&lt;/code&gt; afterwards must report nothing pending, otherwise the host is marked failed. If the host is not allowed to reboot, the summary reports the reboot as pending, which in that case also means the install isn&amp;#8217;t&amp;nbsp;finished.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;--not-running-from-cron&lt;/code&gt; and &lt;code&gt;PAGER=/bin/cat&lt;/code&gt; are the two flags every&amp;nbsp;automated &lt;code&gt;freebsd-update&lt;/code&gt; needs. Without a &lt;span class="caps"&gt;TTY&lt;/span&gt; and without them, it either refuses to run or waits for a pager that nobody will ever&amp;nbsp;quit.&lt;/p&gt;
&lt;h3 id="the-pkgbase-path"&gt;The pkgbase&amp;nbsp;Path&lt;/h3&gt;
&lt;p&gt;One of my hosts runs &lt;a href="https://hofstede.it/three-test-jails-on-a-512-mb-free-tier-vps/"&gt;pkgbase&lt;/a&gt;.&amp;nbsp;There, &lt;code&gt;freebsd-update&lt;/code&gt; refuses to work, and the base system is just another set of packages. The playbook detects it the same&amp;nbsp;way &lt;code&gt;freebsd-update&lt;/code&gt; itself does, by&amp;nbsp;asking &lt;code&gt;pkg&lt;/code&gt; who&amp;nbsp;owns &lt;code&gt;/usr/bin/uname&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Host | Detect pkgbase&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;/usr/local/sbin/pkg&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;which&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;/usr/bin/uname&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_pkgbase_probe&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;failed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;check_mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;

&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Host | pkgbase detection must match the inventory&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.assert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;that&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;(_fu_pkgbase_probe.rc == 0) == (inventory_hostname in groups[&amp;#39;freebsd_pkgbase&amp;#39;])&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;fail_msg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;pkgbase&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;detection&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;and&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;inventory&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;group&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;disagree.&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Fix&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;the&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;inventory.&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Detection alone would work. Detection &lt;strong&gt;plus&lt;/strong&gt; an inventory group that has to agree with it means a host that changes its nature without anyone updating the inventory fails loudly instead of being patched the wrong way. On pkgbase&amp;nbsp;hosts, &lt;code&gt;community.general.pkgng&lt;/code&gt; with &lt;code&gt;name: "*"&lt;/code&gt; and &lt;code&gt;state: latest&lt;/code&gt; updates base and ports in one go, and a failure there is fatal, because it is the base&amp;nbsp;system.&lt;/p&gt;
&lt;h3 id="rebooting-carefully"&gt;Rebooting,&amp;nbsp;Carefully&lt;/h3&gt;
&lt;p&gt;After all updates, the playbook&amp;nbsp;compares &lt;code&gt;freebsd-version -k&lt;/code&gt; (installed kernel) with the running&amp;nbsp;kernel. &lt;code&gt;fbsd_update_reboot&lt;/code&gt; decides what happens&amp;nbsp;next: &lt;code&gt;auto&lt;/code&gt; (the default) reboots only if the two&amp;nbsp;differ, &lt;code&gt;always&lt;/code&gt; reboots regardless of the kernel,&amp;nbsp;and &lt;code&gt;never&lt;/code&gt; only reports a pending reboot. A reboot additionally requires&amp;nbsp;that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;/var/run/noshutdown&lt;/code&gt; does not&amp;nbsp;exist,&lt;/li&gt;
&lt;li&gt;the run is not in check&amp;nbsp;mode.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;It reboots&amp;nbsp;with &lt;code&gt;shutdown -r&lt;/code&gt; rather&amp;nbsp;than &lt;code&gt;reboot&lt;/code&gt;,&amp;nbsp;so &lt;code&gt;rc.shutdown&lt;/code&gt; stops jails and services cleanly. Afterwards it verifies the running kernel, waits until every jail that was running before is running again, and on routers waits&amp;nbsp;for &lt;code&gt;service frr status&lt;/code&gt;. Those are startup checks: they prove that the jails and &lt;span class="caps"&gt;FRR&lt;/span&gt; came up, not that mail is delivered or that &lt;span class="caps"&gt;BGP&lt;/span&gt; sessions and routes are back. Everything runs&amp;nbsp;with &lt;code&gt;serial: 1&lt;/code&gt; and &lt;code&gt;max_fail_percentage: 0&lt;/code&gt;: the first broken host stops the&amp;nbsp;rollout.&lt;/p&gt;
&lt;p&gt;Errors that are not about the base system (a single jail&amp;nbsp;failing &lt;code&gt;pkg upgrade&lt;/code&gt;, a release that cannot be updated) are &lt;strong&gt;collected&lt;/strong&gt; instead of aborting. The host is marked failed at the end with a summary, but a broken &lt;span class="caps"&gt;PHP&lt;/span&gt; package in one jail doesn&amp;#8217;t leave the other eight&amp;nbsp;unpatched.&lt;/p&gt;
&lt;h3 id="what-a-patch-doesnt-restart"&gt;What a Patch Doesn&amp;#8217;t&amp;nbsp;Restart&lt;/h3&gt;
&lt;p&gt;Installed is not the same as running. A kernel update ends in a reboot, and a changed jail base ends in a jail restart (see below). Two cases are not covered by the playbook today: a&amp;nbsp;userland-only &lt;code&gt;freebsd-update&lt;/code&gt; on the host, where the kernel version stays the same, and&amp;nbsp;a &lt;code&gt;pkg upgrade&lt;/code&gt; that replaces binaries or libraries on the host or in a jail. FreeBSD packages generally don&amp;#8217;t restart running services on upgrade, so those daemons keep running the old code until someone restarts them. For now, the job summary lists which hosts and jails had packages upgraded,&amp;nbsp;and &lt;code&gt;fbsd_update_reboot: always&lt;/code&gt; is the blunt instrument for a host where I want everything restarted. Detecting the affected services and restarting exactly those is the next gap to&amp;nbsp;close.&lt;/p&gt;
&lt;h2 id="jails-two-kinds-two-strategies"&gt;Jails: Two Kinds, Two&amp;nbsp;Strategies&lt;/h2&gt;
&lt;p&gt;This is where most &amp;#8220;patch everything&amp;#8221; playbooks give up. I have two different kinds of jails, and each needs different&amp;nbsp;handling.&lt;/p&gt;
&lt;h3 id="thin-bastille-jails"&gt;Thin Bastille&amp;nbsp;Jails&lt;/h3&gt;
&lt;p&gt;Thin Bastille jails nullfs-mount a shared, read-only base release. Updating the release updates every jail that uses&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Bastille | bastille update for releases&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;/usr/local/bin/bastille&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;update&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;item.name&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;PAGER&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/bin/cat&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;loop&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;_fu_releases&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;loop_control&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;index_var&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_idx&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_rel_update&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;changed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;&amp;gt;-&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;&amp;#39;Installing updates&amp;#39; in _fu_rel_update.stdout&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;or _fu_rel_update.stdout is search(&amp;#39;Number of packages to be &amp;#39;)&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;failed_when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;&amp;gt;-&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;_fu_rel_update.rc != 0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;or &amp;#39;[ERROR]&amp;#39; in (_fu_rel_update.stdout ~ _fu_rel_update.stderr)&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ignore_errors&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;item.usable&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;item.pkgbase or (_fu_rel_ready.results[_idx].rc | default(2)) == 0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Before that, the playbook runs its&amp;nbsp;own &lt;code&gt;freebsd-update fetch&lt;/code&gt; and &lt;code&gt;updatesready&lt;/code&gt; against each release directory, with the&amp;nbsp;arguments &lt;code&gt;bastille update&lt;/code&gt; uses&amp;nbsp;(&lt;code&gt;-b &amp;lt;release&amp;gt; -d &amp;lt;release&amp;gt;/var/db/freebsd-update --currently-running &amp;lt;release&amp;gt;&lt;/code&gt;), because the Bastille version I run doesn&amp;#8217;t&amp;nbsp;propagate &lt;code&gt;freebsd-update&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s exit code. That gives three states per release: nothing staged (the update is skipped), updated&amp;nbsp;(&lt;code&gt;changed&lt;/code&gt;, recognised from the output&amp;nbsp;of &lt;code&gt;freebsd-update&lt;/code&gt; or, for pkgbase&amp;nbsp;releases, &lt;code&gt;pkg&lt;/code&gt;), and failed (collected as an error and excluded from the restart below).&amp;nbsp;For &lt;code&gt;freebsd-update&lt;/code&gt; releases, a&amp;nbsp;final &lt;code&gt;updatesready&lt;/code&gt; afterwards must report nothing pending, so &amp;#8220;updated&amp;#8221; means installed, not just attempted.&amp;nbsp;Then &lt;code&gt;bastille pkg &amp;lt;jail&amp;gt; upgrade -y&lt;/code&gt; runs per running jail, skipping jails&amp;nbsp;without &lt;code&gt;pkg&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The step I missed in the first version: &lt;strong&gt;an updated release only takes effect for running services after the jail restarts&lt;/strong&gt;. On my first real run, all nine jails reported the new patch level and kept running the old libraries. Now the playbook finds the jails&amp;nbsp;whose &lt;code&gt;fstab&lt;/code&gt; mounts an updated release and restarts exactly those, unless the host reboots&amp;nbsp;anyway:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Bastille | Find jails using an updated release&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.find&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;_fu_jailsdir&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;patterns&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;fstab&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;recurse&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;depth&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;contains&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;^{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;(_fu_releasesdir&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;~&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;~&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;item.item.name)&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;regex_escape&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}\s&amp;#39;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;loop&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;_fu_rel_update.results&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;select(&amp;#39;changed&amp;#39;)&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;reject(&amp;#39;failed&amp;#39;)&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;list&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;register&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;_fu_rel_jails&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="thick-jails-with-jailexec"&gt;Thick Jails With&amp;nbsp;jailexec&lt;/h3&gt;
&lt;p&gt;My mail server runs classic thick jails&amp;nbsp;from &lt;code&gt;jail.conf&lt;/code&gt;. Each has its own base system and&amp;nbsp;needs &lt;code&gt;freebsd-update&lt;/code&gt; &lt;strong&gt;inside&lt;/strong&gt; the jail. For that I use &lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;jailexec&lt;/a&gt;, my connection plugin that SSHes to the jail host and wraps every command&amp;nbsp;in &lt;code&gt;jexec&lt;/code&gt;. The jails don&amp;#8217;t need &lt;span class="caps"&gt;SSH&lt;/span&gt; or an &lt;span class="caps"&gt;IP&lt;/span&gt; address, only&amp;nbsp;Python.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# group_vars/mail_jails.yml&lt;/span&gt;
&lt;span class="nt"&gt;ansible_connection&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;chofstede.jailexec.jailexec&lt;/span&gt;
&lt;span class="nt"&gt;ansible_jail_host&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;192.0.2.25&lt;/span&gt;
&lt;span class="nt"&gt;ansible_jail_privilege_escalation&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;doas&lt;/span&gt;
&lt;span class="nt"&gt;ansible_become&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host_vars/mail-webmail.yml&lt;/span&gt;
&lt;span class="nt"&gt;ansible_jail_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;webmail&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Each jail is a normal inventory host. The jail play runs first, before the host play,&amp;nbsp;with &lt;code&gt;serial: 1&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Jail | Build freebsd-update arguments&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.set_fact&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;_fj_fu_args&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/usr/sbin/freebsd-update&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;--not-running-from-cron&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;--currently-running&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;_fj_userland_before.stdout&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;trim&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;regex_replace(&amp;#39;-p[0-9]+$&amp;#39;,&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;&amp;#39;)&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;--currently-running&lt;/code&gt; is the important part. Inside a&amp;nbsp;jail, &lt;code&gt;freebsd-update&lt;/code&gt; would otherwise derive the version from the &lt;strong&gt;host&amp;#8217;s&lt;/strong&gt; kernel, which is not what the jail&amp;#8217;s userland is running. The playbook passes the jail&amp;#8217;s own userland release instead, then runs the same&amp;nbsp;fetch, &lt;code&gt;updatesready&lt;/code&gt;, install, and verify sequence as on the&amp;nbsp;host.&lt;/p&gt;
&lt;p&gt;If the jail&amp;#8217;s base changed and the host does &lt;strong&gt;not&lt;/strong&gt; reboot, the host play restarts that jail afterwards&amp;nbsp;with &lt;code&gt;service jail restart&lt;/code&gt;. If the host reboots, the jails restart&amp;nbsp;anyway.&lt;/p&gt;
&lt;p&gt;That coupling has a consequence&amp;nbsp;for &lt;code&gt;--limit&lt;/code&gt;. The jails are separate inventory hosts, and Ansible knows nothing about their relationship&amp;nbsp;to &lt;code&gt;mail.example.com&lt;/code&gt; beyond a connection variable. Limiting a run to the mail server skips the jail play; limiting it to one jail skips the host play that would restart it. A run for the mail server therefore always selects&amp;nbsp;both:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible-playbook&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;inventory/hosts&lt;span class="w"&gt; &lt;/span&gt;patch_freebsd.yml&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--limit&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;mail.example.com:mail_jails&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--check&lt;span class="w"&gt; &lt;/span&gt;--diff
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In &lt;span class="caps"&gt;AAP&lt;/span&gt;, the same pattern goes into the job template&amp;#8217;s limit&amp;nbsp;prompt.&lt;/p&gt;
&lt;p&gt;The &lt;a href="https://hofstede.it/my-ansible-plugin-had-a-jail-escape-cve-2026-55074/"&gt;&lt;span class="caps"&gt;CVE&lt;/span&gt; I wrote about two weeks ago&lt;/a&gt; is why this is safe to do at all: since 2.0.0, every file transfer is resolved inside the jail, and jailexec itself only needs&amp;nbsp;one &lt;code&gt;doas&lt;/code&gt; rule on the host,&amp;nbsp;for &lt;code&gt;jexec&lt;/code&gt;. In this setup the service user has&amp;nbsp;full &lt;code&gt;doas&lt;/code&gt; anyway, because the host play patches the host, but a jail-only setup can stay at that single&amp;nbsp;rule.&lt;/p&gt;
&lt;h3 id="how-a-connection-plugin-ends-up-in-a-container"&gt;How a Connection Plugin Ends Up in a&amp;nbsp;Container&lt;/h3&gt;
&lt;p&gt;When I first ran this in &lt;span class="caps"&gt;AAP&lt;/span&gt;, I wondered how jailexec could work at all: the job runs inside an execution environment container that has never heard of my&amp;nbsp;plugin.&lt;/p&gt;
&lt;p&gt;The answer is that Ansible loads plugins from&amp;nbsp;a &lt;code&gt;connection_plugins/&lt;/code&gt; directory next to the playbook. &lt;span class="caps"&gt;AAP&lt;/span&gt; mounts the project checkout into the container, so a copy&amp;nbsp;of &lt;code&gt;jailexec.py&lt;/code&gt; in the playbook repository is enough. That worked on the first try, but a vendored copy is exactly what my &lt;span class="caps"&gt;CVE&lt;/span&gt; article warned about: a file that never updates itself and that none of my dependency scanning would ever&amp;nbsp;flag.&lt;/p&gt;
&lt;p&gt;So jailexec is now also an Ansible&amp;nbsp;collection, &lt;code&gt;chofstede.jailexec&lt;/code&gt;, built from the same single source file that the PyPI package uses. On my side it lives in my Private Automation Hub, and the playbook repository just lists&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# collections/requirements.yml&lt;/span&gt;
&lt;span class="nt"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;community.general&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;=13.0.0,&amp;lt;14.0.0&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;chofstede.jailexec&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;=2.0.3,&amp;lt;3.0.0&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The connection&amp;nbsp;becomes &lt;code&gt;chofstede.jailexec.jailexec&lt;/code&gt;, and there is exactly one place to release it. jailexec is already on &lt;a href="https://github.com/chofstede/ansible_jailexec"&gt;GitHub&lt;/a&gt;, &lt;a href="https://codeberg.org/Larvitz/ansible_jailexec"&gt;Codeberg&lt;/a&gt; and &lt;a href="https://pypi.org/project/ansible-jailexec/"&gt;PyPI&lt;/a&gt;. &lt;strong&gt;The collection is coming to Ansible Galaxy&amp;nbsp;soon.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id="defensive-practices"&gt;Defensive&amp;nbsp;Practices&lt;/h2&gt;
&lt;p&gt;Most of the work in these playbooks isn&amp;#8217;t updating packages. It&amp;#8217;s reducing the ways updating packages can hurt, and noticing quickly when it does&amp;nbsp;anyway.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Dry run first, one host next, the rest last.&lt;/strong&gt; Every change started&amp;nbsp;with &lt;code&gt;--check --diff&lt;/code&gt; against everything, then a real run against one unimportant host (a jail I only use to read mail), then the rest. The dry run caught&amp;nbsp;the &lt;code&gt;doas.conf&lt;/code&gt; permission change before it happened. Read-only tasks&amp;nbsp;carry &lt;code&gt;check_mode: false&lt;/code&gt;, so the check run actually knows&amp;nbsp;something.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Know which hosts must never reboot unattended.&lt;/strong&gt; My mail server&amp;#8217;s jails live on an encrypted dataset that I unlock by hand after every boot. I knew that. I still launched a patch run with a kernel update against it, and the playbook did exactly what it was told: it rebooted the host, waited for the jails, and reported success, because I happened to unlock the dataset within its five-minute window. The fix is one line of&amp;nbsp;inventory:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# host_vars/mail.example.com.yml&lt;/span&gt;
&lt;span class="c1"&gt;# Never reboot automatically: the jail dataset needs a manual &amp;quot;zfs load-key&amp;quot; after boot&lt;/span&gt;
&lt;span class="nt"&gt;fbsd_update_reboot&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;never&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The deeper fix is to ask &amp;#8220;which host reboots?&amp;#8221; before every run, and to say the answer out loud. I&amp;#8217;m also thinking about a pre-reboot check that refuses on its own when a &lt;span class="caps"&gt;ZFS&lt;/span&gt; dataset&amp;nbsp;has &lt;code&gt;keylocation=prompt&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Don&amp;#8217;t let the automation patch its own floor.&lt;/strong&gt; The controller runs on the Proxmox cluster. Proxmox nodes are therefore never rebooted by&amp;nbsp;automation, &lt;code&gt;apt&lt;/code&gt; runs asynchronously, and the workflow patches them&amp;nbsp;last.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Constrain collections to a curated source.&lt;/strong&gt; &lt;span class="caps"&gt;AAP&lt;/span&gt; in my setup pulls collections only from my Private Automation Hub. Public Galaxy is not configured at all. A new collection version reaches production when I approve it in the hub, not when someone publishes it. The requirements file only constrains the major version&amp;nbsp;(&lt;code&gt;&amp;gt;=13.0.0,&amp;lt;14.0.0&lt;/code&gt;). That is a compatibility guard, not a lock: the approval step in the hub decides which versions are eligible at all. Exact reproducibility would take pinned versions or a fixed execution environment. A missing collection fails the project sync instead of quietly coming from the internet. On my&amp;nbsp;workstation, &lt;code&gt;ansible-galaxy&lt;/code&gt; uses the same hub through environment variables (a&amp;nbsp;project&amp;#8217;s &lt;code&gt;ansible.cfg&lt;/code&gt; would&amp;nbsp;shadow &lt;code&gt;~/.ansible.cfg&lt;/code&gt; completely, because Ansible never merges config&amp;nbsp;files).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Check &lt;code&gt;requires_ansible&lt;/code&gt; before you blame the&amp;nbsp;plugin.&lt;/strong&gt; &lt;code&gt;community.general&lt;/code&gt; 13 requires ansible-core 2.18 or later. The default execution environment in my &lt;span class="caps"&gt;AAP&lt;/span&gt; still ships 2.16. Ansible only warns about that and runs anyway, which is a fine way to get mysterious failures in&amp;nbsp;the &lt;code&gt;doas&lt;/code&gt; become plugin months later. The patch job templates use an &lt;span class="caps"&gt;EE&lt;/span&gt; with ansible-core 2.20, and the requirements file stays on the 13.x&amp;nbsp;major.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Ad-hoc commands are not a test.&lt;/strong&gt; An &lt;span class="caps"&gt;AAP&lt;/span&gt;&amp;nbsp;ad-hoc &lt;code&gt;ping&lt;/code&gt; failed against every FreeBSD host and every jail, because ad-hoc commands run without the project and therefore without its collections and plugins. A job template launched in check mode against two hosts is the real smoke&amp;nbsp;test.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Inventory and playbooks are code.&lt;/strong&gt; Both repositories get reviewed changes and signed commits only. I pushed unsigned commits once during this project and was reminded, firmly, by myself, that this is not how my repositories&amp;nbsp;work. &lt;code&gt;git commit --amend -S&lt;/code&gt; and &lt;code&gt;--force-with-lease&lt;/code&gt; fixed it before anyone else&amp;nbsp;noticed.&lt;/p&gt;
&lt;h2 id="what-changed"&gt;What&amp;nbsp;Changed&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Before&lt;/th&gt;
&lt;th&gt;After&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;A weekly &lt;span class="caps"&gt;SSH&lt;/span&gt; ritual per host&lt;/td&gt;
&lt;td&gt;One workflow, launched when I choose&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Patch knowledge in my head&lt;/td&gt;
&lt;td&gt;Two Git repositories, reviewed and signed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;freebsd-update&lt;/code&gt; copy-pasted&lt;/td&gt;
&lt;td&gt;fetch, &lt;code&gt;updatesready&lt;/code&gt;, install, verify, with check mode&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Jails patched when I remembered&lt;/td&gt;
&lt;td&gt;Thin jails via release + restart, thick jails via jailexec&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reboots whenever it felt right&lt;/td&gt;
&lt;td&gt;Explicit per&amp;nbsp;host: &lt;code&gt;auto&lt;/code&gt;, &lt;code&gt;always&lt;/code&gt; or &lt;code&gt;never&lt;/code&gt;; Proxmox only reports&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Collections from wherever&lt;/td&gt;
&lt;td&gt;A curated hub, constrained versions&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The first real runs updated the mail server and its jails, the pkgbase host with all nine Bastille jails, and confirmed that the Mastodon host and its jails were already current. The routers and the Proxmox cluster are next, one at a time. Their playbooks are written and reviewed, but haven&amp;#8217;t touched a production router or hypervisor&amp;nbsp;yet.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;None of the individual pieces in this article is&amp;nbsp;clever. &lt;code&gt;freebsd-update&lt;/code&gt; has&amp;nbsp;had &lt;code&gt;updatesready&lt;/code&gt; for years, Bastille has&amp;nbsp;had &lt;code&gt;bastille update&lt;/code&gt; since forever,&amp;nbsp;and &lt;code&gt;dist-upgrade&lt;/code&gt; is in every Proxmox upgrade guide. The value is in writing all of it down once, in the right order, with the right exceptions, and in a form that a dry run can&amp;nbsp;check.&lt;/p&gt;
&lt;p&gt;If you also have a fleet that is &amp;#8220;mostly automated, except for the weird ones&amp;#8221;, start with the inventory. Once every host&amp;#8217;s groups describe how it must be treated, not just where it lives, the playbooks get much easier to write. What remains are the exceptions, and those are exactly what is worth writing down. And before the first real run, find out which of your hosts cannot boot on its own. Ask me how I&amp;nbsp;know.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;Managing FreeBSD Jails with Ansible: The jailexec Connection&amp;nbsp;Plugin&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/chofstede/ansible_jailexec"&gt;jailexec on GitHub&lt;/a&gt; · &lt;a href="https://codeberg.org/Larvitz/ansible_jailexec"&gt;Codeberg&lt;/a&gt; · &lt;a href="https://pypi.org/project/ansible-jailexec/"&gt;PyPI&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/upgrading-freebsd-150-release-to-151-release-the-official-paths/"&gt;Upgrading FreeBSD 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; to 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;: The Official&amp;nbsp;Paths&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/hetzner-proxmox-cluster/"&gt;Two Sites, One Cluster: My Hetzner Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt;&amp;nbsp;Setup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/reproducible-ansible-with-execution-environments/"&gt;Reproducible Ansible with Execution&amp;nbsp;Environments&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=freebsd-update&amp;amp;sektion=8"&gt;freebsd-update(8)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastille.readthedocs.io/"&gt;Bastille&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pve.proxmox.com/wiki/System_Software_Updates"&gt;Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt;: System Software&amp;nbsp;Updates&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="ansible"/><category term="freebsd"/><category term="jails"/><category term="bastille"/><category term="pkgbase"/><category term="proxmox"/><category term="debian"/><category term="rhel"/><category term="patching"/><category term="automation"/><category term="aap"/><category term="jailexec"/></entry><entry><title>Claude Code on a FreeBSD Desktop: Plasma, xrdp, the Linuxulator and One Very Confusing PATH</title><link href="https://hofstede.it/claude-code-on-a-freebsd-desktop-plasma-xrdp-the-linuxulator-and-one-very-confusing-path/" rel="alternate"/><published>2026-10-07T00:00:00+02:00</published><updated>2026-10-07T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-10-07:/claude-code-on-a-freebsd-desktop-plasma-xrdp-the-linuxulator-and-one-very-confusing-path/</id><summary type="html">&lt;p&gt;A FreeBSD 15.1 Plasma desktop in a Proxmox &lt;span class="caps"&gt;VM&lt;/span&gt;, reachable over xrdp, running Claude Code as a Linux binary through the Linuxulator. Along the way: an Xorg that found no screens, a &lt;span class="caps"&gt;KRDP&lt;/span&gt; package that does not exist, a VirtIO-&lt;span class="caps"&gt;GPU&lt;/span&gt; that gives you a framebuffer but no acceleration, and a &amp;#8220;version regression&amp;#8221; that turned out to be two Claude installations fighting over my &lt;span class="caps"&gt;PATH&lt;/span&gt;. Then I let the agent firewall the machine it was running on, and it did that more carefully than I usually do. It also got one thing wrong, and of course it was the thing about&amp;nbsp;root.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="KDE Plasma 6.7.5 on FreeBSD 15.1 over xrdp: Claude Code in Konsole describing the FreeBSD host under its Rocky Linux userland, fastfetch, and System Settings reporting llvmpipe" src="https://hofstede.it/images/2026-10-07-freebsd-plasma-claude-code-linuxulator.webp" title="A Linux agent in a FreeBSD machine in a Linux hypervisor."&gt;&lt;/p&gt;
&lt;p&gt;The plan was boring on purpose: a FreeBSD desktop &lt;span class="caps"&gt;VM&lt;/span&gt; on my Proxmox cluster, &lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma, reachable remotely, so I have a FreeBSD workstation I can open from any machine. Something to live in, not something to&amp;nbsp;admire.&lt;/p&gt;
&lt;p&gt;Then I wanted Claude Code on it. Claude Code ships no FreeBSD binary. It does ship a Linux binary, and FreeBSD has been running Linux binaries through the Linuxulator for decades. I had already done that for a &lt;a href="https://hofstede.it/running-a-factorio-headless-server-on-freebsd-with-the-linuxulator/"&gt;Factorio server&lt;/a&gt;, so how hard could an &lt;span class="caps"&gt;AI&lt;/span&gt; coding agent&amp;nbsp;be?&lt;/p&gt;
&lt;p&gt;Not very hard, as it turned out. But the road there had four detours, each one teaching something more general than &amp;#8220;how to install X on Y&amp;#8221;. And the ending was better than the plan: once the agent ran stably, I asked it to lock down the network of the very machine it was running on, over the very &lt;span class="caps"&gt;SSH&lt;/span&gt; session it could have cut. It handled that with a dead-man&amp;#8217;s switch and asked me to verify from a fresh connection before committing&amp;nbsp;anything.&lt;/p&gt;
&lt;p&gt;Here is the final stack, from the metal&amp;nbsp;up:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Proxmox VE (QEMU/KVM)
  └── FreeBSD 15.1-RELEASE-p4 (ZFS root, VirtIO disk/NIC/GPU)
        ├── SDDM + Xorg (scfb on the VirtIO framebuffer)   ← local console
        ├── xrdp + xorgxrdp → Xorg :10 → Plasma 6 (X11)    ← how I actually use it
        ├── pf: SSH in, RDP only through an SSH tunnel
        └── Linuxulator (Rocky Linux 9 userland)
              └── Claude Code (Linux ELF, via claude-freebsd wrapper)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-machine"&gt;The&amp;nbsp;Machine&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#no-screens-found"&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;No Screens&amp;nbsp;Found&amp;#8221;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#remote-desktop-xrdp-because-there-is-no-krdp"&gt;Remote Desktop: xrdp, Because There Is No &lt;span class="caps"&gt;KRDP&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#virtio-gpu-a-framebuffer-not-acceleration"&gt;VirtIO-&lt;span class="caps"&gt;GPU&lt;/span&gt;: A Framebuffer, Not Acceleration&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-cloud-on-the-horizon"&gt;The Cloud on the&amp;nbsp;Horizon&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#claude-code-through-the-linuxulator"&gt;Claude Code Through the Linuxulator&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#what-is-actually-running"&gt;What Is Actually&amp;nbsp;Running&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#claude-freebsd"&gt;claude-freebsd&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-mounts-that-matter"&gt;The Mounts That&amp;nbsp;Matter&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-freeze-that-wasnt-a-regression"&gt;The Freeze That Wasn&amp;#8217;t a&amp;nbsp;Regression&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-agent-meets-its-host"&gt;The Agent Meets Its&amp;nbsp;Host&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#letting-the-agent-firewall-the-machine-it-runs-on"&gt;Letting the Agent Firewall the Machine It Runs On&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-policy"&gt;The&amp;nbsp;Policy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-ruleset"&gt;The&amp;nbsp;Ruleset&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-dead-mans-switch"&gt;The Dead-Man&amp;#8217;s&amp;nbsp;Switch&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#verification-and-persistence"&gt;Verification and&amp;nbsp;Persistence&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-the-agent-got-wrong"&gt;What the Agent Got&amp;nbsp;Wrong&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#loose-ends"&gt;Loose&amp;nbsp;Ends&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#was-it-worth-it"&gt;Was It Worth&amp;nbsp;It?&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-machine"&gt;The&amp;nbsp;Machine&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Used here&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hypervisor&lt;/td&gt;
&lt;td&gt;Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt;, &lt;span class="caps"&gt;QEMU&lt;/span&gt;/&lt;span class="caps"&gt;KVM&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Guest&lt;/td&gt;
&lt;td&gt;FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;-p4, amd64, &lt;span class="caps"&gt;GENERIC&lt;/span&gt;, &lt;span class="caps"&gt;UEFI&lt;/span&gt; boot&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Resources&lt;/td&gt;
&lt;td&gt;4 vCPUs, 8 &lt;span class="caps"&gt;GB&lt;/span&gt; &lt;span class="caps"&gt;RAM&lt;/span&gt;, ~84 &lt;span class="caps"&gt;GB&lt;/span&gt; VirtIO disk, &lt;span class="caps"&gt;ZFS&lt;/span&gt; root&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Desktop&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma 6.7.5 on X11, &lt;span class="caps"&gt;SDDM&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Local display&lt;/td&gt;
&lt;td&gt;VirtIO-&lt;span class="caps"&gt;GPU&lt;/span&gt; framebuffer&amp;nbsp;(&lt;code&gt;virtio-gpu-qemu-kmod&lt;/code&gt;)&amp;nbsp;+ &lt;code&gt;xf86-video-scfb&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Remote desktop&lt;/td&gt;
&lt;td&gt;xrdp 0.10.6.1 + xorgxrdp 0.10.5&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rendering&lt;/td&gt;
&lt;td&gt;Mesa llvmpipe.&amp;nbsp;No &lt;code&gt;/dev/dri&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Linux compat&lt;/td&gt;
&lt;td&gt;Linuxulator&amp;nbsp;with &lt;code&gt;linux_base-rl9&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Agent&lt;/td&gt;
&lt;td&gt;Claude Code 2.1.281 via &lt;a href="https://github.com/insanityinside/claude-freebsd"&gt;insanityinside/claude-freebsd&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Firewall&lt;/td&gt;
&lt;td&gt;pf + pflog&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The package repository&amp;nbsp;is &lt;code&gt;latest&lt;/code&gt;,&amp;nbsp;not &lt;code&gt;quarterly&lt;/code&gt;. On a desktop I want current &lt;span class="caps"&gt;KDE&lt;/span&gt;, and I accept the occasional breakage that comes with&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="no-screens-found"&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;No Screens&amp;nbsp;Found&amp;#8221;&lt;/h2&gt;
&lt;p&gt;I installed FreeBSD, added &lt;span class="caps"&gt;KDE&lt;/span&gt;, &lt;span class="caps"&gt;SDDM&lt;/span&gt; and DBus the way the Handbook describes, enabled the services, rebooted. &lt;span class="caps"&gt;SDDM&lt;/span&gt; started. Its X server died before it could report a display number, and &lt;span class="caps"&gt;SDDM&lt;/span&gt; sat there looking at&amp;nbsp;nothing.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;SDDM&lt;/span&gt;&amp;#8217;s own log was useless. Running Xorg by hand was&amp;nbsp;not:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;(EE) open /dev/dri/card0: No such file or directory
(EE) Failed to load module &amp;quot;scfb&amp;quot; (module does not exist, 0)
(EE) Failed to load module &amp;quot;vesa&amp;quot; (module does not exist, 0)
(EE) no screens found
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;#8217;s three failures in a row, and each one is informative. At this point the &lt;span class="caps"&gt;VM&lt;/span&gt; had Proxmox&amp;#8217;s default display, &lt;span class="caps"&gt;QEMU&lt;/span&gt; Standard &lt;span class="caps"&gt;VGA&lt;/span&gt; (&lt;span class="caps"&gt;PCI&lt;/span&gt; &lt;span class="caps"&gt;ID&lt;/span&gt; &lt;code&gt;1234:1111&lt;/code&gt;). Xorg&amp;nbsp;auto-detects &lt;code&gt;modesetting&lt;/code&gt; first, which needs a &lt;span class="caps"&gt;DRM&lt;/span&gt; device. There is none. It then falls back&amp;nbsp;to &lt;code&gt;scfb&lt;/code&gt; and &lt;code&gt;vesa&lt;/code&gt;, neither of which was installed. Out of drivers, out of&amp;nbsp;screens.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The Handbook instructions were not wrong.&lt;/strong&gt; Plasma and &lt;span class="caps"&gt;SDDM&lt;/span&gt; were configured fine. They just assume you already have a working X display driver, and in a &lt;span class="caps"&gt;VM&lt;/span&gt; without &lt;span class="caps"&gt;DRM&lt;/span&gt; you&amp;nbsp;don&amp;#8217;t.&lt;/p&gt;
&lt;p&gt;The fix depends on how the &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;nbsp;boots. &lt;code&gt;scfb&lt;/code&gt; drives the system console framebuffer, which on &lt;span class="caps"&gt;UEFI&lt;/span&gt; is the &lt;span class="caps"&gt;EFI&lt;/span&gt; &lt;span class="caps"&gt;GOP&lt;/span&gt;&amp;nbsp;framebuffer:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;machdep.bootmethod
&lt;span class="c1"&gt;# machdep.bootmethod: UEFI&lt;/span&gt;

pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;xf86-video-scfb
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That was all. Xorg came up, &lt;span class="caps"&gt;SDDM&lt;/span&gt; showed a login screen, Plasma started. If your &lt;span class="caps"&gt;VM&lt;/span&gt; boots via legacy &lt;span class="caps"&gt;BIOS&lt;/span&gt;, &lt;code&gt;scfb&lt;/code&gt; won&amp;#8217;t help you&amp;nbsp;and &lt;code&gt;xf86-video-vesa&lt;/code&gt; is the one to&amp;nbsp;try.&lt;/p&gt;
&lt;h2 id="remote-desktop-xrdp-because-there-is-no-krdp"&gt;Remote Desktop: xrdp, Because There Is No &lt;span class="caps"&gt;KRDP&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Plasma 6 has a native &lt;span class="caps"&gt;RDP&lt;/span&gt; server, &lt;span class="caps"&gt;KRDP&lt;/span&gt;, configurable straight from System Settings. That was my plan for remote&amp;nbsp;access.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;search&lt;span class="w"&gt; &lt;/span&gt;-q&lt;span class="w"&gt; &lt;/span&gt;krdp
&lt;span class="c1"&gt;# (nothing)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Not in the FreeBSD repositories, neither when I set this up nor today. I am not claiming &lt;span class="caps"&gt;KRDP&lt;/span&gt; can&amp;#8217;t work on FreeBSD. It&amp;#8217;s just not packaged, and I wasn&amp;#8217;t in the mood to port it. The search did turn up xrdp, FreeRDP, Remmina and Guacamole. xrdp it&amp;nbsp;is.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;xrdp&lt;span class="w"&gt; &lt;/span&gt;xorgxrdp
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;xrdp_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;xrdp_sesman_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;xrdp doesn&amp;#8217;t share the local display. It starts &lt;strong&gt;its own&lt;/strong&gt; Xorg server per session (with&amp;nbsp;the &lt;code&gt;xorgxrdp&lt;/code&gt; driver), on&amp;nbsp;display &lt;code&gt;:10&lt;/code&gt; and up. What runs in that session is decided&amp;nbsp;by &lt;code&gt;/usr/local/etc/xrdp/startwm.sh&lt;/code&gt;, which ships with Plasma commented out and&amp;nbsp;an &lt;code&gt;xterm&lt;/code&gt; fallback. One uncommented line&amp;nbsp;later:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ck-launch-session&lt;span class="w"&gt; &lt;/span&gt;startplasma-x11
&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;xterm
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;exec xterm&lt;/code&gt; line is effectively dead code&amp;nbsp;now. &lt;code&gt;exec&lt;/code&gt; replaces the shell with the Plasma session, so nothing after it ever runs, and if Plasma dies later, the &lt;span class="caps"&gt;RDP&lt;/span&gt; session simply ends. If you need an xterm for debugging, comment out the Plasma line&amp;nbsp;instead.&lt;/p&gt;
&lt;p&gt;Then start&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;xrdp&lt;span class="w"&gt; &lt;/span&gt;allstart
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note&amp;nbsp;the &lt;code&gt;allstart&lt;/code&gt;. There&amp;#8217;s&amp;nbsp;an &lt;code&gt;xrdp-sesman&lt;/code&gt; process, and&amp;nbsp;an &lt;code&gt;xrdp_sesman_enable&lt;/code&gt; knob, but no&amp;nbsp;separate &lt;code&gt;service xrdp_sesman&lt;/code&gt; command. The xrdp rc.d script manages both,&amp;nbsp;and &lt;code&gt;allstart&lt;/code&gt; is how you tell it to (if that sentence made you want to know more about rc.d, the &lt;a href="https://hofstede.it/freebsd-foundationals-rcd-the-service-framework-that-fits-in-your-head/"&gt;rc.d article&lt;/a&gt; went up three days&amp;nbsp;ago).&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;KRDC&lt;/span&gt; on my laptop connected on the first try and showed a full Plasma desktop. Independent of the local &lt;span class="caps"&gt;SDDM&lt;/span&gt; session, so I can leave the console at the login&amp;nbsp;screen.&lt;/p&gt;
&lt;p&gt;It was also noticeably&amp;nbsp;laggy.&lt;/p&gt;
&lt;h2 id="virtio-gpu-a-framebuffer-not-acceleration"&gt;VirtIO-&lt;span class="caps"&gt;GPU&lt;/span&gt;: A Framebuffer, Not&amp;nbsp;Acceleration&lt;/h2&gt;
&lt;p&gt;Plasma&amp;#8217;s About page answered the &amp;#8220;why is it slow&amp;#8221; question immediately: &lt;strong&gt;Graphics Processor: llvmpipe&lt;/strong&gt;. Every pixel rendered in software on the &lt;span class="caps"&gt;CPU&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The obvious next step was moving off &lt;span class="caps"&gt;QEMU&lt;/span&gt; Standard &lt;span class="caps"&gt;VGA&lt;/span&gt;. I switched the Proxmox display device to VirtIO-&lt;span class="caps"&gt;GPU&lt;/span&gt; and installed the matching FreeBSD&amp;nbsp;driver:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;virtio-gpu-qemu-kmod
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;virtio_gpu_qemu_load=&amp;quot;YES&amp;quot;&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The package insists on being loaded from the loader, not&amp;nbsp;with &lt;code&gt;kldload&lt;/code&gt; later, because it has to take over the console&amp;nbsp;before &lt;code&gt;efifb&lt;/code&gt; gets comfortable. After a&amp;nbsp;reboot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;virtio_pci0: &amp;lt;VirtIO PCI (modern) GPU adapter&amp;gt; ...
vtgpu0: &amp;lt;VirtIO GPU (QEMU/UTM)&amp;gt; on virtio_pci0
VT: Replacing driver &amp;quot;efifb&amp;quot; with new &amp;quot;virtio_gpu_qemu&amp;quot;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Works. The console and the local X server now run on a proper VirtIO framebuffer,&amp;nbsp;and &lt;code&gt;scfb&lt;/code&gt; picks it up without&amp;nbsp;complaint.&lt;/p&gt;
&lt;p&gt;And&amp;nbsp;then:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ls&lt;span class="w"&gt; &lt;/span&gt;/dev/dri
&lt;span class="c1"&gt;# ls: /dev/dri: No such file or directory&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Still&amp;nbsp;nothing. &lt;code&gt;glxinfo&lt;/code&gt; and Plasma both still say&amp;nbsp;llvmpipe.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;VirtIO-&lt;span class="caps"&gt;GPU&lt;/span&gt; works&amp;#8221; means the framebuffer works. It does not mean &lt;span class="caps"&gt;DRM&lt;/span&gt;/&lt;span class="caps"&gt;KMS&lt;/span&gt;, virgl, or hardware-accelerated OpenGL.&lt;/strong&gt; This driver gives FreeBSD a better place to put pixels. It doesn&amp;#8217;t give Mesa a &lt;span class="caps"&gt;GPU&lt;/span&gt; to render them with. The xrdp session is a separate Xorg instance anyway, so it was never going to benefit directly. The lag&amp;nbsp;stayed.&lt;/p&gt;
&lt;p&gt;Is it usable? For terminals, a browser, Kate and Dolphin: yes. For anything that animates a lot: you&amp;#8217;ll notice. I turned off most of the desktop effects and stopped thinking about&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="the-cloud-on-the-horizon"&gt;The Cloud on the&amp;nbsp;Horizon&lt;/h3&gt;
&lt;p&gt;There&amp;#8217;s a bigger issue that I can&amp;#8217;t solve today: &lt;strong&gt;Plasma 6.8 drops the X11 session.&lt;/strong&gt; This entire remote desktop path depends&amp;nbsp;on &lt;code&gt;startplasma-x11&lt;/code&gt; running inside an xrdp-managed Xorg. xorgxrdp will still happily start an Xorg server, but there will be no Plasma X11 session left to run inside it, and &lt;span class="caps"&gt;KRDP&lt;/span&gt; (the Wayland-native answer) isn&amp;#8217;t packaged for&amp;nbsp;FreeBSD.&lt;/p&gt;
&lt;p&gt;So, to be clear about the status of this setup: it&amp;#8217;s what works &lt;strong&gt;today&lt;/strong&gt;, with Plasma 6.7. It isn&amp;#8217;t a future-proof recommendation. I&amp;#8217;ll revisit it when 6.8 lands&amp;nbsp;in &lt;code&gt;latest&lt;/code&gt;, and I suspect I&amp;#8217;ll be pinning packages for a&amp;nbsp;while.&lt;/p&gt;
&lt;h2 id="claude-code-through-the-linuxulator"&gt;Claude Code Through the&amp;nbsp;Linuxulator&lt;/h2&gt;
&lt;p&gt;Now the fun&amp;nbsp;part.&lt;/p&gt;
&lt;h3 id="what-is-actually-running"&gt;What Is Actually&amp;nbsp;Running&lt;/h3&gt;
&lt;p&gt;Some precision first, because it&amp;#8217;s easy to get this wrong in a headline. &lt;strong&gt;Claude Code isn&amp;#8217;t running &amp;#8220;natively&amp;#8221; on FreeBSD.&lt;/strong&gt; It&amp;#8217;s an unmodified Linux &lt;span class="caps"&gt;ELF&lt;/span&gt; binary. The FreeBSD kernel recognizes the Linux &lt;span class="caps"&gt;ELF&lt;/span&gt; brand and translates its system calls through the Linuxulator. There&amp;#8217;s no Linux kernel anywhere: no &lt;span class="caps"&gt;VM&lt;/span&gt;, no container, no emulation in the &lt;span class="caps"&gt;QEMU&lt;/span&gt; sense. But the application binary is built for Linux, links against glibc, and expects a Linux-shaped filesystem around&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;That Linux-shaped filesystem lives&amp;nbsp;under &lt;code&gt;/compat/linux&lt;/code&gt;, provided&amp;nbsp;by &lt;code&gt;linux_base-rl9&lt;/code&gt;, a Rocky Linux 9 userland. Linux processes see it as their root. Native FreeBSD processes see the real system. Two views of the same&amp;nbsp;machine:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ /compat/linux/bin/uname -a
Linux bsdesktop 5.15.0 FreeBSD 15.1-RELEASE-p4 releng/15.1-n283627-db6066678b33 GENERIC x86_64 x86_64 x86_64 GNU/Linux

$ cat /compat/linux/etc/os-release
NAME=&amp;quot;Rocky Linux&amp;quot;
VERSION=&amp;quot;9.7 (Blue Onyx)&amp;quot;

$ uname -a
FreeBSD bsdesktop 15.1-RELEASE-p4 FreeBSD 15.1-RELEASE-p4 releng/15.1-n283627-db6066678b33 GENERIC amd64
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &amp;#8220;Linux 5.15.0&amp;#8221; comes&amp;nbsp;from &lt;code&gt;compat.linux.osrelease&lt;/code&gt;, a sysctl. It&amp;#8217;s whatever kernel version the Linuxulator claims to be, so glibc and friends are satisfied. (Small bonus confusion: the package&amp;nbsp;is &lt;code&gt;linux_base-rl9-9.8&lt;/code&gt;, the release file inside it still says 9.7. I didn&amp;#8217;t chase that&amp;nbsp;one.)&lt;/p&gt;
&lt;p&gt;Keep this split in mind. It becomes important later, because the agent sees the Linux view&amp;nbsp;first.&lt;/p&gt;
&lt;h3 id="claude-freebsd"&gt;claude-freebsd&lt;/h3&gt;
&lt;p&gt;Since version 2.1.113, Claude Code is a compiled binary&amp;nbsp;(&lt;code&gt;bun build --compile&lt;/code&gt;) per platform, and FreeBSD isn&amp;#8217;t a platform. There&amp;#8217;s an &lt;a href="https://github.com/anthropics/claude-code/issues/61313"&gt;open issue&lt;/a&gt; asking for a native build. Until that happens, &lt;a href="https://github.com/insanityinside/claude-freebsd"&gt;insanityinside/claude-freebsd&lt;/a&gt; fills the gap, and it describes itself, honestly, as a&amp;nbsp;stopgap.&lt;/p&gt;
&lt;p&gt;It does exactly what you would otherwise do by hand, plus the things you would&amp;nbsp;forget:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Checks the prerequisites: Linuxulator loaded, glibc present, the required mounts&amp;nbsp;in &lt;code&gt;/etc/fstab&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Fetches&amp;nbsp;the &lt;code&gt;linux-x64&lt;/code&gt; binary from Anthropic&amp;#8217;s download&amp;nbsp;server.&lt;/li&gt;
&lt;li&gt;Verifies its &lt;span class="caps"&gt;SHA256&lt;/span&gt; against Anthropic&amp;#8217;s signed release&amp;nbsp;manifest.&lt;/li&gt;
&lt;li&gt;Installs the binary&amp;nbsp;to &lt;code&gt;/usr/local/libexec/claude-code/claude&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Installs a shell wrapper&amp;nbsp;as &lt;code&gt;/usr/local/bin/claude&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Installs itself&amp;nbsp;as &lt;code&gt;/usr/local/bin/claude-freebsd&lt;/code&gt; for later&amp;nbsp;updates.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The Linuxulator itself has to be on&amp;nbsp;first:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;linux_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
service&lt;span class="w"&gt; &lt;/span&gt;linux&lt;span class="w"&gt; &lt;/span&gt;start
pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;linux_base-rl9
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then the&amp;nbsp;install:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;fetch&lt;span class="w"&gt; &lt;/span&gt;https://raw.githubusercontent.com/insanityinside/claude-freebsd/main/claude-freebsd.sh
chmod&lt;span class="w"&gt; &lt;/span&gt;+x&lt;span class="w"&gt; &lt;/span&gt;claude-freebsd.sh
doas&lt;span class="w"&gt; &lt;/span&gt;./claude-freebsd.sh&lt;span class="w"&gt; &lt;/span&gt;--install
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(The &lt;span class="caps"&gt;README&lt;/span&gt;&amp;nbsp;says &lt;code&gt;sudo&lt;/code&gt;. This box&amp;nbsp;has &lt;code&gt;doas&lt;/code&gt; and&amp;nbsp;no &lt;code&gt;sudo&lt;/code&gt;. More&amp;nbsp;on &lt;code&gt;doas&lt;/code&gt; later, because that&amp;#8217;s where the story has a&amp;nbsp;twist.)&lt;/p&gt;
&lt;p&gt;The wrapper is a plain &lt;span class="caps"&gt;POSIX&lt;/span&gt; shell script. Read it, it&amp;#8217;s short and well commented. Its main&amp;nbsp;jobs:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Disable Claude Code&amp;#8217;s own self-updater&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;DISABLE_AUTOUPDATER=1&lt;/code&gt;). The binary lives in a root-owned path, and a normal user&amp;#8217;s self-update would either fail or, worse, install a second copy somewhere&amp;nbsp;in &lt;code&gt;$HOME&lt;/code&gt;. Updates go&amp;nbsp;through &lt;code&gt;claude-freebsd --update&lt;/code&gt; instead. Remember this point. It&amp;#8217;s the whole plot of the next&amp;nbsp;section.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Check the Linuxulator mounts&lt;/strong&gt; on every launch and print a warning if one is&amp;nbsp;missing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Satisfy Claude&amp;#8217;s own install self-checks&lt;/strong&gt;, which assume a &amp;#8220;native&amp;#8221; Linux install&amp;nbsp;in &lt;code&gt;~/.local/bin&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Print a once-per-day &amp;#8220;update available&amp;#8221;&amp;nbsp;notice.&lt;/strong&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Pinning a version is one&amp;nbsp;flag:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;doas&lt;span class="w"&gt; &lt;/span&gt;claude-freebsd&lt;span class="w"&gt; &lt;/span&gt;--update&lt;span class="w"&gt; &lt;/span&gt;--version&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;.1.281
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="the-mounts-that-matter"&gt;The Mounts That&amp;nbsp;Matter&lt;/h3&gt;
&lt;p&gt;These go&amp;nbsp;into &lt;code&gt;/etc/fstab&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;devfs     /compat/linux/dev      devfs     rw,late
tmpfs     /compat/linux/dev/shm  tmpfs     rw,size=1g,mode=1777,late
fdescfs   /compat/linux/dev/fd   fdescfs   rw,linrdlnk,late
linprocfs /compat/linux/proc     linprocfs rw,late
linsysfs  /compat/linux/sys      linsysfs  rw,late
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The important one&amp;nbsp;is &lt;code&gt;linrdlnk&lt;/code&gt; on &lt;code&gt;fdescfs&lt;/code&gt;. It makes entries&amp;nbsp;in &lt;code&gt;/dev/fd&lt;/code&gt; behave like Linux expects them to: as symlinks you&amp;nbsp;can &lt;code&gt;readlink()&lt;/code&gt;. Without it, Claude Code hangs on startup indefinitely.&amp;nbsp;And &lt;code&gt;mount(8)&lt;/code&gt; doesn&amp;#8217;t display that option, so you can&amp;#8217;t see whether it&amp;#8217;s set. The wrapper works around that by&amp;nbsp;grepping &lt;code&gt;/etc/fstab&lt;/code&gt; on a bare host, and by&amp;nbsp;testing &lt;code&gt;[ -L /compat/linux/dev/fd/2 ]&lt;/code&gt; inside a jail. That&amp;#8217;s a nice bit of&amp;nbsp;craftsmanship.&lt;/p&gt;
&lt;p&gt;These are the requirements of &lt;em&gt;this&lt;/em&gt; wrapper and &lt;em&gt;this&lt;/em&gt; application, not a universal checklist for every Linux program on&amp;nbsp;FreeBSD.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;README&lt;/span&gt; also mentions a nullfs mount of the home directory&amp;nbsp;into &lt;code&gt;/compat/linux/home/&amp;lt;user&amp;gt;&lt;/code&gt;, for the case where you already&amp;nbsp;nullfs-mount &lt;code&gt;/home&lt;/code&gt; there and use per-user &lt;span class="caps"&gt;ZFS&lt;/span&gt; datasets (nullfs isn&amp;#8217;t recursive). I added one to my fstab preemptively. While writing this article I found out that it has never once been&amp;nbsp;mounted: &lt;code&gt;/compat/linux/home&lt;/code&gt; doesn&amp;#8217;t even exist on this box, so&amp;nbsp;the &lt;code&gt;late&lt;/code&gt; mount just fails quietly at boot. Claude works perfectly anyway. A quick look at inode numbers shows why, at least for this box and these&amp;nbsp;paths:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ /compat/linux/bin/stat -c &amp;quot;%n %i&amp;quot; /etc/os-release /home/chofstede
/etc/os-release 202333
/home/chofstede 34

$ stat -f &amp;quot;%N %i&amp;quot; /compat/linux/etc/os-release /etc/os-release /home/chofstede
/compat/linux/etc/os-release 202333
/etc/os-release 12111
/home/chofstede 34
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;When a Linux process&amp;nbsp;opens &lt;code&gt;/etc/os-release&lt;/code&gt;, it gets the Rocky Linux file&amp;nbsp;under &lt;code&gt;/compat/linux&lt;/code&gt;. When it&amp;nbsp;opens &lt;code&gt;/home/chofstede&lt;/code&gt;, there&amp;#8217;s nothing&amp;nbsp;under &lt;code&gt;/compat/linux&lt;/code&gt; to find, and it gets my real home directory. I won&amp;#8217;t generalise that into a rule about Linuxulator path translation, which has more subtleties than one test shows. But in practice, it means: only add that line if you actually have&amp;nbsp;the &lt;code&gt;/home&lt;/code&gt; nullfs it&amp;#8217;s working around. I&amp;#8217;ll be deleting&amp;nbsp;mine.&lt;/p&gt;
&lt;h2 id="the-freeze-that-wasnt-a-regression"&gt;The Freeze That Wasn&amp;#8217;t a&amp;nbsp;Regression&lt;/h2&gt;
&lt;p&gt;First launch: it worked. Claude started, I typed a prompt, it answered. I typed a second prompt and it stopped accepting keyboard input. No crash, no error, just a &lt;span class="caps"&gt;TUI&lt;/span&gt; frozen in&amp;nbsp;place.&lt;/p&gt;
&lt;p&gt;The process was alive, doing&amp;nbsp;nothing:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ ps -o pid,stat,%cpu,wchan,command -p &amp;lt;PID&amp;gt;
  PID STAT %CPU WCHAN  COMMAND
  ...  I+   0.0 futex  /home/chofstede/.local/share/claude/claude-linux-x64
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Zero &lt;span class="caps"&gt;CPU&lt;/span&gt;, sleeping in&amp;nbsp;a &lt;code&gt;futex&lt;/code&gt; wait. On the Linuxulator, futexes are emulated, and a futex-related bug in the compatibility layer is a perfectly plausible explanation for a hang like this. So my working theory was: newer Claude Code release, something in its threading changed, the Linuxulator doesn&amp;#8217;t like it. The obvious experiment: pin an older version through the manager and see if the freeze goes&amp;nbsp;away.&lt;/p&gt;
&lt;p&gt;It didn&amp;#8217;t. Older version installed, same freeze. Older still, same&amp;nbsp;freeze.&lt;/p&gt;
&lt;p&gt;The thing I should have looked at first was right there in&amp;nbsp;the &lt;code&gt;ps&lt;/code&gt; output: &lt;code&gt;/home/chofstede/.local/share/claude/claude-linux-x64&lt;/code&gt;. That is&amp;nbsp;not &lt;code&gt;/usr/local/libexec/claude-code/claude&lt;/code&gt;. &lt;strong&gt;The process I was debugging wasn&amp;#8217;t the binary I was&amp;nbsp;downgrading.&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ ~/.local/bin/claude --version
2.1.292 (Claude Code)

$ /usr/local/bin/claude --version
2.1.281 (Claude Code)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two completely independent installations. An earlier attempt had left a user-local &amp;#8220;native&amp;#8221; install&amp;nbsp;in &lt;code&gt;~/.local/share/claude&lt;/code&gt;, with its launcher&amp;nbsp;in &lt;code&gt;~/.local/bin/claude&lt;/code&gt;, and that&amp;#8217;s what my shell was finding.&amp;nbsp;Every &lt;code&gt;claude-freebsd --update --version ...&lt;/code&gt; dutifully replaced the managed binary&amp;nbsp;in &lt;code&gt;/usr/local/libexec&lt;/code&gt;, which I then never&amp;nbsp;ran.&lt;/p&gt;
&lt;p&gt;There&amp;#8217;s a detail in the wrapper that explains why it didn&amp;#8217;t fix this by itself. To satisfy Claude Code&amp;#8217;s startup self-check, it&amp;nbsp;creates &lt;code&gt;~/.local/bin/claude&lt;/code&gt; as a symlink to the managed binary, but only if nothing exists there&amp;nbsp;yet:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.local/bin/claude&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ln&lt;span class="w"&gt; &lt;/span&gt;-sf&lt;span class="w"&gt; &lt;/span&gt;/usr/local/libexec/claude-code/claude&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;&lt;span class="s2"&gt;/.local/bin/claude&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;#8217;s the polite thing to do. It doesn&amp;#8217;t overwrite files it doesn&amp;#8217;t own. It also means a stale install in that spot survives&amp;nbsp;indefinitely.&lt;/p&gt;
&lt;p&gt;Once I&amp;nbsp;launched &lt;code&gt;/usr/local/bin/claude&lt;/code&gt; explicitly, the freezes were gone. The session that followed ran for many minutes across a lot of tool calls, including everything in the next sections, without a single freeze. The&amp;nbsp;cleanup:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;rm&lt;span class="w"&gt; &lt;/span&gt;~/.local/bin/claude&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# stale launcher; the wrapper recreates it correctly&lt;/span&gt;
rm&lt;span class="w"&gt; &lt;/span&gt;-rf&lt;span class="w"&gt; &lt;/span&gt;~/.local/share/claude&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# the stale 2.1.292 install&lt;/span&gt;
&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;claude&lt;span class="w"&gt;                         &lt;/span&gt;&lt;span class="c1"&gt;# verify: only /usr/local/bin/claude&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now, the honest part, because it would be easy to write a much worse sentence&amp;nbsp;here:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;I can &lt;strong&gt;not&lt;/strong&gt; tell you &amp;#8220;2.1.292 is broken on FreeBSD and 2.1.281 works&amp;#8221;. All I know is that the stray user-local 2.1.292 hung reproducibly, and the managed 2.1.281 didn&amp;#8217;t. Different install method, different launch path, different environment variables (the wrapper sets several). Any of those could be the&amp;nbsp;cause.&lt;/li&gt;
&lt;li&gt;I can &lt;strong&gt;not&lt;/strong&gt; tell you that the older versions I &amp;#8220;tested&amp;#8221; were bad either. I never ran them. That experiment was invalid from the&amp;nbsp;start.&lt;/li&gt;
&lt;li&gt;2.1.281 isn&amp;#8217;t the &amp;#8220;correct FreeBSD version&amp;#8221;. It&amp;#8217;s the version that was running when things&amp;nbsp;worked.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The actual lesson has nothing to do with FreeBSD: &lt;strong&gt;before you debug a version, prove which binary you are running.&lt;/strong&gt; These would have saved me an&amp;nbsp;hour:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;claude
pgrep&lt;span class="w"&gt; &lt;/span&gt;-fl&lt;span class="w"&gt; &lt;/span&gt;claude
ps&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;pid,stat,%cpu,wchan,command&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;PID&amp;gt;
/usr/local/bin/claude&lt;span class="w"&gt; &lt;/span&gt;--version
~/.local/bin/claude&lt;span class="w"&gt; &lt;/span&gt;--version
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I know this. I&amp;#8217;ve told other people this. I still spent an hour downgrading a binary I wasn&amp;#8217;t&amp;nbsp;running.&lt;/p&gt;
&lt;h2 id="the-agent-meets-its-host"&gt;The Agent Meets Its&amp;nbsp;Host&lt;/h2&gt;
&lt;p&gt;With a stable session, I gave Claude a deliberately open task: look around and tell me what this machine&amp;nbsp;is.&lt;/p&gt;
&lt;p&gt;Its first commands saw the Linux view, because that&amp;#8217;s the world a Linux process lives&amp;nbsp;in. &lt;code&gt;uname&lt;/code&gt; said Linux&amp;nbsp;5.15.0, &lt;code&gt;/etc/os-release&lt;/code&gt; said Rocky Linux 9.7. A less careful agent (or a less careful human) would have stopped there and started&amp;nbsp;writing &lt;code&gt;dnf&lt;/code&gt; commands.&lt;/p&gt;
&lt;p&gt;Then:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ ps aux --sort=-%cpu
ps: illegal option -- -
usage: ps [--libxo] [-aCcdeHhjlmrSTuvwXxZ] ...

$ free -h
/bin/bash: line 1: free: command not found
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;--libxo&lt;/code&gt; in a usage message is about as FreeBSD as it gets. Claude noticed the mismatch,&amp;nbsp;ran &lt;code&gt;freebsd-version -ku&lt;/code&gt;,&amp;nbsp;found &lt;code&gt;linux_enable="YES"&lt;/code&gt; in &lt;code&gt;rc.conf&lt;/code&gt;, and concluded, correctly, that it was a Linux binary running on FreeBSD through the Linuxulator, and that the Rocky Linux branding was only the compatibility userland. From that point on it switched to native&amp;nbsp;tools: &lt;code&gt;sysctl&lt;/code&gt;, &lt;code&gt;pciconf&lt;/code&gt;, &lt;code&gt;zpool&lt;/code&gt;, &lt;code&gt;sockstat&lt;/code&gt;, &lt;code&gt;pkg&lt;/code&gt;, &lt;code&gt;pw&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Its inventory was accurate and complete: &lt;span class="caps"&gt;QEMU&lt;/span&gt;/&lt;span class="caps"&gt;KVM&lt;/span&gt; guest, VirtIO disk, &lt;span class="caps"&gt;NIC&lt;/span&gt; and &lt;span class="caps"&gt;GPU&lt;/span&gt;, &lt;span class="caps"&gt;ZFS&lt;/span&gt; root on a single vdev, Plasma on Xorg under&amp;nbsp;xrdp, &lt;code&gt;qemu-guest-agent&lt;/code&gt;, my active &lt;span class="caps"&gt;SSH&lt;/span&gt; session, 856 packages. And one finding that made me slightly&amp;nbsp;uncomfortable:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ pfctl -s info
pfctl: /dev/pf: No such file or directory
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;No firewall at all.&lt;/strong&gt;&amp;nbsp;And &lt;code&gt;xrdp.ini&lt;/code&gt; had &lt;code&gt;port=3389&lt;/code&gt; with no address, which means xrdp listens on every interface. My remote desktop was open to the whole&amp;nbsp;network.&lt;/p&gt;
&lt;p&gt;On my internal network that wasn&amp;#8217;t a catastrophe. But it isn&amp;#8217;t how I want anything to&amp;nbsp;look.&lt;/p&gt;
&lt;h2 id="letting-the-agent-firewall-the-machine-it-runs-on"&gt;Letting the Agent Firewall the Machine It Runs&amp;nbsp;On&lt;/h2&gt;
&lt;p&gt;My instruction, verbatim: &lt;em&gt;&amp;#8220;Only ssh and rdp should be allowed, and &lt;span class="caps"&gt;RDP&lt;/span&gt; only tunneled via &lt;span class="caps"&gt;SSH&lt;/span&gt;. I establish&amp;nbsp;an &lt;code&gt;ssh -L&lt;/code&gt; tunnel and then connect to&amp;nbsp;localhost.&amp;#8221;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Think about the risk profile for a moment. The agent is running on the box I reach over &lt;span class="caps"&gt;SSH&lt;/span&gt;, and the change is being made over that same access path. It&amp;#8217;s about to enable a packet filter on the interface that carries the session. One misordered rule and the agent cuts off its own oxygen, and mine with it. I&amp;#8217;d be opening the Proxmox console to clean&amp;nbsp;up.&lt;/p&gt;
&lt;h3 id="the-policy"&gt;The&amp;nbsp;Policy&lt;/h3&gt;
&lt;p&gt;Claude broke the requirement down into three&amp;nbsp;rules:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;span class="caps"&gt;TCP&lt;/span&gt;/22 reachable from the&amp;nbsp;network.&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;TCP&lt;/span&gt;/3389 &lt;strong&gt;not&lt;/strong&gt; reachable from the network, even though xrdp listens on all&amp;nbsp;interfaces.&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;RDP&lt;/span&gt; still works when it arrives&amp;nbsp;at &lt;code&gt;127.0.0.1:3389&lt;/code&gt;, which is where the far end&amp;nbsp;of &lt;code&gt;ssh -L 3389:localhost:3389&lt;/code&gt; lands.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Rule 3 is solved by one&amp;nbsp;line: &lt;code&gt;set skip on lo0&lt;/code&gt;. The tunnel&amp;#8217;s local leg is sshd connecting&amp;nbsp;to &lt;code&gt;127.0.0.1:3389&lt;/code&gt;, which travels over the loopback interface, and pf doesn&amp;#8217;t filter loopback at all. No matter how strict the rules&amp;nbsp;on &lt;code&gt;vtnet0&lt;/code&gt; are, the tunnel is&amp;nbsp;unaffected.&lt;/p&gt;
&lt;h3 id="the-ruleset"&gt;The&amp;nbsp;Ruleset&lt;/h3&gt;
&lt;p&gt;The&amp;nbsp;final &lt;code&gt;/etc/pf.conf&lt;/code&gt;, after a second round I&amp;#8217;ll get to in a&amp;nbsp;moment:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ssh_bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;

&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;skip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lo0&lt;/span&gt;

&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fragment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;reassemble&lt;/span&gt;

&lt;span class="n"&gt;antispoof&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;

&lt;span class="c1"&gt;# Outbound: unrestricted&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Inbound: reject sources already flagged for SSH abuse&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ssh_bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;

&lt;span class="c1"&gt;# Inbound: allow SSH, rate-limited per source.&lt;/span&gt;
&lt;span class="c1"&gt;# &amp;gt;10 simultaneous connections or &amp;gt;5 new connections/60s from one source&lt;/span&gt;
&lt;span class="c1"&gt;# gets that source added to &amp;lt;ssh_bruteforce&amp;gt; and its existing states killed.&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;ssh_bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Inbound: RDP is explicitly rejected from the network.&lt;/span&gt;
&lt;span class="c1"&gt;# RDP must only be reached via `ssh -L &amp;lt;port&amp;gt;:localhost:3389 ...`, whose local&lt;/span&gt;
&lt;span class="c1"&gt;# leg hits 127.0.0.1 on lo0 (exempted above by `set skip on lo0`).&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3389&lt;/span&gt;

&lt;span class="c1"&gt;# Default deny everything else inbound&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The comments are Claude&amp;#8217;s, and I&amp;#8217;ve left them in because they&amp;#8217;re right. A few design choices worth pointing&amp;nbsp;out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;quick&lt;/code&gt; everywhere&lt;/strong&gt;, so first match wins and the file reads top to bottom. That makes ordering load-bearing: the &lt;span class="caps"&gt;SSH&lt;/span&gt; &lt;code&gt;pass&lt;/code&gt; must come before the&amp;nbsp;catch-all &lt;code&gt;block&lt;/code&gt;, and&amp;nbsp;the &lt;code&gt;&amp;lt;ssh_bruteforce&amp;gt;&lt;/code&gt; block must come before the &lt;span class="caps"&gt;SSH&lt;/span&gt; &lt;code&gt;pass&lt;/code&gt;, or a flagged source would just keep hitting the rate&amp;nbsp;limiter.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The &lt;span class="caps"&gt;RDP&lt;/span&gt; rule is technically redundant.&lt;/strong&gt; The default deny would catch 3389 anyway. It&amp;#8217;s there so &lt;span class="caps"&gt;RDP&lt;/span&gt; probes get their&amp;nbsp;own &lt;code&gt;log&lt;/code&gt; entry&amp;nbsp;in &lt;code&gt;pflog&lt;/code&gt; instead of disappearing into the general deny noise. I like&amp;nbsp;that.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Outbound is wide open.&lt;/strong&gt; That&amp;#8217;s a desktop.&amp;nbsp;Browsing, &lt;code&gt;pkg&lt;/code&gt;, &lt;span class="caps"&gt;NTP&lt;/span&gt; and &lt;span class="caps"&gt;DNS&lt;/span&gt; are the&amp;nbsp;point.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No &lt;code&gt;inet&lt;/code&gt;/&lt;code&gt;inet6&lt;/code&gt; split.&lt;/strong&gt; The box is dual-stacked, and rules without an address family apply to both. One ruleset, both protocols, no forgotten v6&amp;nbsp;hole.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No &lt;span class="caps"&gt;ICMP&lt;/span&gt;.&lt;/strong&gt; That&amp;#8217;s a deliberate omission, since I asked for &amp;#8220;only &lt;span class="caps"&gt;SSH&lt;/span&gt; and &lt;span class="caps"&gt;RDP&lt;/span&gt;&amp;#8221;. It does mean ping from outside fails. The &lt;a href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/"&gt;pf guide&lt;/a&gt; has the rate-limited &lt;span class="caps"&gt;ICMP&lt;/span&gt; rules if you want them&amp;nbsp;back.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The first round was the same file without the table and the rate limiting. Once &lt;span class="caps"&gt;SSH&lt;/span&gt; was the &lt;em&gt;only&lt;/em&gt; door, it made sense to put a basic brute-force brake on it. That&amp;#8217;s not an intrusion prevention system. It&amp;#8217;s a per-source connection counter that throws an address into a table when it gets&amp;nbsp;greedy.&lt;/p&gt;
&lt;h3 id="the-dead-mans-switch"&gt;The Dead-Man&amp;#8217;s&amp;nbsp;Switch&lt;/h3&gt;
&lt;p&gt;This is the part I&amp;#8217;d steal for my own runbooks. Before touching the live ruleset, Claude armed&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;doas&lt;span class="w"&gt; &lt;/span&gt;sh&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;nohup sh -c &amp;quot;sleep 180 &amp;amp;&amp;amp; pfctl -d&amp;quot; &amp;gt;failsafe.log 2&amp;gt;&amp;amp;1 &amp;amp; echo $! &amp;gt;failsafe.pid&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A background process that &lt;strong&gt;disables pf unconditionally after 180 seconds&lt;/strong&gt;, unless someone kills it first. If the new rules lock everyone out, the machine reverts to &amp;#8220;no firewall&amp;#8221; on its own: the exact state it was in before, which was known to work. Nobody needs to open a&amp;nbsp;console.&lt;/p&gt;
&lt;p&gt;The full sequence, which it followed for both rounds (with 120 seconds for the second, lower-risk&amp;nbsp;change):&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Write the ruleset to a scratch&amp;nbsp;file.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;doas pfctl -nf &amp;lt;file&amp;gt;&lt;/code&gt;: parse only, nothing&amp;nbsp;loaded.&lt;/li&gt;
&lt;li&gt;Arm the failsafe and record its &lt;span class="caps"&gt;PID&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;Install&amp;nbsp;to &lt;code&gt;/etc/pf.conf&lt;/code&gt;, load the module, enable and load the&amp;nbsp;rules.&lt;/li&gt;
&lt;li&gt;Check &lt;code&gt;pfctl -s states&lt;/code&gt; for the &lt;strong&gt;existing&lt;/strong&gt; &lt;span class="caps"&gt;SSH&lt;/span&gt; session. If it&amp;#8217;s&amp;nbsp;still &lt;code&gt;ESTABLISHED&lt;/code&gt; after the reload, the rules aren&amp;#8217;t killing live traffic. (By default, pf keeps its state table across a ruleset reload, so established connections carry on through the switch. That makes the reload much less risky. It says nothing about whether &lt;em&gt;new&lt;/em&gt; connections will get through, which is what step 6 is&amp;nbsp;for.)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ask me&lt;/strong&gt; to verify from my actual client: a &lt;em&gt;new&lt;/em&gt; &lt;span class="caps"&gt;SSH&lt;/span&gt; connection works, &lt;span class="caps"&gt;RDP&lt;/span&gt; through the tunnel works, and direct &lt;span class="caps"&gt;RDP&lt;/span&gt;&amp;nbsp;fails.&lt;/li&gt;
&lt;li&gt;Only after my confirmation: kill the failsafe &lt;span class="caps"&gt;PID&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;Only after that: make it&amp;nbsp;persistent.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Step 6 is the one that impressed me. An existing session surviving a reload proves very little. The interesting question is whether a &lt;strong&gt;new&lt;/strong&gt; connection gets through, and the agent can&amp;#8217;t test that from inside its own session. So it stopped and asked. That&amp;#8217;s what a careful human admin does, and it&amp;#8217;s often what a hurried one skips. I&amp;#8217;ve skipped it&amp;nbsp;myself.&lt;/p&gt;
&lt;p&gt;A few FreeBSD speed bumps showed up along the way, and the agent handled them without&amp;nbsp;drama: &lt;code&gt;/etc/pf.conf&lt;/code&gt; isn&amp;#8217;t writable by a normal user (staged in scratch,&amp;nbsp;then &lt;code&gt;doas cp&lt;/code&gt;),&amp;nbsp;and &lt;code&gt;pfctl&lt;/code&gt; fails&amp;nbsp;with &lt;code&gt;Failed to open netlink: No such file or directory&lt;/code&gt; until &lt;code&gt;pf.ko&lt;/code&gt; is loaded&amp;nbsp;(&lt;code&gt;doas kldload pf&lt;/code&gt;).&lt;/p&gt;
&lt;h3 id="verification-and-persistence"&gt;Verification and&amp;nbsp;Persistence&lt;/h3&gt;
&lt;p&gt;The compiled ruleset after loading shows&amp;nbsp;what &lt;code&gt;antispoof&lt;/code&gt; expands to, and that pf calls 3389 by its &lt;span class="caps"&gt;IANA&lt;/span&gt;&amp;nbsp;name:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ doas pfctl -s rules
scrub in on vtnet0 all fragment reassemble
block drop in quick on ! vtnet0 inet6 from 2a06:9801:1c:2001::/64 to any
block drop in quick on vtnet0 inet6 from fe80::be24:11ff:fe1a:3271 to any
block drop in quick inet6 from 2a06:9801:1c:2001::203 to any
block drop in quick on ! vtnet0 inet from 10.254.253.0/24 to any
block drop in quick inet from 10.254.253.203 to any
pass out quick on vtnet0 all flags S/SA keep state
pass in quick on vtnet0 proto tcp from any to any port = ssh flags S/SA keep state
block return in log quick on vtnet0 proto tcp from any to any port = ms-wbt-server
block return in quick on vtnet0 all

$ doas pfctl -s states | grep &amp;#39;:22&amp;#39;
all tcp 10.254.253.203:22 &amp;lt;- 10.254.253.11:3443       ESTABLISHED:ESTABLISHED
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I checked from my laptop: new &lt;span class="caps"&gt;SSH&lt;/span&gt; works, direct &lt;span class="caps"&gt;RDP&lt;/span&gt; times out, and this&amp;nbsp;works:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;-L&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;3389&lt;/span&gt;:localhost:3389&lt;span class="w"&gt; &lt;/span&gt;chofstede@bsdesktop
&lt;span class="c1"&gt;# then point KRDC at localhost:3389&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then, and only then, persistence,&amp;nbsp;using &lt;code&gt;sysrc&lt;/code&gt; rather than an&amp;nbsp;editor:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;doas&lt;span class="w"&gt; &lt;/span&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;pf_rules&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/etc/pf.conf&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="nv"&gt;pflog_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;pflog_logfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/log/pflog&amp;quot;&lt;/span&gt;
doas&lt;span class="w"&gt; &lt;/span&gt;service&lt;span class="w"&gt; &lt;/span&gt;pflog&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Day-to-day handling of the abuse&amp;nbsp;table:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;doas&lt;span class="w"&gt; &lt;/span&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;ssh_bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="c1"&gt;# who&amp;#39;s blocked&lt;/span&gt;
doas&lt;span class="w"&gt; &lt;/span&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;ssh_bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;flush&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# unblock everyone&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="what-the-agent-got-wrong"&gt;What the Agent Got&amp;nbsp;Wrong&lt;/h2&gt;
&lt;p&gt;In its notes on the session, Claude recorded one thing it couldn&amp;#8217;t explain. It had looked&amp;nbsp;at &lt;code&gt;/usr/local/etc/doas.conf&lt;/code&gt;, concluded that it contained only the commented-out sample rules, and then observed&amp;nbsp;that &lt;code&gt;doas&lt;/code&gt; gave it root without a password anyway. It flagged that as an unresolved inconsistency and recommended checking which config&amp;nbsp;file &lt;code&gt;doas&lt;/code&gt; actually&amp;nbsp;reads.&lt;/p&gt;
&lt;p&gt;Credit where it&amp;#8217;s due: flagging it instead of glossing over it was the right call. But the diagnosis was wrong, and I&amp;nbsp;checked:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ ls -la /usr/local/etc/doas*
-rw-r--r--  1 root wheel 750 Oct  4 04:36 /usr/local/etc/doas.conf
-rw-r--r--  1 root wheel 750 Oct  4 04:36 /usr/local/etc/doas.conf.sample
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Same size, same timestamp. At install time I&amp;#8217;d copied the sample&amp;nbsp;to &lt;code&gt;doas.conf&lt;/code&gt; and intended to edit it later. And the sample isn&amp;#8217;t commented out. Only its explanations&amp;nbsp;are:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Permit members of the wheel group to perform actions as root.
permit :wheel

# Same without having to enter the password
permit nopass :wheel

# Permit user alice to run commands as a root user.
permit alice as root

# Permit user bob to run programs as root, maintaining
# environment variables. Useful for GUI applications.
permit keepenv bob as root
...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every one of&amp;nbsp;those &lt;code&gt;permit&lt;/code&gt; lines is live.&amp;nbsp;In &lt;code&gt;doas.conf&lt;/code&gt;, &lt;strong&gt;the last matching rule wins&lt;/strong&gt;, so for&amp;nbsp;a &lt;code&gt;wheel&lt;/code&gt; member &lt;code&gt;permit nopass :wheel&lt;/code&gt; overrides &lt;code&gt;permit :wheel&lt;/code&gt;: passwordless root. There&amp;#8217;s no mystery. The agent read a file full&amp;nbsp;of &lt;code&gt;#&lt;/code&gt; lines and pattern-matched &amp;#8220;sample file, all comments&amp;#8221;, when every second line wasn&amp;#8217;t a&amp;nbsp;comment.&lt;/p&gt;
&lt;p&gt;And the actual problem is worse than the one it suspected.&amp;nbsp;Passwordless &lt;code&gt;doas&lt;/code&gt; for my own account was intentional for this lab box. But this file also grants root&amp;nbsp;to &lt;code&gt;alice&lt;/code&gt;, &lt;code&gt;bob&lt;/code&gt;, &lt;code&gt;cindy&lt;/code&gt; and &lt;code&gt;david&lt;/code&gt;. None of those users exist. If any of them ever does (a test account, a package that creates a user, a colleague who happens to be called Bob), they get root with zero further configuration. The real fix is&amp;nbsp;a &lt;code&gt;doas.conf&lt;/code&gt; that contains exactly what I&amp;nbsp;mean:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;permit nopass :wheel
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two lessons, one for each of us. For me: don&amp;#8217;t deploy sample configs &amp;#8220;temporarily&amp;#8221;. For agent-assisted admin work in general: the agent was excellent at the procedural part (stage, validate, failsafe, verify, persist) and wrong on a reading-comprehension question about a privilege file. I&amp;#8217;d have expected it the other way round. Verify the claims about privileges especially, because those are exactly the ones where &amp;#8220;it seemed fine&amp;#8221; is the most expensive&amp;nbsp;mistake.&lt;/p&gt;
&lt;h2 id="loose-ends"&gt;Loose&amp;nbsp;Ends&lt;/h2&gt;
&lt;p&gt;Things that are deliberately or not-yet-deliberately&amp;nbsp;unfinished:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The &lt;code&gt;&amp;lt;ssh_bruteforce&amp;gt;&lt;/code&gt; table never expires.&lt;/strong&gt; An address stays blocked until I flush it or reboot. A root cron job&amp;nbsp;with &lt;code&gt;pfctl -t ssh_bruteforce -T expire 86400&lt;/code&gt; would age entries out after a day. Not done&amp;nbsp;yet.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;xrdp still&amp;nbsp;binds &lt;code&gt;*:3389&lt;/code&gt;.&lt;/strong&gt; The tunnel-only policy is enforced by pf alone. xrdp can bind to loopback directly&amp;nbsp;with &lt;code&gt;port=tcp://.:3389&lt;/code&gt; in &lt;code&gt;xrdp.ini&lt;/code&gt;, and then a disabled or broken firewall no longer exposes &lt;span class="caps"&gt;RDP&lt;/span&gt;. That&amp;#8217;s the right belt-and-braces fix, and I&amp;#8217;ll make&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;xrdp runs as&amp;nbsp;root.&lt;/strong&gt; &lt;code&gt;xrdp.ini&lt;/code&gt; itself&amp;nbsp;says &lt;code&gt;runtime_user&lt;/code&gt;/&lt;code&gt;runtime_group&lt;/code&gt; are &amp;#8220;&lt;span class="caps"&gt;HIGHLY&lt;/span&gt; &lt;span class="caps"&gt;RECOMMENDED&lt;/span&gt;&amp;#8221;. They&amp;#8217;re commented out by&amp;nbsp;default.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No acceleration.&lt;/strong&gt; If a proper &lt;span class="caps"&gt;DRM&lt;/span&gt;/&lt;span class="caps"&gt;KMS&lt;/span&gt; VirtIO-&lt;span class="caps"&gt;GPU&lt;/span&gt; driver shows up for FreeBSD, I&amp;#8217;ll try it. Until then, it&amp;#8217;s&amp;nbsp;llvmpipe.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Plasma 6.8.&lt;/strong&gt; Discussed above. This setup has an expiration date unless &lt;span class="caps"&gt;KRDP&lt;/span&gt; or another Wayland-native &lt;span class="caps"&gt;RDP&lt;/span&gt; path shows up in&amp;nbsp;ports.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The dead nullfs line in my fstab&lt;/strong&gt; is going&amp;nbsp;away.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="was-it-worth-it"&gt;Was It Worth&amp;nbsp;It?&lt;/h2&gt;
&lt;p&gt;Yes. I now have a FreeBSD desktop I open from anywhere with an &lt;span class="caps"&gt;SSH&lt;/span&gt; tunnel and &lt;span class="caps"&gt;KRDC&lt;/span&gt;, and a coding agent that runs on it without complaint. Every individual piece was solvable in minutes once I knew what it actually was. The time went into finding that&amp;nbsp;out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;&lt;span class="caps"&gt;SDDM&lt;/span&gt; is broken&amp;#8221; was really &amp;#8220;no X driver for this virtual&amp;nbsp;display&amp;#8221;.&lt;/li&gt;
&lt;li&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;VirtIO-&lt;span class="caps"&gt;GPU&lt;/span&gt; works&amp;#8221; was really &amp;#8220;the framebuffer&amp;nbsp;works&amp;#8221;.&lt;/li&gt;
&lt;li&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Claude regressed on the Linuxulator&amp;#8221; was really &amp;#8220;I was running a different&amp;nbsp;binary&amp;#8221;.&lt;/li&gt;
&lt;li&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;doas has a config mystery&amp;#8221; was really &amp;#8220;the sample file isn&amp;#8217;t a comment&amp;nbsp;block&amp;#8221;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Four times the label on the problem was wrong. That&amp;#8217;s the general lesson, and it applies to human and agent&amp;nbsp;alike.&lt;/p&gt;
&lt;p&gt;The part I keep coming back to is the agent itself. A Linux binary, on a FreeBSD kernel, behind a Rocky Linux mask, in a &lt;span class="caps"&gt;VM&lt;/span&gt; on a Linux hypervisor, saw through the mask in two commands. Then it built a firewall for the machine it lived on, with a rollback timer, and waited for me to confirm a fresh login before it committed anything. That&amp;#8217;s a better change process than a good fraction of the production changes I&amp;#8217;ve seen in my career. It also misread a config file about root access, and that&amp;#8217;s exactly why a human still has to look at the&amp;nbsp;result.&lt;/p&gt;
&lt;p&gt;Thanks to &lt;a href="https://github.com/insanityinside/claude-freebsd"&gt;insanityinside&lt;/a&gt; for the wrapper. It&amp;#8217;s a small project that does one job properly, and its &lt;span class="caps"&gt;README&lt;/span&gt; answered most of my questions before I had&amp;nbsp;them.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="linuxulator"/><category term="claude-code"/><category term="ai"/><category term="agents"/><category term="kde"/><category term="plasma"/><category term="xrdp"/><category term="proxmox"/><category term="virtio-gpu"/><category term="pf"/><category term="doas"/></entry><entry><title>FreeBSD Foundationals: rc.d - The Service Framework That Fits in Your Head</title><link href="https://hofstede.it/freebsd-foundationals-rcd-the-service-framework-that-fits-in-your-head/" rel="alternate"/><published>2026-10-04T00:00:00+02:00</published><updated>2026-10-04T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-10-04:/freebsd-foundationals-rcd-the-service-framework-that-fits-in-your-head/</id><summary type="html">&lt;p&gt;The fourth in the FreeBSD Foundationals series. This one picks up where the boot article stopped, at the moment init runs /etc/rc. It covers how rc finds and orders scripts with rcorder, how rc.conf and its layers are really just shell, what rc.subr gives you for free, how to drive it all with service and sysrc (including inside jails), how to write a correct rc.d script for your own daemon using daemon(8) supervision, and the pidfile trap that makes &amp;#8220;service stop&amp;#8221; restart your process instead of stopping it. Then an honest audit of the rc.d scripts running my own Mastodon, Zigbee2MQTT and NetBox jails. Plus service jails, debugging, and an honest comparison with systemd&amp;nbsp;units.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="The FreeBSD daemon conducting an orderly startup of storage, networking, services and jails beside an rc.d terminal" src="https://hofstede.it/images/2026-10-04-freebsd-foundationals-rc-d.webp" title="FreeBSD rc.d: bringing services up in order."&gt;&lt;/p&gt;
&lt;p&gt;The &lt;a href="https://hofstede.it/freebsd-foundationals-the-boot-process-from-the-loader-to-boot-environments/"&gt;previous article&lt;/a&gt; followed FreeBSD from power-on through the loader and the kernel, and stopped at a box in the diagram that just&amp;nbsp;said &lt;code&gt;rc(8) scripts&lt;/code&gt;. That box is where your machine actually becomes useful: networking comes up, filesystems get mounted, jails start, and every daemon you care about gets launched. This article opens that&amp;nbsp;box.&lt;/p&gt;
&lt;p&gt;This is the fourth article in the &lt;strong&gt;FreeBSD Foundationals&lt;/strong&gt; series. The &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;first covered Jails&lt;/a&gt;, the &lt;a href="https://hofstede.it/freebsd-foundationals-zfs-the-last-filesystem-youll-ever-need/"&gt;second covered &lt;span class="caps"&gt;ZFS&lt;/span&gt;&lt;/a&gt;, and the &lt;a href="https://hofstede.it/freebsd-foundationals-the-boot-process-from-the-loader-to-boot-environments/"&gt;third covered the boot process&lt;/a&gt;. The boot article promised that packages and ports would come next. I changed the order, because rc.d is the direct continuation of the boot story, and because nearly every FreeBSD article on this blog ends with &amp;#8220;and here&amp;#8217;s the rc.d script&amp;#8221; without ever explaining how rc.d works. Packages are next, I&amp;nbsp;promise.&lt;/p&gt;
&lt;p&gt;rc.d is also one of the best arguments for FreeBSD&amp;#8217;s design approach. The whole service framework is a few thousand lines of &lt;span class="caps"&gt;POSIX&lt;/span&gt; shell that you can read in an afternoon. When something goes wrong, the answer is in a file you can open&amp;nbsp;with &lt;code&gt;less&lt;/code&gt;, not hidden in a&amp;nbsp;binary.&lt;/p&gt;
&lt;p&gt;By the end of this article you&amp;#8217;ll understand how rc decides what starts and in what order, where configuration really comes from,&amp;nbsp;what &lt;code&gt;rc.subr&lt;/code&gt; does for you, how to write a service script that behaves correctly on start, stop, status and shutdown, and which handful of mistakes account for most broken rc.d scripts I&amp;#8217;ve&amp;nbsp;seen.&lt;/p&gt;
&lt;p&gt;All examples were checked against the FreeBSD 15.1&amp;nbsp;sources.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#from-init-to-rc-what-actually-runs"&gt;From init to rc: What Actually&amp;nbsp;Runs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rcconf-is-a-shell-script"&gt;rc.conf Is a Shell Script&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-configuration-layers"&gt;The Configuration&amp;nbsp;Layers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#yes-yes-true-1"&gt;&lt;span class="caps"&gt;YES&lt;/span&gt;, yes, true,&amp;nbsp;1&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#rcorder-dependency-ordering-in-four-comment-lines"&gt;rcorder: Dependency Ordering in Four Comment Lines&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-placeholder-scripts"&gt;The Placeholder&amp;nbsp;Scripts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#seeing-the-order"&gt;Seeing the&amp;nbsp;Order&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-rcorder-is-not"&gt;What rcorder Is&amp;nbsp;Not&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#anatomy-of-an-rcd-script"&gt;Anatomy of an rc.d&amp;nbsp;Script&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-rcsubr-gives-you-for-free"&gt;What rc.subr Gives You for Free&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-verb-prefixes"&gt;The Verb&amp;nbsp;Prefixes&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#service-and-sysrc-the-day-to-day-interface"&gt;service and sysrc: The Day-to-Day Interface&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#inside-jails"&gt;Inside&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#writing-your-own-a-supervised-service-done-right"&gt;Writing Your Own: A Supervised Service Done Right&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-daemon8-flags"&gt;The daemon(8)&amp;nbsp;Flags&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-pidfile-trap-p-vs-p"&gt;The Pidfile Trap: -p vs&amp;nbsp;-P&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-webhookd_args-and-not-webhookd_flags"&gt;Why webhookd_args and Not&amp;nbsp;webhookd_flags&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-webhookd_user-is-cleared"&gt;Why webhookd_user Is&amp;nbsp;Cleared&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-rest"&gt;The&amp;nbsp;Rest&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#case-study-auditing-my-own-scripts"&gt;Case Study: Auditing My Own Scripts&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#mastodon_web-working-around-rcsubr"&gt;mastodon_web: Working Around&amp;nbsp;rc.subr&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#zigbee2mqtt-nearly-right"&gt;zigbee2mqtt: Nearly&amp;nbsp;Right&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#netbox-one-flag-away-from-the-trap"&gt;netbox: One Flag Away from the&amp;nbsp;Trap&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-pattern"&gt;The&amp;nbsp;Pattern&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#service-jails-isolation-as-an-rcconf-one-liner"&gt;Service Jails: Isolation as an rc.conf&amp;nbsp;One-Liner&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#debugging-rcd"&gt;Debugging&amp;nbsp;rc.d&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#coming-from-systemd"&gt;Coming From&amp;nbsp;systemd&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#common-pitfalls-the-checklist"&gt;Common Pitfalls: The&amp;nbsp;Checklist&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="from-init-to-rc-what-actually-runs"&gt;From init to rc: What Actually&amp;nbsp;Runs&lt;/h2&gt;
&lt;p&gt;When the kernel finishes booting, it&amp;nbsp;starts &lt;code&gt;init(8)&lt;/code&gt; as &lt;span class="caps"&gt;PID&lt;/span&gt; 1. In multi-user mode, init does almost nothing itself. It&amp;nbsp;runs &lt;code&gt;/bin/sh /etc/rc autoboot&lt;/code&gt; and waits for it to finish.&amp;nbsp;Once &lt;code&gt;/etc/rc&lt;/code&gt; exits, init&amp;nbsp;spawns &lt;code&gt;getty&lt;/code&gt; on the terminals listed&amp;nbsp;in &lt;code&gt;/etc/ttys&lt;/code&gt;, and you get a login&amp;nbsp;prompt.&lt;/p&gt;
&lt;p&gt;So &amp;#8220;the boot sequence&amp;#8221; from here on is&amp;nbsp;simply &lt;code&gt;/etc/rc&lt;/code&gt;, which is a short shell script. Stripped of comments, it does&amp;nbsp;this:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Sources &lt;code&gt;/etc/rc.subr&lt;/code&gt;, the function library every rc.d script&amp;nbsp;uses.&lt;/li&gt;
&lt;li&gt;Loads the system configuration&amp;nbsp;(&lt;code&gt;/etc/defaults/rc.conf&lt;/code&gt;,&amp;nbsp;then &lt;code&gt;/etc/rc.conf&lt;/code&gt;, and so on; more on that&amp;nbsp;shortly).&lt;/li&gt;
&lt;li&gt;Works out which scripts to &lt;strong&gt;skip&lt;/strong&gt;: anything&amp;nbsp;tagged &lt;code&gt;nostart&lt;/code&gt;,&amp;nbsp;plus &lt;code&gt;nojail&lt;/code&gt; scripts if running inside a jail,&amp;nbsp;plus &lt;code&gt;nojailvnet&lt;/code&gt; if it&amp;#8217;s a non-&lt;span class="caps"&gt;VNET&lt;/span&gt; jail,&amp;nbsp;plus &lt;code&gt;firstboot&lt;/code&gt; scripts unless&amp;nbsp;the &lt;code&gt;/firstboot&lt;/code&gt; sentinel file&amp;nbsp;exists.&lt;/li&gt;
&lt;li&gt;Asks &lt;code&gt;rcorder(8)&lt;/code&gt; to sort all the executable scripts&amp;nbsp;in &lt;code&gt;/etc/rc.d/&lt;/code&gt; into dependency&amp;nbsp;order.&lt;/li&gt;
&lt;li&gt;Runs them in that order, but &lt;strong&gt;stops&amp;nbsp;at &lt;code&gt;FILESYSTEMS&lt;/code&gt;&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Now that all local filesystems are mounted&amp;nbsp;(and &lt;code&gt;/usr/local&lt;/code&gt; is therefore guaranteed to be readable), adds the scripts&amp;nbsp;from &lt;code&gt;local_startup&lt;/code&gt;, which&amp;nbsp;is &lt;code&gt;/usr/local/etc/rc.d&lt;/code&gt; by default, and&amp;nbsp;runs &lt;code&gt;rcorder&lt;/code&gt; again over the combined&amp;nbsp;set.&lt;/li&gt;
&lt;li&gt;Runs everything that hasn&amp;#8217;t run&amp;nbsp;yet.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The two-pass design is the detail people don&amp;#8217;t know about. Scripts from packages live&amp;nbsp;under &lt;code&gt;/usr/local&lt;/code&gt;, which on some systems is a separate filesystem that isn&amp;#8217;t mounted yet during the first pass. So rc runs the base system up to the point where filesystems are mounted&amp;nbsp;(the &lt;code&gt;early_late_divider&lt;/code&gt;, which defaults&amp;nbsp;to &lt;code&gt;FILESYSTEMS&lt;/code&gt;), and only then looks&amp;nbsp;at &lt;code&gt;/usr/local/etc/rc.d&lt;/code&gt;. In practice this means a package&amp;#8217;s script can never run before the local filesystems are mounted, whatever&amp;nbsp;its &lt;code&gt;REQUIRE&lt;/code&gt; line&amp;nbsp;says.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;init(8)
  └── /etc/rc autoboot
        ├── . /etc/rc.subr
        ├── load_rc_config            (defaults → rc.conf → rc.conf.local)
        │
        ├── pass 1: rcorder /etc/rc.d/*
        │     ... hostid, zfs, zvol, fsck, root, mountcritlocal ...
        │     └── FILESYSTEMS          ← early_late_divider: stop here
        │
        ├── pass 2: rcorder /etc/rc.d/* /usr/local/etc/rc.d/*
        │     (skips what already ran)
        │     ... NETWORKING, SERVERS, DAEMON, LOGIN, jail, your apps ...
        │
        └── exit 0  →  init starts getty  →  login:
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Shutdown is the same story in&amp;nbsp;reverse. &lt;code&gt;shutdown -r now&lt;/code&gt; makes init&amp;nbsp;run &lt;code&gt;/etc/rc.shutdown&lt;/code&gt;, which&amp;nbsp;calls &lt;code&gt;rcorder -k shutdown&lt;/code&gt; to select &lt;strong&gt;only&lt;/strong&gt; the scripts carrying&amp;nbsp;the &lt;code&gt;shutdown&lt;/code&gt; keyword, reverses the list, and invokes each script&amp;nbsp;with &lt;code&gt;faststop&lt;/code&gt;: the&amp;nbsp;ordinary &lt;code&gt;stop&lt;/code&gt; method with&amp;nbsp;the &lt;code&gt;fast&lt;/code&gt; prefix (more on prefixes below). A watchdog&amp;nbsp;(&lt;code&gt;rcshutdown_timeout&lt;/code&gt;, 90 seconds by default) kills the whole thing if a script hangs. After that, init sends &lt;span class="caps"&gt;SIGTERM&lt;/span&gt; and then &lt;span class="caps"&gt;SIGKILL&lt;/span&gt; to whatever is left. Keep that in mind: it becomes important when we write our own&amp;nbsp;script.&lt;/p&gt;
&lt;h2 id="rcconf-is-a-shell-script"&gt;rc.conf Is a Shell&amp;nbsp;Script&lt;/h2&gt;
&lt;p&gt;This is the most important mental shift for people coming from other systems: &lt;strong&gt;&lt;code&gt;/etc/rc.conf&lt;/code&gt; is not a config file in the &lt;span class="caps"&gt;INI&lt;/span&gt; sense. It is shell code, and it is sourced&lt;/strong&gt;&amp;nbsp;by &lt;code&gt;/etc/rc&lt;/code&gt; and by every single rc.d script when it&amp;nbsp;runs.&lt;/p&gt;
&lt;p&gt;That has a few&amp;nbsp;consequences:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;sshd_enable="YES"&lt;/code&gt; is a shell variable assignment. No spaces&amp;nbsp;around &lt;code&gt;=&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;A syntax error&amp;nbsp;in &lt;code&gt;rc.conf&lt;/code&gt; (an unbalanced quote, say) doesn&amp;#8217;t break one service. It breaks &lt;strong&gt;all&lt;/strong&gt; of them, because every script sources the same&amp;nbsp;file.&lt;/li&gt;
&lt;li&gt;You &lt;em&gt;can&lt;/em&gt; put logic in there. You shouldn&amp;#8217;t, but you can, and some people&amp;nbsp;do.&lt;/li&gt;
&lt;li&gt;The last assignment wins.&amp;nbsp;If &lt;code&gt;ifconfig_vtnet0&lt;/code&gt; is set twice, the second one is what rc&amp;nbsp;sees.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="the-configuration-layers"&gt;The Configuration&amp;nbsp;Layers&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;load_rc_config&lt;/code&gt; in &lt;code&gt;rc.subr&lt;/code&gt; reads configuration in this order, each layer overriding the one&amp;nbsp;before:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Order&lt;/th&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/etc/defaults/rc.conf&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;FreeBSD defaults. Never edit; upgrades overwrite it.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/etc/defaults/vendor.conf&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Optional vendor overrides of the FreeBSD defaults, for appliances and derived products.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/etc/rc.conf&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Your system configuration.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/etc/rc.conf.local&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Historical local override file, read because it&amp;#8217;s listed&amp;nbsp;in &lt;code&gt;rc_conf_files&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/etc/rc.conf.d/&amp;lt;name&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Per-service configuration, read only by the&amp;nbsp;service &lt;code&gt;&amp;lt;name&amp;gt;&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;&lt;code&gt;&amp;lt;prefix&amp;gt;/rc.conf.d/&amp;lt;name&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Same, for&amp;nbsp;every &lt;code&gt;local_startup&lt;/code&gt; entry with its&amp;nbsp;trailing &lt;code&gt;/rc.d&lt;/code&gt; removed. With the default,&amp;nbsp;that&amp;#8217;s &lt;code&gt;/usr/local/etc/rc.conf.d/&amp;lt;name&amp;gt;&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Layers 5 and 6 are&amp;nbsp;underused. &lt;code&gt;/etc/rc.conf.d/&amp;lt;name&amp;gt;&lt;/code&gt; can be a &lt;strong&gt;file&lt;/strong&gt;, or a &lt;strong&gt;directory&lt;/strong&gt; whose files are all sourced. Two things make this&amp;nbsp;useful:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Configuration management.&lt;/strong&gt; A tool can&amp;nbsp;own &lt;code&gt;/etc/rc.conf.d/nginx&lt;/code&gt; completely without touching (or templating) the&amp;nbsp;shared &lt;code&gt;rc.conf&lt;/code&gt;. One file per service means no merge conflicts and no&amp;nbsp;line-editing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Scoping.&lt;/strong&gt; A per-service file is only sourced when &lt;em&gt;that&lt;/em&gt; service runs. The rest of the system never sees those&amp;nbsp;variables.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The defaults file is worth reading at least&amp;nbsp;once. &lt;code&gt;less /etc/defaults/rc.conf&lt;/code&gt; is the best documentation of what knobs exist for base-system services. The &lt;a href="https://hofstede.it/freebsd-foundationals-the-boot-process-from-the-loader-to-boot-environments/"&gt;boot article&lt;/a&gt; already&amp;nbsp;covered &lt;code&gt;sysrc&lt;/code&gt; for editing all of this safely, and everything there applies here:&amp;nbsp;use &lt;code&gt;sysrc&lt;/code&gt; instead of an editor, and the &amp;#8220;breaks every service&amp;#8221; risk mostly&amp;nbsp;disappears.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Write into a per-service file instead of rc.conf&lt;/span&gt;
install&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.conf.d
sysrc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.conf.d/nginx&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;nginx_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;nginx_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-q&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Quick syntax check of the shared file after manual edits&lt;/span&gt;
sh&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.conf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;OK
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="yes-yes-true-1"&gt;&lt;span class="caps"&gt;YES&lt;/span&gt;, yes, true,&amp;nbsp;1&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;checkyesno&lt;/code&gt;, the function every script uses to test&amp;nbsp;an &lt;code&gt;_enable&lt;/code&gt; variable,&amp;nbsp;accepts &lt;code&gt;YES&lt;/code&gt;, &lt;code&gt;TRUE&lt;/code&gt;, &lt;code&gt;ON&lt;/code&gt; and &lt;code&gt;1&lt;/code&gt; (case-insensitive) as true&amp;nbsp;and &lt;code&gt;NO&lt;/code&gt;, &lt;code&gt;FALSE&lt;/code&gt;, &lt;code&gt;OFF&lt;/code&gt; and &lt;code&gt;0&lt;/code&gt; as false. Anything else (an empty string, a typo&amp;nbsp;like &lt;code&gt;YSE&lt;/code&gt;) produces a warning at&amp;nbsp;boot: &lt;code&gt;$foo_enable is not set properly - see rc.conf(5)&lt;/code&gt;. If you ever see that line, you have a&amp;nbsp;typo.&lt;/p&gt;
&lt;h2 id="rcorder-dependency-ordering-in-four-comment-lines"&gt;rcorder: Dependency Ordering in Four Comment&amp;nbsp;Lines&lt;/h2&gt;
&lt;p&gt;There&amp;#8217;s no manually maintained central list of what starts when. Instead, every rc.d script declares its relationship to other scripts in specially formatted comments,&amp;nbsp;and &lt;code&gt;rcorder(8)&lt;/code&gt; topologically sorts them. Here&amp;#8217;s the header&amp;nbsp;of &lt;code&gt;/etc/rc.d/sshd&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# PROVIDE: sshd&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: LOGIN FILESYSTEMS&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The four directives (strictly&amp;nbsp;speaking, &lt;code&gt;KEYWORD&lt;/code&gt; is a selection tag, not a&amp;nbsp;dependency):&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Directive&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;PROVIDE:&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Names this script provides. Usually just its own name.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;REQUIRE:&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Names that must run &lt;strong&gt;before&lt;/strong&gt; this script.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;BEFORE:&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Names this script must run &lt;strong&gt;before&lt;/strong&gt;. The inverse&amp;nbsp;of &lt;code&gt;REQUIRE&lt;/code&gt;, useful for inserting yourself early without editing other scripts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;KEYWORD:&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Tags used to include or exclude&amp;nbsp;scripts: &lt;code&gt;shutdown&lt;/code&gt;, &lt;code&gt;nojail&lt;/code&gt;, &lt;code&gt;nojailvnet&lt;/code&gt;, &lt;code&gt;firstboot&lt;/code&gt;, &lt;code&gt;nostart&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="the-placeholder-scripts"&gt;The Placeholder&amp;nbsp;Scripts&lt;/h3&gt;
&lt;p&gt;You&amp;#8217;ll&amp;nbsp;notice &lt;code&gt;sshd&lt;/code&gt; requires &lt;code&gt;LOGIN&lt;/code&gt; and &lt;code&gt;FILESYSTEMS&lt;/code&gt;, which aren&amp;#8217;t services. They&amp;#8217;re &lt;strong&gt;placeholders&lt;/strong&gt;: empty scripts&amp;nbsp;in &lt;code&gt;/etc/rc.d&lt;/code&gt; that exist only to provide a name and mark a milestone in the boot sequence.&amp;nbsp;Here&amp;#8217;s &lt;code&gt;/etc/rc.d/DAEMON&lt;/code&gt; in&amp;nbsp;full:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# PROVIDE: DAEMON&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: NETWORKING SERVERS&lt;/span&gt;

&lt;span class="c1"&gt;#   This is a dummy dependency, to ensure that general purpose daemons&lt;/span&gt;
&lt;span class="c1"&gt;#   are run _after_ the above are.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The milestones, in&amp;nbsp;order:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Placeholder&lt;/th&gt;
&lt;th&gt;Reached when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;FILESYSTEMS&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;All local filesystems are mounted. Also the early/late divider.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;NETWORKING&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Interfaces are configured, routing is up.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;SERVERS&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Basic system servers (syslogd, etc.) are running.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DAEMON&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;General-purpose daemons can start.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;LOGIN&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Everything is up; users may log in.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For a typical third-party&amp;nbsp;daemon, &lt;code&gt;REQUIRE: DAEMON&lt;/code&gt; is a good starting point.&amp;nbsp;Add &lt;code&gt;NETWORKING&lt;/code&gt;, &lt;code&gt;LOGIN&lt;/code&gt; or a concrete provider&amp;nbsp;(say, &lt;code&gt;REQUIRE: postgresql&lt;/code&gt;) when the service actually depends on that&amp;nbsp;milestone. &lt;code&gt;LOGIN&lt;/code&gt; in particular is the checkpoint for login services and things that run commands as users, such as cron and inetd. It isn&amp;#8217;t a catch-all for &amp;#8220;anything from a&amp;nbsp;package&amp;#8221;.&lt;/p&gt;
&lt;h3 id="seeing-the-order"&gt;Seeing the&amp;nbsp;Order&lt;/h3&gt;
&lt;p&gt;You never need to&amp;nbsp;guess. &lt;code&gt;service -r&lt;/code&gt; prints the exact order rc would use at&amp;nbsp;boot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;-E&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;FILESYSTEMS|NETWORKING|DAEMON|LOGIN|nginx|jail&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;38:/etc/rc.d/FILESYSTEMS
71:/etc/rc.d/NETWORKING
92:/etc/rc.d/DAEMON
133:/etc/rc.d/LOGIN
141:/usr/local/etc/rc.d/nginx
157:/etc/rc.d/jail
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(Line numbers will differ on your system.) If you&amp;#8217;ve just written a script and want to know where it lands, this is the&amp;nbsp;command.&lt;/p&gt;
&lt;h3 id="what-rcorder-is-not"&gt;What rcorder Is&amp;nbsp;Not&lt;/h3&gt;
&lt;p&gt;Here&amp;#8217;s a point that confuses anyone coming from systemd: &lt;strong&gt;&lt;code&gt;REQUIRE&lt;/code&gt; is about ordering, not activation.&lt;/strong&gt; If your script&amp;nbsp;says &lt;code&gt;REQUIRE: postgresql&lt;/code&gt; and PostgreSQL isn&amp;#8217;t enabled, nothing starts PostgreSQL for you, and your script still runs, just in the position where PostgreSQL &lt;em&gt;would&lt;/em&gt; have run. Likewise, if PostgreSQL fails to start, rc carries on and starts your script&amp;nbsp;anyway.&lt;/p&gt;
&lt;p&gt;rcorder decides the order. Whether something runs is decided by&amp;nbsp;its &lt;code&gt;_enable&lt;/code&gt; variable, and that&amp;#8217;s all. If your service really can&amp;#8217;t work without another one, check for it in&amp;nbsp;a &lt;code&gt;start_precmd&lt;/code&gt; (shown later) and fail with a clear&amp;nbsp;message.&lt;/p&gt;
&lt;h2 id="anatomy-of-an-rcd-script"&gt;Anatomy of an rc.d&amp;nbsp;Script&lt;/h2&gt;
&lt;p&gt;Let&amp;#8217;s read the&amp;nbsp;real &lt;code&gt;sshd&lt;/code&gt; script from 15.1, trimmed to the parts that&amp;nbsp;matter:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# PROVIDE: sshd&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: LOGIN FILESYSTEMS&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;

.&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.subr

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sshd&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;desc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Secure Shell Daemon&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rcvar&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sshd_enable&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/sbin/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;start_precmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sshd_precmd&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;reload_precmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sshd_configtest&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;restart_precmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sshd_configtest&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;configtest_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sshd_configtest&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;extra_commands&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;configtest keygen reload&amp;quot;&lt;/span&gt;

sshd_configtest&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Performing sanity check on &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; configuration.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;eval&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;sshd_flags&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-t
&lt;span class="o"&gt;}&lt;/span&gt;

sshd_precmd&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;run_rc_command&lt;span class="w"&gt; &lt;/span&gt;keygen
&lt;span class="w"&gt;    &lt;/span&gt;run_rc_command&lt;span class="w"&gt; &lt;/span&gt;configtest
&lt;span class="o"&gt;}&lt;/span&gt;

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;
run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The useful mental model, and the thesis of this whole article: &lt;strong&gt;an rc.d script normally declares a service rather than implementing its lifecycle.&lt;/strong&gt; rc.subr owns the lifecycle; the script supplies metadata, configuration, and the&amp;nbsp;exceptions.&lt;/p&gt;
&lt;p&gt;Which is why there&amp;#8217;s surprisingly little here. The script doesn&amp;#8217;t contain any code to start a process, find its &lt;span class="caps"&gt;PID&lt;/span&gt;, stop it, or report status. It &lt;strong&gt;declares&lt;/strong&gt; a handful of variables, defines two small helper functions, and hands control&amp;nbsp;to &lt;code&gt;run_rc_command&lt;/code&gt;. Everything else comes&amp;nbsp;from &lt;code&gt;rc.subr&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The structure is always the&amp;nbsp;same:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Header&lt;/strong&gt;&amp;nbsp;with &lt;code&gt;PROVIDE&lt;/code&gt;/&lt;code&gt;REQUIRE&lt;/code&gt;/&lt;code&gt;KEYWORD&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;. /etc/rc.subr&lt;/code&gt;&lt;/strong&gt; to load the&amp;nbsp;library.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Declarations&lt;/strong&gt;: &lt;code&gt;name&lt;/code&gt;, &lt;code&gt;rcvar&lt;/code&gt;, &lt;code&gt;command&lt;/code&gt;, &lt;code&gt;pidfile&lt;/code&gt;, and&amp;nbsp;hooks.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;load_rc_config $name&lt;/code&gt;&lt;/strong&gt; to read rc.conf and the per-service files. This must come &lt;strong&gt;after&lt;/strong&gt;&amp;nbsp;setting &lt;code&gt;name&lt;/code&gt;, and &lt;strong&gt;before&lt;/strong&gt; you use any config&amp;nbsp;variable.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;run_rc_command "$1"&lt;/code&gt;&lt;/strong&gt; to&amp;nbsp;dispatch &lt;code&gt;start&lt;/code&gt;, &lt;code&gt;stop&lt;/code&gt;, &lt;code&gt;status&lt;/code&gt; or whatever the user asked&amp;nbsp;for.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="what-rcsubr-gives-you-for-free"&gt;What rc.subr Gives You for&amp;nbsp;Free&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;run_rc_command&lt;/code&gt; implements &lt;code&gt;start&lt;/code&gt;, &lt;code&gt;stop&lt;/code&gt;, &lt;code&gt;restart&lt;/code&gt;, &lt;code&gt;status&lt;/code&gt;, &lt;code&gt;poll&lt;/code&gt;, &lt;code&gt;rcvar&lt;/code&gt;, &lt;code&gt;enabled&lt;/code&gt;, &lt;code&gt;describe&lt;/code&gt;, &lt;code&gt;enable&lt;/code&gt;, &lt;code&gt;disable&lt;/code&gt; and &lt;code&gt;delete&lt;/code&gt;, plus anything listed&amp;nbsp;in &lt;code&gt;extra_commands&lt;/code&gt;. What each of them does is driven by variables. These are the ones worth&amp;nbsp;knowing:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Variable&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Service name. Required. Prefix for&amp;nbsp;all &lt;code&gt;${name}_*&lt;/code&gt; variables.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rcvar&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The enable variable,&amp;nbsp;conventionally &lt;code&gt;${name}_enable&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;desc&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;One-line description, shown&amp;nbsp;by &lt;code&gt;service foo describe&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;command&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Full path of the program to run.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;command_args&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Arguments set &lt;strong&gt;by the script&lt;/strong&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;pidfile&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Where to find the &lt;span class="caps"&gt;PID&lt;/span&gt;. Strongly recommended.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;procname&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Process name to verify against the &lt;span class="caps"&gt;PID&lt;/span&gt;. Defaults&amp;nbsp;to &lt;code&gt;command&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;required_files&lt;/code&gt;, &lt;code&gt;required_dirs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Refuse to start if these don&amp;#8217;t exist.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;required_modules&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Kernel modules&amp;nbsp;to &lt;code&gt;kldload&lt;/code&gt; before starting.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sig_stop&lt;/code&gt;, &lt;code&gt;sig_reload&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Signals for stop (default &lt;span class="caps"&gt;TERM&lt;/span&gt;) and reload (default &lt;span class="caps"&gt;HUP&lt;/span&gt;).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;extra_commands&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Additional verbs, each implemented by&amp;nbsp;a &lt;code&gt;&amp;lt;verb&amp;gt;_cmd&lt;/code&gt; function.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;&amp;lt;verb&amp;gt;_precmd&lt;/code&gt;, &lt;code&gt;&amp;lt;verb&amp;gt;_postcmd&lt;/code&gt;, &lt;code&gt;&amp;lt;verb&amp;gt;_cmd&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Hooks before, after, or instead of the default for any verb.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;And these &lt;strong&gt;per-service variables are set by the admin in rc.conf&lt;/strong&gt;, not in the script, and are honoured automatically by any script built&amp;nbsp;on &lt;code&gt;rc.subr&lt;/code&gt;:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;rc.conf variable&lt;/th&gt;
&lt;th&gt;Effect&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_enable&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Start at boot.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_flags&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Extra arguments, added&amp;nbsp;before &lt;code&gt;command_args&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_user&lt;/code&gt;, &lt;code&gt;${name}_group&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Run as this user&amp;nbsp;(via &lt;code&gt;su -m&lt;/code&gt;).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_chdir&lt;/code&gt;, &lt;code&gt;${name}_chroot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Working directory, or chroot.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_env&lt;/code&gt;, &lt;code&gt;${name}_env_file&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Environment variables, inline or from a file.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_nice&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Scheduling priority.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_fib&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Routing table,&amp;nbsp;via &lt;code&gt;setfib(1)&lt;/code&gt;. Ties in nicely with &lt;a href="https://hofstede.it/dual-fib-policy-routing-on-freebsd-two-upstreams-one-server-zero-confusion/"&gt;dual-&lt;span class="caps"&gt;FIB&lt;/span&gt; policy routing&lt;/a&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_limits&lt;/code&gt;, &lt;code&gt;${name}_login_class&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Resource limits&amp;nbsp;via &lt;code&gt;limits(1)&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_oomprotect&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Protect from the &lt;span class="caps"&gt;OOM&lt;/span&gt; killer&amp;nbsp;(&lt;code&gt;YES&lt;/code&gt; or &lt;code&gt;ALL&lt;/code&gt; to include children).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;${name}_svcj&lt;/code&gt;, &lt;code&gt;${name}_svcj_options&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Run inside a service jail. Covered below.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Read that list again with a specific service in mind. You can put any service on the second &lt;span class="caps"&gt;FIB&lt;/span&gt;, renice it, protect it from the &lt;span class="caps"&gt;OOM&lt;/span&gt; killer, give it an environment file, or jail it, &lt;strong&gt;without touching its script&lt;/strong&gt;, just by adding a line&amp;nbsp;to &lt;code&gt;rc.conf&lt;/code&gt;. A port author doesn&amp;#8217;t have to anticipate any of&amp;nbsp;this; &lt;code&gt;rc.subr&lt;/code&gt; handles it uniformly for every&amp;nbsp;service.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Run unbound on FIB 1, with OOM protection, without editing its script&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;unbound_fib&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;unbound_oomprotect&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
service&lt;span class="w"&gt; &lt;/span&gt;unbound&lt;span class="w"&gt; &lt;/span&gt;restart
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="the-verb-prefixes"&gt;The Verb&amp;nbsp;Prefixes&lt;/h3&gt;
&lt;p&gt;Every verb accepts a prefix that changes its&amp;nbsp;checks:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Prefix&lt;/th&gt;
&lt;th&gt;Effect&lt;/th&gt;
&lt;th&gt;Typical use&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;one&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Ignore &lt;code&gt;_enable&lt;/code&gt;; run anyway, with all other checks.&lt;/td&gt;
&lt;td&gt;&lt;code&gt;service nginx onestart&lt;/code&gt; to test a service you haven&amp;#8217;t enabled yet.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;force&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Ignore &lt;code&gt;_enable&lt;/code&gt; &lt;strong&gt;and&lt;/strong&gt;&amp;nbsp;failing &lt;code&gt;precmd&lt;/code&gt;/&lt;code&gt;required_*&lt;/code&gt; checks.&lt;/td&gt;
&lt;td&gt;Last resort.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;fast&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Skip the &amp;#8220;already running?&amp;#8221; check.&lt;/td&gt;
&lt;td&gt;Used by rc at boot for speed.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;quiet&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Suppress &amp;#8220;Starting foo.&amp;#8221; and not-enabled messages.&lt;/td&gt;
&lt;td&gt;Used by rc for non-autoboot.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;code&gt;onestart&lt;/code&gt;/&lt;code&gt;onestop&lt;/code&gt;/&lt;code&gt;onestatus&lt;/code&gt; are the ones you&amp;#8217;ll use daily. If you&amp;nbsp;get &lt;code&gt;Cannot 'start' nginx. Set nginx_enable to YES in /etc/rc.conf or use 'onestart' instead of 'start'.&lt;/code&gt;, that&amp;#8217;s rc.d telling you exactly&amp;nbsp;this.&lt;/p&gt;
&lt;h2 id="service-and-sysrc-the-day-to-day-interface"&gt;service and sysrc: The Day-to-Day&amp;nbsp;Interface&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;service(8)&lt;/code&gt; is a small wrapper that finds the script&amp;nbsp;in &lt;code&gt;/etc/rc.d&lt;/code&gt; or &lt;code&gt;local_startup&lt;/code&gt; and runs it. Two details&amp;nbsp;matter.&lt;/p&gt;
&lt;p&gt;First, it deliberately runs the script with the same restricted environment it gets at&amp;nbsp;boot: &lt;code&gt;HOME=/&lt;/code&gt; and a&amp;nbsp;minimal &lt;code&gt;PATH=/sbin:/bin:/usr/sbin:/usr/bin&lt;/code&gt;. Don&amp;#8217;t rely on your interactive&amp;nbsp;shell&amp;#8217;s &lt;code&gt;PATH&lt;/code&gt; or exported variables. (On&amp;nbsp;15.1, &lt;code&gt;service.sh&lt;/code&gt; implements this by starting the script&amp;nbsp;through &lt;code&gt;env -i&lt;/code&gt;, so nothing else from your shell gets through either.) This matches the boot environment, so &lt;strong&gt;always test&amp;nbsp;with &lt;code&gt;service&lt;/code&gt;, not by running the script directly from your shell&lt;/strong&gt;. A script that only works when run directly is a script that will fail at 3 &lt;span class="caps"&gt;AM&lt;/span&gt; after a&amp;nbsp;reboot.&lt;/p&gt;
&lt;p&gt;Second, it can do most things you&amp;#8217;d otherwise do by&amp;nbsp;hand:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;reload
service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;configtest&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="c1"&gt;# any extra_command works&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;describe&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="c1"&gt;# prints $desc&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;enabled&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;on&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# exit status only, for scripts&lt;/span&gt;

service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="c1"&gt;# sets nginx_enable=YES via sysrc&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;disable&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="c1"&gt;# sets nginx_enable=NO&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;delete&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="c1"&gt;# removes nginx_enable entirely&lt;/span&gt;

service&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt;                          &lt;/span&gt;&lt;span class="c1"&gt;# list enabled services, in boot order&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt;                          &lt;/span&gt;&lt;span class="c1"&gt;# list all available scripts&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt;                          &lt;/span&gt;&lt;span class="c1"&gt;# full boot order (rcorder output)&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;-R&lt;span class="w"&gt;                          &lt;/span&gt;&lt;span class="c1"&gt;# restart all enabled local services&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;service -e&lt;/code&gt; is underrated. It&amp;#8217;s the one-line answer to &amp;#8220;what does this machine actually run?&amp;#8221;, and it&amp;#8217;s the first thing I type on a box I didn&amp;#8217;t&amp;nbsp;build.&lt;/p&gt;
&lt;h3 id="inside-jails"&gt;Inside&amp;nbsp;Jails&lt;/h3&gt;
&lt;p&gt;Both tools speak jail natively, which ties straight back to the &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;first article in this series&lt;/a&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Manage services in a jail from the host - no ssh, no jexec shell&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;www&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;restart
service&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;www&lt;span class="w"&gt; &lt;/span&gt;-e

&lt;span class="c1"&gt;# Edit the jail&amp;#39;s rc.conf from the host&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;www&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;nginx_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A full jail runs its&amp;nbsp;own &lt;code&gt;/etc/rc&lt;/code&gt; when it starts&amp;nbsp;(via &lt;code&gt;exec.start = "/bin/sh /etc/rc"&lt;/code&gt; in the jail config), with all the same machinery. The only difference is the skip list: scripts&amp;nbsp;tagged &lt;code&gt;nojail&lt;/code&gt; don&amp;#8217;t run, which is why you don&amp;#8217;t&amp;nbsp;see &lt;code&gt;devd&lt;/code&gt; or &lt;code&gt;ntpd&lt;/code&gt; or disk-related scripts trying to start inside a jail. Non-&lt;span class="caps"&gt;VNET&lt;/span&gt; jails additionally&amp;nbsp;skip &lt;code&gt;nojailvnet&lt;/code&gt; scripts, so there&amp;#8217;s no attempt to configure interfaces they don&amp;#8217;t&amp;nbsp;own.&lt;/p&gt;
&lt;h2 id="writing-your-own-a-supervised-service-done-right"&gt;Writing Your Own: A Supervised Service Done&amp;nbsp;Right&lt;/h2&gt;
&lt;p&gt;Time to write one. The scenario is the most common one on my servers: a modern application that runs in the &lt;strong&gt;foreground&lt;/strong&gt;, logs to stdout, writes no pidfile of its own, and should be restarted if it crashes. Think of a Go binary, a Node app, or a Python service. I&amp;#8217;ll call&amp;nbsp;it &lt;code&gt;webhookd&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;FreeBSD&amp;#8217;s answer to &amp;#8220;supervise a foreground process&amp;#8221;&amp;nbsp;is &lt;code&gt;daemon(8)&lt;/code&gt;, which is in base. It detaches from the terminal, drops privileges, redirects output to syslog or a file, writes pidfiles, and optionally restarts the child when it exits. Combined&amp;nbsp;with &lt;code&gt;rc.subr&lt;/code&gt;, you get roughly what a systemd unit&amp;nbsp;with &lt;code&gt;Restart=on-failure&lt;/code&gt; gives you, in about 40 lines of&amp;nbsp;shell:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# PROVIDE: webhookd&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: DAEMON NETWORKING&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;

.&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.subr

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;webhookd&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;desc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Webhook receiver&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rcvar&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;webhookd_enable&amp;quot;&lt;/span&gt;

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;

&lt;span class="c1"&gt;# Defaults; override any of these in rc.conf or /etc/rc.conf.d/webhookd&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;webhookd_enable&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;webhookd_user&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;webhookd&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;webhookd_config&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/local/etc/webhookd.yml&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;webhookd_args&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--listen [::1]:9000&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Clear webhookd_user before rc.subr sees it: daemon(8) handles the&lt;/span&gt;
&lt;span class="c1"&gt;# privilege drop, and rc.subr would otherwise su the supervisor itself.&lt;/span&gt;
&lt;span class="nv"&gt;_webhookd_runas&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;webhookd_user&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;webhookd_user&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;procname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/sbin/daemon&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/sbin/daemon&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command_args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-r -R 5 -P &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; -t &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; -S -T &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; \&lt;/span&gt;
&lt;span class="s2"&gt;    -u &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;_webhookd_runas&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; \&lt;/span&gt;
&lt;span class="s2"&gt;    /usr/local/bin/webhookd --config &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;webhookd_config&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;webhookd_args&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;required_files&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;webhookd_config&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;start_precmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;webhookd_precmd&amp;quot;&lt;/span&gt;

webhookd_precmd&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;id&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;_webhookd_runas&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;err&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;user &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;_webhookd_runas&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; does not exist&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Install it&amp;nbsp;as &lt;code&gt;/usr/local/etc/rc.d/webhookd&lt;/code&gt;, make it executable, and enable&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;install&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0555&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;webhookd.rc&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/rc.d/webhookd
service&lt;span class="w"&gt; &lt;/span&gt;webhookd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;webhookd&lt;span class="w"&gt; &lt;/span&gt;start
service&lt;span class="w"&gt; &lt;/span&gt;webhookd&lt;span class="w"&gt; &lt;/span&gt;status
&lt;span class="c1"&gt;# webhookd is running as pid 4182.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Almost every line in that script is there to avoid a specific bug. Let&amp;#8217;s go through&amp;nbsp;them.&lt;/p&gt;
&lt;h3 id="the-daemon8-flags"&gt;The daemon(8)&amp;nbsp;Flags&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Flag&lt;/th&gt;
&lt;th&gt;Why&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;-r -R 5&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Supervise: restart the child if it exits, after 5 seconds&amp;nbsp;(plain &lt;code&gt;-r&lt;/code&gt; uses 1 second). The delay stops a crash loop from pinning a &lt;span class="caps"&gt;CPU&lt;/span&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;-P ${pidfile}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Write the &lt;strong&gt;supervisor&amp;#8217;s&lt;/strong&gt; &lt;span class="caps"&gt;PID&lt;/span&gt;. This is the important one, see below.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;-t ${name}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Process title,&amp;nbsp;so &lt;code&gt;ps&lt;/code&gt; shows &lt;code&gt;daemon: webhookd[4183]&lt;/code&gt; instead of a long command line.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;-S -T ${name}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Send the child&amp;#8217;s stdout/stderr to syslog with the&amp;nbsp;tag &lt;code&gt;webhookd&lt;/code&gt; (facility &lt;code&gt;daemon&lt;/code&gt;,&amp;nbsp;priority &lt;code&gt;notice&lt;/code&gt; by default).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;-u ${_webhookd_runas}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Run the &lt;strong&gt;child&lt;/strong&gt; as an unprivileged user; the supervisor stays root.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;If you&amp;#8217;d rather have a log file than syslog,&amp;nbsp;replace &lt;code&gt;-S -T&lt;/code&gt; with &lt;code&gt;-o /var/log/webhookd.log -H&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;-H&lt;/code&gt; makes daemon reopen the file on &lt;span class="caps"&gt;SIGHUP&lt;/span&gt;,&amp;nbsp;so &lt;code&gt;newsyslog&lt;/code&gt; can rotate&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="the-pidfile-trap-p-vs-p"&gt;The Pidfile Trap: -p vs&amp;nbsp;-P&lt;/h3&gt;
&lt;p&gt;This is the bug I see most often, and I&amp;#8217;ve seen it in production scripts written by experienced&amp;nbsp;people. &lt;code&gt;daemon(8)&lt;/code&gt; has two pidfile&amp;nbsp;options:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;-p child_pidfile&lt;/code&gt;&lt;/strong&gt; records the &lt;span class="caps"&gt;PID&lt;/span&gt; of the&amp;nbsp;application.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;-P supervisor_pidfile&lt;/code&gt;&lt;/strong&gt; records the &lt;span class="caps"&gt;PID&lt;/span&gt; of&amp;nbsp;the &lt;code&gt;daemon&lt;/code&gt; process&amp;nbsp;itself.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Now&amp;nbsp;combine &lt;code&gt;-p&lt;/code&gt; with &lt;code&gt;-r&lt;/code&gt;. You&amp;nbsp;run &lt;code&gt;service webhookd stop&lt;/code&gt;. rc.subr reads the pidfile, finds the &lt;strong&gt;application&amp;#8217;s&lt;/strong&gt; &lt;span class="caps"&gt;PID&lt;/span&gt;, and sends it &lt;span class="caps"&gt;SIGTERM&lt;/span&gt;. The application exits cleanly. The supervisor, which is still running, does exactly what you told it to: it notices the child died and &lt;strong&gt;restarts it&lt;/strong&gt;. &lt;code&gt;service stop&lt;/code&gt; reported success, and the service is running again five seconds&amp;nbsp;later.&lt;/p&gt;
&lt;p&gt;With &lt;code&gt;-P&lt;/code&gt;, &lt;code&gt;stop&lt;/code&gt; signals the supervisor instead. daemon forwards &lt;span class="caps"&gt;SIGTERM&lt;/span&gt; to the child, waits for it to exit, removes the pidfiles, and exits itself. That&amp;#8217;s what you actually wanted.&amp;nbsp;The &lt;code&gt;daemon(8)&lt;/code&gt; man page says as much in its own&amp;nbsp;words: &lt;code&gt;--supervisor-pidfile&lt;/code&gt; &lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;is especially important if you&amp;nbsp;use &lt;code&gt;--restart&lt;/code&gt; in an rc&amp;nbsp;script&amp;#8221;.&lt;/p&gt;
&lt;p&gt;Because the pidfile now points&amp;nbsp;at &lt;code&gt;/usr/sbin/daemon&lt;/code&gt;, &lt;code&gt;procname&lt;/code&gt; has to match. rc.subr doesn&amp;#8217;t blindly trust a pidfile: it checks that the &lt;span class="caps"&gt;PID&lt;/span&gt; belongs to a process whose name&amp;nbsp;matches &lt;code&gt;procname&lt;/code&gt;, so a stale pidfile pointing at a recycled &lt;span class="caps"&gt;PID&lt;/span&gt; doesn&amp;#8217;t make it kill some unrelated process. daemon&amp;#8217;s retitled process shows up&amp;nbsp;as &lt;code&gt;daemon: webhookd[4183]&lt;/code&gt;, and rc.subr explicitly accepts&amp;nbsp;the &lt;code&gt;daemon:&lt;/code&gt; form,&amp;nbsp;so &lt;code&gt;procname="/usr/sbin/daemon"&lt;/code&gt; is correct. (It&amp;#8217;s also the default&amp;nbsp;when &lt;code&gt;command&lt;/code&gt; is daemon, but I set it explicitly so the next person reading the script doesn&amp;#8217;t have to know&amp;nbsp;that.)&lt;/p&gt;
&lt;p&gt;You can see both&amp;nbsp;processes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ps&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;pid,ppid,user,command&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/var/run/webhookd.pid&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;pgrep&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/var/run/webhookd.pid&lt;span class="k"&gt;))&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;  PID  PPID USER     COMMAND
 4182     1 root     daemon: webhookd[4183] (daemon)
 4183  4182 webhookd /usr/local/bin/webhookd --config /usr/local/etc/webhookd.yml --listen [::1]:9000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="why-webhookd_args-and-not-webhookd_flags"&gt;Why webhookd_args and Not&amp;nbsp;webhookd_flags&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;rc.subr&lt;/code&gt; builds the start command&amp;nbsp;as &lt;code&gt;$command ${name}_flags $command_args&lt;/code&gt;.&amp;nbsp;Normally &lt;code&gt;command&lt;/code&gt; is the application,&amp;nbsp;so &lt;code&gt;_flags&lt;/code&gt; lands where you&amp;#8217;d expect. But&amp;nbsp;here &lt;code&gt;command&lt;/code&gt; is &lt;code&gt;/usr/sbin/daemon&lt;/code&gt;. Anything an admin puts&amp;nbsp;in &lt;code&gt;webhookd_flags&lt;/code&gt; ends up &lt;strong&gt;between &lt;code&gt;daemon&lt;/code&gt; and its own options&lt;/strong&gt;, and daemon tries to&amp;nbsp;parse &lt;code&gt;--listen&lt;/code&gt; as one of its own flags and&amp;nbsp;fails.&lt;/p&gt;
&lt;p&gt;So for a daemon-wrapped service, don&amp;#8217;t&amp;nbsp;use &lt;code&gt;_flags&lt;/code&gt; for application arguments. Use your own variable&amp;nbsp;(&lt;code&gt;webhookd_args&lt;/code&gt; above) and put it at the end&amp;nbsp;of &lt;code&gt;command_args&lt;/code&gt;, after the application path, where it belongs. Many ports follow the same convention for the same&amp;nbsp;reason.&lt;/p&gt;
&lt;h3 id="why-webhookd_user-is-cleared"&gt;Why webhookd_user Is&amp;nbsp;Cleared&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;${name}_user&lt;/code&gt; is one of the variables rc.subr honours automatically: it would wrap the whole start command&amp;nbsp;in &lt;code&gt;su -m webhookd&lt;/code&gt;. That would run &lt;strong&gt;daemon itself&lt;/strong&gt; unprivileged, which then can&amp;#8217;t write its pidfile&amp;nbsp;to &lt;code&gt;/var/run&lt;/code&gt;, and can&amp;#8217;t&amp;nbsp;call &lt;code&gt;-u&lt;/code&gt; because only root can change user. The script keeps the&amp;nbsp;familiar &lt;code&gt;webhookd_user&lt;/code&gt; knob for the admin, copies it, and clears the original so rc.subr doesn&amp;#8217;t act on it. Privilege dropping happens exactly once, in&amp;nbsp;daemon(8).&lt;/p&gt;
&lt;p&gt;An alternative is to&amp;nbsp;keep &lt;code&gt;${name}_user&lt;/code&gt;,&amp;nbsp;drop &lt;code&gt;-u&lt;/code&gt;, and put the pidfile in a directory the service user owns&amp;nbsp;(e.g. &lt;code&gt;/var/run/webhookd/&lt;/code&gt;). Both work. The important thing is to choose &lt;strong&gt;one&lt;/strong&gt; mechanism and not&amp;nbsp;both.&lt;/p&gt;
&lt;h3 id="the-rest"&gt;The&amp;nbsp;Rest&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;KEYWORD: shutdown&lt;/code&gt;&lt;/strong&gt; puts the script&amp;nbsp;in &lt;code&gt;rc.shutdown&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s list,&amp;nbsp;so &lt;code&gt;stop&lt;/code&gt; is called in reverse order on shutdown, before init sends &lt;span class="caps"&gt;SIGTERM&lt;/span&gt; to everything left. Without it your service gets no ordered stop, only init&amp;#8217;s blanket &lt;span class="caps"&gt;SIGTERM&lt;/span&gt; and &lt;span class="caps"&gt;SIGKILL&lt;/span&gt; at the very end. For anything that needs an orderly shutdown (databases, queues, stateful applications), treat this keyword as&amp;nbsp;mandatory.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;: ${var:="default"}&lt;/code&gt;&lt;/strong&gt;&amp;nbsp;after &lt;code&gt;load_rc_config&lt;/code&gt; sets a default only if the admin hasn&amp;#8217;t. This is the standard idiom;&amp;nbsp;the &lt;code&gt;:&lt;/code&gt; is the shell&amp;#8217;s no-op command, used only for its side effect of assigning the&amp;nbsp;default.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;required_files&lt;/code&gt;&lt;/strong&gt;&amp;nbsp;makes &lt;code&gt;start&lt;/code&gt; fail fast&amp;nbsp;with &lt;code&gt;webhookd: required file /usr/local/etc/webhookd.yml does not exist&lt;/code&gt; instead of starting a supervisor whose child crash-loops&amp;nbsp;forever.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;start_precmd&lt;/code&gt;&lt;/strong&gt; is where you check preconditions or prepare the environment (create a runtime directory, verify a dependency). If it returns non-zero, the start is&amp;nbsp;aborted.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And don&amp;#8217;t forget the script has to be &lt;strong&gt;executable&lt;/strong&gt;, and it must contain&amp;nbsp;a &lt;code&gt;# PROVIDE:&lt;/code&gt; line. At boot, rc only picks up files&amp;nbsp;in &lt;code&gt;local_startup&lt;/code&gt; that are both. A non-executable script or one&amp;nbsp;without &lt;code&gt;PROVIDE&lt;/code&gt; is silently skipped. No error, no log,&amp;nbsp;nothing.&lt;/p&gt;
&lt;h2 id="case-study-auditing-my-own-scripts"&gt;Case Study: Auditing My Own&amp;nbsp;Scripts&lt;/h2&gt;
&lt;p&gt;Writing a clean example is easy. It&amp;#8217;s more honest to look at what&amp;#8217;s actually running on my servers. I went through the hand-written rc.d scripts in my jails while writing this article, and they make a good tour of everything above: one is nearly perfect, one is a single edit away from the pidfile trap, and one has grown into a small shell program that works around rc.subr instead of using&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="mastodon_web-working-around-rcsubr"&gt;mastodon_web: Working Around&amp;nbsp;rc.subr&lt;/h3&gt;
&lt;p&gt;The &lt;a href="https://hofstede.it/migrating-burningboardnet-mastodon-instance-to-a-multi-jail-freebsd-setup/"&gt;multi-jail Mastodon article&lt;/a&gt; from last December published the first version of this script. Its stop function looked like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mastodon_web_stop&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-9&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon/&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;_supervisor.pid&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-15&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon/&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;.pid&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Look at the order: &lt;span class="caps"&gt;SIGKILL&lt;/span&gt; the supervisor first, so it can&amp;#8217;t restart anything, then &lt;span class="caps"&gt;SIGTERM&lt;/span&gt; the child. That&amp;#8217;s the workaround you end up with when you hit the pidfile trap without knowing what it&amp;nbsp;is. &lt;code&gt;daemon -r&lt;/code&gt; kept bringing puma back, so the supervisor got shot before it could. It does work. But the supervisor dies without cleaning up its pidfiles, and the function returns immediately without waiting for puma to&amp;nbsp;exit.&lt;/p&gt;
&lt;p&gt;The version running in production today has&amp;nbsp;no &lt;code&gt;-r&lt;/code&gt; at all, and a gentler stop function.&amp;nbsp;Why &lt;code&gt;-r&lt;/code&gt; disappeared, I honestly can&amp;#8217;t tell you any more. These scripts are old enough that the reasoning is lost, which is a good argument for writing it down in a comment. Here&amp;#8217;s the core of it, with the addresses replaced by documentation&amp;nbsp;ones:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/local/bin/ruby33&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;procname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/local/bin/ruby33&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;start_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastodon_start&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;stop_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastodon_stop&amp;quot;&lt;/span&gt;

mastodon_start&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/bin/mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon&lt;span class="w"&gt; &lt;/span&gt;/var/log/mastodon
&lt;span class="w"&gt;    &lt;/span&gt;/usr/sbin/chown&lt;span class="w"&gt; &lt;/span&gt;mastodon:mastodon&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon&lt;span class="w"&gt; &lt;/span&gt;/var/log/mastodon

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:~/bin
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;RAILS_ENV&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;production
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;BIND&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;198&lt;/span&gt;.51.100.9
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PORT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;3000&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/home/mastodon/live&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/usr/sbin/daemon&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;mastodon&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;supervisor_pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;/var/log/mastodon/puma.log&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/usr/local/bin/bundle&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;puma&lt;span class="w"&gt; &lt;/span&gt;-C&lt;span class="w"&gt; &lt;/span&gt;config/puma.rb
&lt;span class="o"&gt;}&lt;/span&gt;

mastodon_stop&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;supervisor_pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-TERM&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;supervisor_pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-0&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-TERM&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It runs, and it has done for months. But measured against what rc.subr offers, it has five&amp;nbsp;problems:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;stop&lt;/code&gt; doesn&amp;#8217;t wait.&lt;/strong&gt; rc.subr&amp;#8217;s&amp;nbsp;default &lt;code&gt;stop&lt;/code&gt; sends the signal and then waits for the PIDs to actually exit. A&amp;nbsp;custom &lt;code&gt;stop_cmd&lt;/code&gt; replaces all of that.&amp;nbsp;So &lt;code&gt;service mastodon_web restart&lt;/code&gt; sends &lt;span class="caps"&gt;SIGTERM&lt;/span&gt; and immediately starts a new puma while the old one is still finishing requests and holding port 3000. If the new one loses the race for the port, it exits, and&amp;nbsp;without &lt;code&gt;-r&lt;/code&gt; nothing brings it back. On&amp;nbsp;shutdown, &lt;code&gt;rc.shutdown&lt;/code&gt; similarly moves on before puma is&amp;nbsp;gone.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No supervision.&lt;/strong&gt;&amp;nbsp;Without &lt;code&gt;-r&lt;/code&gt;, a crashed puma or Sidekiq stays dead until someone&amp;nbsp;notices.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;-T&lt;/code&gt; quietly enables&amp;nbsp;syslog.&lt;/strong&gt; &lt;code&gt;-T&lt;/code&gt; sets the &lt;em&gt;syslog tag&lt;/em&gt;, and daemon(8) turns on syslog output whenever any syslog option is given. So every line puma writes goes&amp;nbsp;to &lt;code&gt;/var/log/mastodon/puma.log&lt;/code&gt; &lt;strong&gt;and&lt;/strong&gt; to syslog. I&amp;nbsp;wanted &lt;code&gt;-t&lt;/code&gt;, the process&amp;nbsp;title.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Everything is hard-coded.&lt;/strong&gt; Addresses, ports and environment sit in the script, three times over (web, streaming, sidekiq), instead of in rc.conf where rc.subr would apply&amp;nbsp;them.&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;status&lt;/code&gt; lies.&lt;/strong&gt; While writing this, I ran it on the production jail, with Mastodon happily serving&amp;nbsp;requests:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;text
root@mastodonweb:~ # service mastodon_web status
mastodon_web is not running.&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;The pidfile is the child&amp;#8217;s&amp;nbsp;(&lt;code&gt;-p&lt;/code&gt;)&amp;nbsp;and &lt;code&gt;procname&lt;/code&gt; is &lt;code&gt;/usr/local/bin/ruby33&lt;/code&gt;. rc.subr only believes a pidfile if the process behind that &lt;span class="caps"&gt;PID&lt;/span&gt;&amp;nbsp;matches &lt;code&gt;procname&lt;/code&gt;, and here it doesn&amp;#8217;t, so it concludes nothing is running. That&amp;#8217;s not just cosmetic. The same lookup drives the &amp;#8220;already running?&amp;#8221; check&amp;nbsp;in &lt;code&gt;start&lt;/code&gt;,&amp;nbsp;so &lt;code&gt;service mastodon_web start&lt;/code&gt; on a running system will start a &lt;strong&gt;second&lt;/strong&gt; puma next to the first&amp;nbsp;one.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Here&amp;#8217;s the rewrite. The script now just declares the&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# PROVIDE: mastodon_web&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: DAEMON&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;

.&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.subr

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastodon_web&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;desc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Mastodon Web Service&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rcvar&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastodon_web_enable&amp;quot;&lt;/span&gt;

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;

:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;mastodon_web_enable&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;mastodon_web_chdir&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/home/mastodon/live&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;

&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/mastodon/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/sbin/daemon&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command_args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-r -R 5 -P &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; -t &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; -u mastodon \&lt;/span&gt;
&lt;span class="s2"&gt;    -f -o /var/log/mastodon/puma.log -H \&lt;/span&gt;
&lt;span class="s2"&gt;    /usr/local/bin/bundle exec puma -C config/puma.rb&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;start_precmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastodon_web_precmd&amp;quot;&lt;/span&gt;

mastodon_web_precmd&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;mastodon&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;mastodon&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon&lt;span class="w"&gt; &lt;/span&gt;/var/log/mastodon
&lt;span class="o"&gt;}&lt;/span&gt;

run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And the instance-specific settings move&amp;nbsp;to &lt;code&gt;/etc/rc.conf.d/mastodon_web&lt;/code&gt;, where&amp;nbsp;rc.subr&amp;#8217;s &lt;code&gt;${name}_env&lt;/code&gt; picks them&amp;nbsp;up:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;mastodon_web_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;mastodon_web_env&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;PATH=/usr/local/bin:/usr/bin:/bin \&lt;/span&gt;
&lt;span class="s2"&gt;    RAILS_ENV=production BIND=198.51.100.9 PORT=3000 \&lt;/span&gt;
&lt;span class="s2"&gt;    PROMETHEUS_EXPORTER_LOCAL=false PROMETHEUS_EXPORTER_HOST=2001:db8:9000::100&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There&amp;#8217;s no custom start, stop or status code left. One pidfile, pointing at the supervisor,&amp;nbsp;with &lt;code&gt;procname&lt;/code&gt; defaulting&amp;nbsp;to &lt;code&gt;/usr/sbin/daemon&lt;/code&gt;, which matches.&amp;nbsp;So &lt;code&gt;status&lt;/code&gt; tells the truth&amp;nbsp;and &lt;code&gt;start&lt;/code&gt; refuses to start a second&amp;nbsp;copy. &lt;code&gt;stop&lt;/code&gt; is rc.subr&amp;#8217;s own, so it waits,&amp;nbsp;and &lt;code&gt;restart&lt;/code&gt; no longer&amp;nbsp;races. &lt;code&gt;-r -R 5&lt;/code&gt; brings puma back after a&amp;nbsp;crash. &lt;code&gt;${name}_chdir&lt;/code&gt; replaces&amp;nbsp;the &lt;code&gt;cd&lt;/code&gt;, &lt;code&gt;install -d&lt;/code&gt; replaces&amp;nbsp;the &lt;code&gt;mkdir&lt;/code&gt;/&lt;code&gt;chown&lt;/code&gt; pair, and the streaming and sidekiq scripts become the same twenty lines with a different name and&amp;nbsp;command.&lt;/p&gt;
&lt;p&gt;One more thing from the Sidekiq script: it&amp;nbsp;says &lt;code&gt;REQUIRE: DAEMON postgresql&lt;/code&gt;. But in this setup PostgreSQL runs in the&amp;nbsp;separate &lt;code&gt;database&lt;/code&gt; jail. Inside the Sidekiq jail there is no script that&amp;nbsp;provides &lt;code&gt;postgresql&lt;/code&gt;, so that dependency matches nothing, and nobody complains. rcorder only sees the scripts of the system it&amp;#8217;s running on. &lt;strong&gt;Ordering between jails is the jail manager&amp;#8217;s job&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;depend&lt;/code&gt; in jail.conf(5)), not rc.d&amp;#8217;s. If Sidekiq must not start before the database is reachable, that belongs in&amp;nbsp;a &lt;code&gt;start_precmd&lt;/code&gt; that waits for the&amp;nbsp;port.&lt;/p&gt;
&lt;h3 id="zigbee2mqtt-nearly-right"&gt;zigbee2mqtt: Nearly&amp;nbsp;Right&lt;/h3&gt;
&lt;p&gt;From the &lt;a href="https://hofstede.it/my-home-assistant-setup-bhyve-freebsd-jails-zigbee-homematic-and-matter-over-thread/"&gt;Home Assistant setup&lt;/a&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;zigbee2mqtt_enable&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;zigbee2mqtt_user&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;z2m&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;

&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/zigbee2mqtt/zigbee2mqtt.pid&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/usr/sbin/daemon
&lt;span class="nv"&gt;command_args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-f -P &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; /usr/local/bin/zigbee2mqtt-start&amp;quot;&lt;/span&gt;

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;
run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This one is close to ideal. It uses rc.subr&amp;#8217;s default methods, the pidfile is the supervisor&amp;#8217;s&amp;nbsp;(&lt;code&gt;-P&lt;/code&gt;),&amp;nbsp;and &lt;code&gt;procname&lt;/code&gt; defaults&amp;nbsp;to &lt;code&gt;/usr/sbin/daemon&lt;/code&gt;, which matches. It also uses the &lt;em&gt;other&lt;/em&gt; privilege-dropping approach from&amp;nbsp;earlier: &lt;code&gt;zigbee2mqtt_user&lt;/code&gt; makes&amp;nbsp;rc.subr &lt;code&gt;su&lt;/code&gt; the whole daemon invocation&amp;nbsp;to &lt;code&gt;z2m&lt;/code&gt;, which works because the pidfile lives in a&amp;nbsp;directory &lt;code&gt;z2m&lt;/code&gt; owns.&lt;/p&gt;
&lt;p&gt;The only thing missing is the reason to use daemon(8) in the first&amp;nbsp;place: &lt;code&gt;-r&lt;/code&gt;. Zigbee2MQTT is the service in my house most likely to exit on its own, typically when the &lt;span class="caps"&gt;USB&lt;/span&gt; coordinator hiccups. Because the pidfile is&amp;nbsp;already &lt;code&gt;-P&lt;/code&gt;,&amp;nbsp;adding &lt;code&gt;-r -R 10&lt;/code&gt; is safe,&amp;nbsp;and &lt;code&gt;service zigbee2mqtt stop&lt;/code&gt; will still stop&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;(The defaults are also set &lt;em&gt;before&lt;/em&gt; &lt;code&gt;load_rc_config&lt;/code&gt;. That happens to work, because rc.conf assigns its values unconditionally and overwrites them. Putting the defaults&amp;nbsp;after &lt;code&gt;load_rc_config&lt;/code&gt; is the convention, and it&amp;#8217;s correct whatever the config file&amp;nbsp;does.)&lt;/p&gt;
&lt;h3 id="netbox-one-flag-away-from-the-trap"&gt;netbox: One Flag Away from the&amp;nbsp;Trap&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;procname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/opt/netbox/venv/bin/python3.11&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/sbin/daemon&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command_args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-u netbox -p &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; /opt/netbox/venv/bin/gunicorn \&lt;/span&gt;
&lt;span class="s2"&gt;    --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is correct as it stands. The pidfile is the &lt;strong&gt;child&amp;#8217;s&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;-p&lt;/code&gt;),&amp;nbsp;and &lt;code&gt;procname&lt;/code&gt; is set to the Python interpreter, because the gunicorn master shows up&amp;nbsp;in &lt;code&gt;ps&lt;/code&gt; as &lt;code&gt;/opt/netbox/venv/bin/python3.11 /opt/netbox/venv/bin/gunicorn ...&lt;/code&gt;. &lt;code&gt;stop&lt;/code&gt; sends &lt;span class="caps"&gt;SIGTERM&lt;/span&gt; to gunicorn, gunicorn shuts down gracefully, and daemon, which has nothing to restart, exits&amp;nbsp;too.&lt;/p&gt;
&lt;p&gt;But it&amp;#8217;s exactly one change away from the trap. The obvious improvement is&amp;nbsp;&amp;#8220;add &lt;code&gt;-r&lt;/code&gt; so it restarts on crash&amp;#8221;. Do only that,&amp;nbsp;and &lt;code&gt;service netbox stop&lt;/code&gt; will stop gunicorn and daemon will immediately start it again. The correct change is three edits together:&amp;nbsp;add &lt;code&gt;-r&lt;/code&gt;,&amp;nbsp;switch &lt;code&gt;-p&lt;/code&gt; to &lt;code&gt;-P&lt;/code&gt;, and&amp;nbsp;remove &lt;code&gt;procname&lt;/code&gt; so it defaults to the daemon&amp;nbsp;binary.&lt;/p&gt;
&lt;p&gt;It has a second issue too:&amp;nbsp;no &lt;code&gt;-f&lt;/code&gt;, &lt;code&gt;-o&lt;/code&gt; or &lt;code&gt;-S&lt;/code&gt;. When none of those are given, daemon(8) writes the child&amp;#8217;s output to its own stdout, which at boot is the console. Inside a jail, that means gunicorn&amp;#8217;s error log ends up in the jail&amp;#8217;s console log, or nowhere.&amp;nbsp;Adding &lt;code&gt;-S -T netbox&lt;/code&gt; sends it to syslog&amp;nbsp;instead.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;command_args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-r -R 5 -P &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; -t &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; -S -T &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; -u netbox \&lt;/span&gt;
&lt;span class="s2"&gt;    /opt/netbox/venv/bin/gunicorn \&lt;/span&gt;
&lt;span class="s2"&gt;    --pythonpath /opt/netbox/netbox --config /opt/netbox/gunicorn.py netbox.wsgi&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="the-pattern"&gt;The&amp;nbsp;Pattern&lt;/h3&gt;
&lt;p&gt;Put the three side by side and one rule stands out: &lt;strong&gt;the more a script relies on rc.subr, the fewer bugs it has.&lt;/strong&gt; The script with&amp;nbsp;custom &lt;code&gt;start_cmd&lt;/code&gt; and &lt;code&gt;stop_cmd&lt;/code&gt; had the most problems, because each custom function has to reimplement something rc.subr already does correctly: waiting on stop, checking PIDs, applying the environment. If you find yourself writing&amp;nbsp;a &lt;code&gt;stop_cmd&lt;/code&gt;, check first whether the real problem is a pidfile pointing at the wrong&amp;nbsp;process.&lt;/p&gt;
&lt;p&gt;One last thing about jails: when a jail stops, it runs its&amp;nbsp;own &lt;code&gt;/etc/rc.shutdown&lt;/code&gt;,&amp;nbsp;so &lt;code&gt;KEYWORD: shutdown&lt;/code&gt; matters inside jails just as much as on the host. All three scripts got that&amp;nbsp;right.&lt;/p&gt;
&lt;h2 id="service-jails-isolation-as-an-rcconf-one-liner"&gt;Service Jails: Isolation as an rc.conf&amp;nbsp;One-Liner&lt;/h2&gt;
&lt;p&gt;FreeBSD 15.0 added &lt;strong&gt;service jails&lt;/strong&gt; to the rc framework, and they&amp;#8217;re the most interesting change to rc.d in years.&amp;nbsp;Set &lt;code&gt;${name}_svcj=YES&lt;/code&gt; and rc.subr starts the service inside a dedicated, throwaway jail&amp;nbsp;named &lt;code&gt;svcj-&amp;lt;name&amp;gt;&lt;/code&gt;, without you writing a jail.conf&amp;nbsp;entry:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;webhookd_svcj&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;webhookd_svcj_options&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;net_basic&amp;quot;&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;webhookd&lt;span class="w"&gt; &lt;/span&gt;restart
jls&lt;span class="w"&gt; &lt;/span&gt;name&lt;span class="w"&gt; &lt;/span&gt;path
&lt;span class="c1"&gt;# svcj-webhookd  /&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;_svcj_options&lt;/code&gt; grants specific&amp;nbsp;capabilities: &lt;code&gt;net_basic&lt;/code&gt; (IPv4/IPv6 with reserved&amp;nbsp;ports), &lt;code&gt;netv4&lt;/code&gt;, &lt;code&gt;netv6&lt;/code&gt;, &lt;code&gt;net_raw&lt;/code&gt;, &lt;code&gt;net_all&lt;/code&gt;, &lt;code&gt;sysvipc&lt;/code&gt;, &lt;code&gt;sysvipcnew&lt;/code&gt;, &lt;code&gt;mlock&lt;/code&gt;, &lt;code&gt;routing&lt;/code&gt;, &lt;code&gt;settime&lt;/code&gt;, &lt;code&gt;vmm&lt;/code&gt; and &lt;code&gt;nfsd&lt;/code&gt;. By default the service gets none of&amp;nbsp;them. &lt;code&gt;${name}_svcj_ipaddrs&lt;/code&gt; restricts it to specific&amp;nbsp;addresses.&lt;/p&gt;
&lt;p&gt;Be clear about what this does and doesn&amp;#8217;t buy you. A service jail&amp;nbsp;uses &lt;code&gt;path=/&lt;/code&gt;: it &lt;strong&gt;shares the host&amp;#8217;s filesystem&lt;/strong&gt;. It does not&amp;nbsp;hide &lt;code&gt;/etc&lt;/code&gt; or &lt;code&gt;/home&lt;/code&gt; from the service. What it does isolate is the process table (the service can&amp;#8217;t see or signal other processes) and the network privileges it&amp;#8217;s allowed. SysV &lt;span class="caps"&gt;IPC&lt;/span&gt; is off unless you ask for&amp;nbsp;it: &lt;code&gt;sysvipc&lt;/code&gt; shares the host&amp;#8217;s &lt;span class="caps"&gt;IPC&lt;/span&gt; namespace,&amp;nbsp;while &lt;code&gt;sysvipcnew&lt;/code&gt; gives the service a separate one of its own. Think of it as a cheap reduction in what a compromised daemon can do, not as a replacement for the full jails from the &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;first article&lt;/a&gt;. That&amp;#8217;s exactly why&amp;nbsp;base &lt;code&gt;sshd&lt;/code&gt; opts out&amp;nbsp;of &lt;code&gt;svcj_all_enable&lt;/code&gt; by default&amp;nbsp;(&lt;code&gt;: ${sshd_svcj:="NO"}&lt;/code&gt;): an sshd that can&amp;#8217;t see other processes isn&amp;#8217;t much use as a login service. You can still put it in a service jail explicitly&amp;nbsp;with &lt;code&gt;sshd_svcj=YES&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The framework is also still under active development. At the end of August 2026 a batch of fixes landed&amp;nbsp;in &lt;code&gt;main&lt;/code&gt;, and they were merged&amp;nbsp;to &lt;code&gt;stable/15&lt;/code&gt; on 6 September. They include&amp;nbsp;making &lt;code&gt;svcj_all_enable&lt;/code&gt; actually work, running a script&amp;#8217;s&amp;nbsp;own &lt;code&gt;restart_cmd&lt;/code&gt; and &lt;code&gt;status_cmd&lt;/code&gt; inside its service jail, and sending stop and reload signals from inside the jail. That last one matters if you&amp;nbsp;combine &lt;code&gt;${name}_svcj&lt;/code&gt; with &lt;code&gt;${name}_user&lt;/code&gt;: before the fix, stop and reload could fail and leave both the service and its jail running. None of this is in 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;. If you rely on service jails, test stop, reload, and any custom status or restart methods on the exact FreeBSD build you&amp;nbsp;deploy.&lt;/p&gt;
&lt;p&gt;For services that just listen on a port and talk to a database, adding one rc.conf line costs nothing, and it&amp;#8217;s worth&amp;nbsp;doing.&lt;/p&gt;
&lt;h2 id="debugging-rcd"&gt;Debugging&amp;nbsp;rc.d&lt;/h2&gt;
&lt;p&gt;When a service won&amp;#8217;t start and the error message isn&amp;#8217;t&amp;nbsp;helpful:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# 1. Does rc.d even think it&amp;#39;s enabled, and under which variable?&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;webhookd&lt;span class="w"&gt; &lt;/span&gt;rcvar

&lt;span class="c1"&gt;# 2. Trace the script under the same environment service(8) uses&lt;/span&gt;
env&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;-L&lt;span class="w"&gt; &lt;/span&gt;-/daemon&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;HOME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/sbin:/bin:/usr/sbin:/usr/bin&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/bin/sh&lt;span class="w"&gt; &lt;/span&gt;-x&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/rc.d/webhookd&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;less
&lt;span class="c1"&gt;#    (sh -x shows every expansion; you&amp;#39;ll see the exact command line rc.subr builds)&lt;/span&gt;

&lt;span class="c1"&gt;# 3. Turn on rc.subr&amp;#39;s own debug output, for this run only&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;-E&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;rc_debug&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES&lt;span class="w"&gt; &lt;/span&gt;webhookd&lt;span class="w"&gt; &lt;/span&gt;start

&lt;span class="c1"&gt;# 4. Same, but for the whole boot: set it, reboot, read the console/messages&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;rc_debug&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# remember to remove it afterwards&lt;/span&gt;

&lt;span class="c1"&gt;# 5. Where does it land in the boot order?&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;webhookd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Step 2 is exactly&amp;nbsp;what &lt;code&gt;service&lt;/code&gt; does internally,&amp;nbsp;plus &lt;code&gt;sh -x&lt;/code&gt;. Running&amp;nbsp;plain &lt;code&gt;sh -x&lt;/code&gt; from your shell is fine for debugging shell expansion, but it traces the script with &lt;em&gt;your&lt;/em&gt; environment, which is the thing we just said not to trust. Step 2 resolves most problems on its own, because it shows you the exact command line that rc.subr builds, including&amp;nbsp;every &lt;code&gt;_flags&lt;/code&gt;, &lt;code&gt;_env&lt;/code&gt; and &lt;code&gt;_user&lt;/code&gt; expansion. When the generated command is&amp;nbsp;wrong, &lt;code&gt;sh -x&lt;/code&gt; shows you how it got that&amp;nbsp;way.&lt;/p&gt;
&lt;p&gt;For boot-time-only failures (works&amp;nbsp;with &lt;code&gt;service start&lt;/code&gt;, fails at boot), the cause is almost always ordering. The service starts before something it needs: a network address that isn&amp;#8217;t configured yet, a &lt;span class="caps"&gt;ZFS&lt;/span&gt; dataset that isn&amp;#8217;t mounted, a database that&amp;#8217;s still starting.&amp;nbsp;Check &lt;code&gt;service -r&lt;/code&gt;,&amp;nbsp;fix &lt;code&gt;REQUIRE&lt;/code&gt;, and if the dependency takes a while to become &lt;em&gt;ready&lt;/em&gt; (not just &lt;em&gt;started&lt;/em&gt;), wait for it&amp;nbsp;in &lt;code&gt;start_precmd&lt;/code&gt;. For network readiness specifically, base&amp;nbsp;has &lt;code&gt;netwait&lt;/code&gt; (&lt;code&gt;netwait_enable&lt;/code&gt;, &lt;code&gt;netwait_ip&lt;/code&gt;, &lt;code&gt;netwait_if&lt;/code&gt; in rc.conf), which blocks boot until an address answers ping or an interface has&amp;nbsp;link.&lt;/p&gt;
&lt;h2 id="coming-from-systemd"&gt;Coming From&amp;nbsp;systemd&lt;/h2&gt;
&lt;p&gt;Many readers of this blog run Linux, mostly &lt;span class="caps"&gt;RHEL&lt;/span&gt; and Fedora, and I do too, as the &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman deep dive&lt;/a&gt; shows. So here&amp;#8217;s the mapping, without the flame&amp;nbsp;war:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;systemd&lt;/th&gt;
&lt;th&gt;rc.d&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Unit file&amp;nbsp;in &lt;code&gt;/etc/systemd/system/&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Script&amp;nbsp;in &lt;code&gt;/usr/local/etc/rc.d/&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;systemctl enable --now foo&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;service foo enable &amp;amp;&amp;amp; service foo start&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;systemctl status foo&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;service foo status&lt;/code&gt; + syslog&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Drop-in &lt;code&gt;foo.service.d/override.conf&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/etc/rc.conf.d/foo&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;After=&lt;/code&gt; / &lt;code&gt;Before=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;REQUIRE:&lt;/code&gt; / &lt;code&gt;BEFORE:&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Requires=&lt;/code&gt; / &lt;code&gt;Wants=&lt;/code&gt; (activation)&lt;/td&gt;
&lt;td&gt;No equivalent; ordering only&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Restart=on-failure&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;daemon -r&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;User=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;${name}_user&lt;/code&gt; or &lt;code&gt;daemon -u&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Environment=&lt;/code&gt; / &lt;code&gt;EnvironmentFile=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;${name}_env&lt;/code&gt; / &lt;code&gt;${name}_env_file&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Nice=&lt;/code&gt;, &lt;code&gt;LimitNOFILE=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;${name}_nice&lt;/code&gt;, &lt;code&gt;${name}_limits&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;StandardOutput=journal&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;daemon -S&lt;/code&gt; → syslog (a rough equivalent; journald does far more)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sandboxing&amp;nbsp;(&lt;code&gt;PrivateTmp=&lt;/code&gt;, &lt;code&gt;ProtectSystem=&lt;/code&gt;&amp;#8230;)&lt;/td&gt;
&lt;td&gt;Service jails (narrower), or a full jail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;cgroup resource control&lt;/td&gt;
&lt;td&gt;&lt;a href="https://hofstede.it/freebsd-resource-monitoring-accounting-and-troubleshooting/"&gt;&lt;code&gt;rctl&lt;/code&gt;&lt;/a&gt; per jail/user/process&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Socket activation, timers, parallel start&lt;/td&gt;
&lt;td&gt;&lt;code&gt;inetd&lt;/code&gt;, &lt;code&gt;cron&lt;/code&gt;, sequential start&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;They&amp;#8217;re built on different philosophies, and each has real strengths. systemd is a supervisor with a dependency graph, parallel startup, and a lot of sandboxing and resource control built in. rc.d is a sequential, ordered set of shell scripts with a shared library,&amp;nbsp;plus &lt;code&gt;daemon(8)&lt;/code&gt; when you need supervision. rc.d boots more slowly on a machine with hundreds of services, and it has nothing like systemd&amp;#8217;s built-in sandboxing knobs. In exchange, the whole thing is a few files of shell you can read and trace&amp;nbsp;with &lt;code&gt;sh -x&lt;/code&gt;, and a server with fifteen services, which is most of mine, starts in seconds&amp;nbsp;anyway.&lt;/p&gt;
&lt;h2 id="common-pitfalls-the-checklist"&gt;Common Pitfalls: The&amp;nbsp;Checklist&lt;/h2&gt;
&lt;p&gt;Everything below is covered above. This section is the short version, for when you come back to this article with a broken&amp;nbsp;script.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;daemon -r&lt;/code&gt; with &lt;code&gt;-p&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;-P&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;service stop&lt;/code&gt; kills the child, the supervisor restarts it, and stop &amp;#8220;succeeds&amp;#8221; while the service keeps running.&amp;nbsp;With &lt;code&gt;-r&lt;/code&gt;, the rc.d pidfile must be the supervisor&amp;#8217;s&amp;nbsp;(&lt;code&gt;-P&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Application arguments&amp;nbsp;in &lt;code&gt;${name}_flags&lt;/code&gt; for daemon-wrapped services.&lt;/strong&gt; They end up as arguments&amp;nbsp;to &lt;code&gt;daemon&lt;/code&gt;, not the application. Use a&amp;nbsp;custom &lt;code&gt;${name}_args&lt;/code&gt; variable at the end&amp;nbsp;of &lt;code&gt;command_args&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Running&amp;nbsp;both &lt;code&gt;${name}_user&lt;/code&gt; and &lt;code&gt;daemon -u&lt;/code&gt;.&lt;/strong&gt; The supervisor&amp;nbsp;gets &lt;code&gt;su&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;d to an unprivileged user, can&amp;#8217;t write its pidfile, and can&amp;#8217;t drop privileges again. Pick one&amp;nbsp;mechanism.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A&amp;nbsp;custom &lt;code&gt;stop_cmd&lt;/code&gt; that doesn&amp;#8217;t wait.&lt;/strong&gt; rc.subr&amp;#8217;s default stop waits for the process to exit; your replacement probably doesn&amp;#8217;t, and&amp;nbsp;then &lt;code&gt;restart&lt;/code&gt; races the old process for its port. Usually the fix is to delete the custom function and correct the&amp;nbsp;pidfile.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;daemon -T&lt;/code&gt; when you&amp;nbsp;meant &lt;code&gt;-t&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;-T&lt;/code&gt; is the syslog tag and implicitly turns on syslog output, so logs get duplicated into&amp;nbsp;syslog. &lt;code&gt;-t&lt;/code&gt; is the process&amp;nbsp;title.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;REQUIRE&lt;/code&gt; across jails.&lt;/strong&gt; rcorder only sees scripts on its own system.&amp;nbsp;A &lt;code&gt;REQUIRE: postgresql&lt;/code&gt; in an application jail does nothing if PostgreSQL runs in another&amp;nbsp;jail.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Missing &lt;code&gt;KEYWORD: shutdown&lt;/code&gt;.&lt;/strong&gt; The service gets no ordered stop. On shutdown it just receives init&amp;#8217;s &lt;span class="caps"&gt;SIGTERM&lt;/span&gt; along with everything else, and a &lt;span class="caps"&gt;SIGKILL&lt;/span&gt; shortly afterwards. Databases do not appreciate&amp;nbsp;this.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Assuming &lt;code&gt;REQUIRE&lt;/code&gt; starts dependencies.&lt;/strong&gt; It only orders. An un-enabled or failed dependency is silently skipped, and your service starts&amp;nbsp;anyway.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Testing by running the script from your shell.&lt;/strong&gt; Your shell has a&amp;nbsp;full &lt;code&gt;PATH&lt;/code&gt; and environment; the service at boot doesn&amp;#8217;t. Test&amp;nbsp;with &lt;code&gt;service&lt;/code&gt;, which reproduces the boot environment. Use absolute paths for everything in a&amp;nbsp;script.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Non-executable script, or&amp;nbsp;no &lt;code&gt;# PROVIDE:&lt;/code&gt; line.&lt;/strong&gt; Silently ignored at&amp;nbsp;boot. &lt;code&gt;service -r | grep yourservice&lt;/code&gt; tells you immediately whether rc can see&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Editing scripts&amp;nbsp;in &lt;code&gt;/etc/rc.d&lt;/code&gt;.&lt;/strong&gt; These belong to the base system and are replaced on upgrade (and with pkgbase, they&amp;#8217;re files owned by a package). Change behaviour through rc.conf&amp;nbsp;variables, &lt;code&gt;/etc/rc.conf.d/&amp;lt;name&amp;gt;&lt;/code&gt;,&amp;nbsp;or &lt;code&gt;${name}_prepend&lt;/code&gt;, not by editing the script. Same rule as never&amp;nbsp;editing &lt;code&gt;/etc/defaults/rc.conf&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A slow-stopping service on&amp;nbsp;shutdown.&lt;/strong&gt; &lt;code&gt;rc.shutdown&lt;/code&gt; has a global 90-second watchdog&amp;nbsp;(&lt;code&gt;rcshutdown_timeout&lt;/code&gt;). If a database needs longer to flush, raise the timeout, or the stop gets killed halfway&amp;nbsp;through.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A typo&amp;nbsp;in &lt;code&gt;rc.conf&lt;/code&gt;.&lt;/strong&gt; It&amp;#8217;s a shell script sourced by everything.&amp;nbsp;Use &lt;code&gt;sysrc&lt;/code&gt;, and if you edited by&amp;nbsp;hand, &lt;code&gt;sh -n /etc/rc.conf&lt;/code&gt; before&amp;nbsp;rebooting.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The rc.d system is small enough to understand completely, and that&amp;#8217;s its biggest&amp;nbsp;strength. &lt;code&gt;init&lt;/code&gt; runs &lt;code&gt;/etc/rc&lt;/code&gt;. &lt;code&gt;/etc/rc&lt;/code&gt; asks &lt;code&gt;rcorder&lt;/code&gt; to sort scripts by&amp;nbsp;their &lt;code&gt;PROVIDE&lt;/code&gt;/&lt;code&gt;REQUIRE&lt;/code&gt; comments and runs them in two passes, base first&amp;nbsp;and &lt;code&gt;/usr/local&lt;/code&gt; after the filesystems are mounted. Every script sources the same layered shell configuration, declares a few variables, and&amp;nbsp;lets &lt;code&gt;rc.subr&lt;/code&gt; do the work. Shutdown is the same list, filtered&amp;nbsp;by &lt;code&gt;KEYWORD: shutdown&lt;/code&gt; and run in&amp;nbsp;reverse.&lt;/p&gt;
&lt;p&gt;The mental model to carry away: &lt;strong&gt;an rc.d script declares a service; it doesn&amp;#8217;t implement one.&lt;/strong&gt;&amp;nbsp;Declare &lt;code&gt;name&lt;/code&gt;, &lt;code&gt;rcvar&lt;/code&gt;, &lt;code&gt;command&lt;/code&gt; and &lt;code&gt;pidfile&lt;/code&gt;, add hooks only where you need behaviour, and every admin knob&amp;nbsp;(&lt;code&gt;_user&lt;/code&gt;, &lt;code&gt;_env&lt;/code&gt;, &lt;code&gt;_fib&lt;/code&gt;, &lt;code&gt;_nice&lt;/code&gt;, &lt;code&gt;_svcj&lt;/code&gt;) works for free. When you need&amp;nbsp;supervision, &lt;code&gt;daemon(8)&lt;/code&gt; provides it, and as long as the pidfile points at the supervisor&amp;nbsp;(&lt;code&gt;-P&lt;/code&gt;), start and stop behave as&amp;nbsp;expected.&lt;/p&gt;
&lt;p&gt;If you take four habits from this article: test&amp;nbsp;with &lt;code&gt;service&lt;/code&gt;, never by running the script directly;&amp;nbsp;use &lt;code&gt;-P&lt;/code&gt; whenever you&amp;nbsp;use &lt;code&gt;daemon -r&lt;/code&gt;;&amp;nbsp;put &lt;code&gt;KEYWORD: shutdown&lt;/code&gt; on anything that holds state; and when in&amp;nbsp;doubt, &lt;code&gt;sh -x&lt;/code&gt; the script and read what rc.subr actually built. It&amp;#8217;s all shell. You can always just read&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The next article in this series is the one I promised last time: the &lt;strong&gt;base system versus ports and packages&lt;/strong&gt;, including where the software comes from, why base and third-party software are deliberately separate, and&amp;nbsp;how &lt;code&gt;freebsd-update&lt;/code&gt;,&amp;nbsp;pkgbase, &lt;code&gt;pkg&lt;/code&gt; and &lt;code&gt;poudriere&lt;/code&gt; fit&amp;nbsp;together.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/config/#configtuning-rcd"&gt;FreeBSD Handbook - Managing Services in&amp;nbsp;FreeBSD&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/articles/rc-scripting/"&gt;Practical rc.d scripting in &lt;span class="caps"&gt;BSD&lt;/span&gt;&lt;/a&gt; - the canonical&amp;nbsp;tutorial&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=rc&amp;amp;sektion=8"&gt;rc(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=rc.subr&amp;amp;sektion=8"&gt;rc.subr(8) man page&lt;/a&gt; - every&amp;nbsp;variable &lt;code&gt;run_rc_command&lt;/code&gt; understands&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=rc.conf&amp;amp;sektion=5"&gt;rc.conf(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=rcorder&amp;amp;sektion=8"&gt;rcorder(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=service&amp;amp;sektion=8"&gt;service(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=sysrc&amp;amp;sektion=8"&gt;sysrc(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=daemon&amp;amp;sektion=8"&gt;daemon(8) man page&lt;/a&gt; - see the note&amp;nbsp;on &lt;code&gt;--supervisor-pidfile&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/#service-jails"&gt;FreeBSD Handbook - Service&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="rc.d"/><category term="rc.subr"/><category term="rcorder"/><category term="rc.conf"/><category term="sysrc"/><category term="service"/><category term="daemon"/><category term="init"/><category term="service-jails"/><category term="jails"/></entry><entry><title>My Ansible Plugin Had a Jail Escape: CVE-2026-55074</title><link href="https://hofstede.it/my-ansible-plugin-had-a-jail-escape-cve-2026-55074/" rel="alternate"/><published>2026-09-27T00:00:00+02:00</published><updated>2026-09-27T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-09-27:/my-ansible-plugin-had-a-jail-escape-cve-2026-55074/</id><summary type="html">&lt;p&gt;For almost a year, my jailexec connection plugin let a compromised FreeBSD jail redirect a root-owned write onto the jail host. The input validation was fine. The problem was on which side of the jail boundary the paths were resolved. This is the story of the bug, the fix in 2.0.0, and the &lt;span class="caps"&gt;CVE&lt;/span&gt; that came with&amp;nbsp;it.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A glowing file inside a steel jail cage travels through a pipe in the cage wall and lands in a folder sunk into the host server below" src="https://hofstede.it/images/2026-09-27-jailexec-cve-2026-55074.webp" title="Whatever happens in the jail, stays in the jail. Unless root on the host moves it out for you."&gt;&lt;/p&gt;
&lt;p&gt;At the end of last year I &lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;wrote about &lt;strong&gt;jailexec&lt;/strong&gt;&lt;/a&gt;, my Ansible connection plugin that manages FreeBSD jails by SSHing to the jail host and running everything&amp;nbsp;through &lt;code&gt;jexec&lt;/code&gt;. That article has a section called &lt;strong&gt;Security Design&lt;/strong&gt;. It proudly explains the two-stage file transfer: upload to a temporary location on the host, then move it into the jail with privilege&amp;nbsp;escalation.&lt;/p&gt;
&lt;p&gt;That two-stage file transfer was a jail&amp;nbsp;escape.&lt;/p&gt;
&lt;p&gt;Every release before 2.0.0 let a process with root inside a managed jail turn a routine Ansible file transfer into a &lt;strong&gt;root-owned write to an arbitrary directory on the jail host&lt;/strong&gt;. It is now tracked as &lt;a href="https://github.com/chofstede/ansible_jailexec/security/advisories/GHSA-cxgv-hp74-jj7r"&gt;&lt;span class="caps"&gt;CVE&lt;/span&gt;-2026-55074&lt;/a&gt; (&lt;span class="caps"&gt;GHSA&lt;/span&gt;-cxgv-hp74-jj7r). It was fixed in 2.0.0 on&amp;nbsp;2026-06-10.&lt;/p&gt;
&lt;p&gt;This article explains what went wrong, why the input validation I was quite pleased with did not help at all, and what the release and disclosure looked like for a small one-person&amp;nbsp;project.&lt;/p&gt;
&lt;p&gt;To be clear about scope: this was a bug in &lt;strong&gt;my&lt;/strong&gt; third-party plugin, not in ansible-core, the&amp;nbsp;built-in &lt;code&gt;ssh&lt;/code&gt; connection plugin, or FreeBSD jails. Both did exactly what they were told. I told them the wrong&amp;nbsp;thing.&lt;/p&gt;
&lt;h2 id="tldr"&gt;&lt;span class="caps"&gt;TL&lt;/span&gt;;&lt;span class="caps"&gt;DR&lt;/span&gt;&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;span class="caps"&gt;CVE&lt;/span&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;CVE&lt;/span&gt;-2026-55074&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Advisory&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/chofstede/ansible_jailexec/security/advisories/GHSA-cxgv-hp74-jj7r"&gt;&lt;span class="caps"&gt;GHSA&lt;/span&gt;-cxgv-hp74-jj7r&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Package&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ansible-jailexec&lt;/code&gt; (PyPI),&amp;nbsp;or &lt;code&gt;jailexec.py&lt;/code&gt; copied by hand&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Affected&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;all&amp;nbsp;versions &lt;code&gt;&amp;lt; 2.0.0&lt;/code&gt; (1.0.0 through 1.3.0)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Fixed&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;2.0.0&lt;/code&gt; (current: &lt;code&gt;2.0.2&lt;/code&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Weakness&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;CWE&lt;/span&gt;-59, link following&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Severity&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;High. My own &lt;span class="caps"&gt;CVSS&lt;/span&gt; 3.1&amp;nbsp;estimate: &lt;code&gt;AV:N/AC:H/PR:H/UI:N/S:C/C:H/I:H/A:H&lt;/code&gt; (8.0)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;If you use jailexec in any form,&amp;nbsp;upgrade:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pip&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--upgrade&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ansible-jailexec&amp;gt;=2.0.0&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you&amp;nbsp;copied &lt;code&gt;jailexec.py&lt;/code&gt; into&amp;nbsp;a &lt;code&gt;connection_plugins/&lt;/code&gt; directory as described in my original article, replace it with the current version from &lt;a href="https://codeberg.org/Larvitz/ansible_jailexec"&gt;Codeberg&lt;/a&gt; or &lt;a href="https://github.com/chofstede/ansible_jailexec"&gt;GitHub&lt;/a&gt;. There is no configuration workaround for older&amp;nbsp;versions.&lt;/p&gt;
&lt;h2 id="how-transfers-used-to-work"&gt;How transfers used to&amp;nbsp;work&lt;/h2&gt;
&lt;p&gt;Commands were never the&amp;nbsp;problem. &lt;code&gt;exec_command&lt;/code&gt; always ran&amp;nbsp;as &lt;code&gt;doas jexec &amp;lt;jail&amp;gt; /bin/sh -c '...'&lt;/code&gt;, so the shell and everything it touched lived inside the&amp;nbsp;jail.&lt;/p&gt;
&lt;p&gt;File transfers were different. Here&amp;nbsp;is &lt;code&gt;put_file&lt;/code&gt; from 1.3.0, lightly&amp;nbsp;trimmed:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;_jail_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="sd"&gt;&amp;quot;&amp;quot;&amp;quot;Map a path inside the jail to its absolute path on the host.&amp;quot;&amp;quot;&amp;quot;&lt;/span&gt;
    &lt;span class="n"&gt;ensure_no_traversal&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;root&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_resolve_jail_root&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;posixpath&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;normpath&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;posixpath&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;

&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;put_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;in_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out_path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;dest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_jail_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out_path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;dest_dir&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;posixpath&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dirname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;staged&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;posixpath&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;STAGING_DIR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;STAGING_PREFIX&lt;/span&gt;&lt;span class="si"&gt;}{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;urandom&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hex&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;put_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;in_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;staged&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;prefix&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_privesc_argv&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;move&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;_shelljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;mkdir&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;-p&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dest_dir&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; &amp;amp;&amp;amp; &amp;quot;&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;_shelljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;mv&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;staged&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stderr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exec_command&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;move&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The plugin&amp;nbsp;asked &lt;code&gt;jls&lt;/code&gt; for the jail&amp;#8217;s root directory&amp;nbsp;(say &lt;code&gt;/usr/local/bastille/jails/web/root&lt;/code&gt;), glued the in-jail destination onto it, uploaded the file&amp;nbsp;to &lt;code&gt;/tmp&lt;/code&gt; on the host, and then ran, &lt;strong&gt;on the host, as root&lt;/strong&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;doas&lt;span class="w"&gt; &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bastille/jails/web/root/usr/local/etc/nginx
doas&lt;span class="w"&gt; &lt;/span&gt;mv&lt;span class="w"&gt; &lt;/span&gt;/tmp/ansible-jailexec-3f9a...&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bastille/jails/web/root/usr/local/etc/nginx/nginx.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It looks reasonable. The path starts with the jail&amp;nbsp;root, &lt;code&gt;..&lt;/code&gt; components are rejected, every argument&amp;nbsp;is &lt;code&gt;shlex.quote&lt;/code&gt;d. What could go&amp;nbsp;wrong?&lt;/p&gt;
&lt;h2 id="symlinks-do-not-care-where-you-meant-to-be"&gt;Symlinks do not care where you meant to&amp;nbsp;be&lt;/h2&gt;
&lt;p&gt;A symbolic link is resolved by whoever walks the path, relative to &lt;strong&gt;that process&amp;#8217;s&lt;/strong&gt; root&amp;nbsp;directory.&lt;/p&gt;
&lt;p&gt;Inside a jail, the root directory is the jail&amp;#8217;s root. A&amp;nbsp;symlink &lt;code&gt;/usr/local/etc -&amp;gt; /etc&lt;/code&gt; created inside the jail points to the jail&amp;#8217;s&amp;nbsp;own &lt;code&gt;/etc&lt;/code&gt; when the jail looks at&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The host has a different root directory. When root on the host&amp;nbsp;walks &lt;code&gt;/usr/local/bastille/jails/web/root/usr/local/etc/...&lt;/code&gt; and hits that same symlink, the absolute&amp;nbsp;target &lt;code&gt;/etc&lt;/code&gt; means the &lt;strong&gt;host&amp;#8217;s&lt;/strong&gt; &lt;code&gt;/etc&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;So the attack needed nothing&amp;nbsp;clever:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Get root inside a jail managed by jailexec (a vulnerable web application, a compromised package, a tenant you do not fully&amp;nbsp;trust).&lt;/li&gt;
&lt;li&gt;Replace a directory that Ansible will later write into with an absolute symlink to a host&amp;nbsp;directory.&lt;/li&gt;
&lt;li&gt;Wait for the next playbook&amp;nbsp;run.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The&amp;nbsp;host-side &lt;code&gt;mkdir -p&lt;/code&gt; follows the symlink. The&amp;nbsp;host-side &lt;code&gt;mv&lt;/code&gt; follows it too. The file lands in a host directory of the attacker&amp;#8217;s choosing, owned by root. The filename and content come from the playbook, the location comes from the attacker. From there, getting to a full host compromise is a matter of patience and picking the right&amp;nbsp;directory.&lt;/p&gt;
&lt;p&gt;That is exactly the guarantee a jail is supposed to provide, and exactly the guarantee my plugin was supposed to preserve: whatever happens in the jail, stays in the&amp;nbsp;jail.&lt;/p&gt;
&lt;h2 id="why-the-validation-did-not-help"&gt;Why the validation did not&amp;nbsp;help&lt;/h2&gt;
&lt;p&gt;I had input validation. Jail names were matched against a strict regex. Paths&amp;nbsp;containing &lt;code&gt;..&lt;/code&gt; were rejected. Everything crossing the &lt;span class="caps"&gt;SSH&lt;/span&gt; wire was&amp;nbsp;quoted.&lt;/p&gt;
&lt;p&gt;All of that checks the &lt;strong&gt;string&lt;/strong&gt;. None of it checks what the &lt;strong&gt;filesystem&lt;/strong&gt; does with the&amp;nbsp;string. &lt;code&gt;/usr/local/etc/nginx/nginx.conf&lt;/code&gt; is a perfectly clean path. It just happened to contain a directory that the jail had replaced with a&amp;nbsp;trapdoor.&lt;/p&gt;
&lt;p&gt;You cannot fix this with a better regex. Checking for symlinks before&amp;nbsp;the &lt;code&gt;mv&lt;/code&gt; is a race condition (the jail can swap the directory between your check and your&amp;nbsp;write). &lt;code&gt;realpath&lt;/code&gt; plus a prefix check has the same problem. The only robust fix is to make sure the path is resolved by a process that cannot see the host in the first&amp;nbsp;place.&lt;/p&gt;
&lt;h2 id="which-tasks-were-affected"&gt;Which tasks were&amp;nbsp;affected&lt;/h2&gt;
&lt;p&gt;The advisory&amp;nbsp;lists &lt;code&gt;copy&lt;/code&gt;, &lt;code&gt;template&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;fetch&lt;/code&gt; style tasks, anything that&amp;nbsp;calls &lt;code&gt;put_file&lt;/code&gt;. It is worth knowing that Ansible also&amp;nbsp;uses &lt;code&gt;put_file&lt;/code&gt; to upload the module payload itself into the remote temporary directory unless pipelining is enabled. &amp;#8220;I never copy files into my jails&amp;#8221; was therefore not a reliable reason to&amp;nbsp;relax.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;fetch_file&lt;/code&gt; had a related, milder flaw: it pulled the host-side path over &lt;span class="caps"&gt;SFTP&lt;/span&gt; as the unprivileged &lt;span class="caps"&gt;SSH&lt;/span&gt; user. A symlink inside the jail could point that read at a host file readable by the &lt;span class="caps"&gt;SSH&lt;/span&gt; user. The read was never privileged, but it was still resolved on the wrong side of the boundary, and 2.0.0 fixes it the same&amp;nbsp;way.&lt;/p&gt;
&lt;h2 id="the-fix-resolve-paths-inside-the-jail"&gt;The fix: resolve paths inside the&amp;nbsp;jail&lt;/h2&gt;
&lt;p&gt;Version 2.0.0 stops touching the jail&amp;#8217;s filesystem from the host entirely. The staged file is streamed into the jail over stdin, and a shell running &lt;strong&gt;inside&lt;/strong&gt; the jail does the&amp;nbsp;writing:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;put_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;in_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;out_path&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;dest&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_in_jail_path&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;out_path&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;staged&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_staging_path&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

    &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;put_file&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;in_path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;staged&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="c1"&gt;# The write happens *inside* the jail: jexec confines mkdir/cat to&lt;/span&gt;
    &lt;span class="c1"&gt;# the jail&amp;#39;s root, so a symlink planted inside the jail cannot&lt;/span&gt;
    &lt;span class="c1"&gt;# redirect a privileged write onto a host path.&lt;/span&gt;
    &lt;span class="n"&gt;inner&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mkdir -p &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;shlex&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;posixpath&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;dirname&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; &amp;amp;&amp;amp; &amp;quot;&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cat &amp;gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;shlex&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;transfer&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;_shelljoin&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;_jexec_argv&lt;/span&gt;&lt;span class="p"&gt;(),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/bin/sh&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;-c&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inner&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot; &amp;lt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;shlex&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;staged&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; &amp;amp;&amp;amp; rm -f &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;shlex&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;staged&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;rc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;stderr&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;super&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exec_command&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;transfer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;sudoable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;False&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On the host, this becomes&amp;nbsp;roughly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;doas&lt;span class="w"&gt; &lt;/span&gt;jexec&lt;span class="w"&gt; &lt;/span&gt;web&lt;span class="w"&gt; &lt;/span&gt;/bin/sh&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;mkdir -p /usr/local/etc/nginx &amp;amp;&amp;amp; cat &amp;gt; /usr/local/etc/nginx/nginx.conf&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;/tmp/ansible-jailexec-3f9a...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The staged file is opened on the host by the unprivileged &lt;span class="caps"&gt;SSH&lt;/span&gt; login shell and handed over as stdin. Everything after that, including every symlink lookup, happens inside the jail. A malicious symlink can now redirect the write to another place &lt;strong&gt;inside the same jail&lt;/strong&gt;, which the jail&amp;#8217;s root could already write to anyway. Nothing&amp;nbsp;gained.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;fetch_file&lt;/code&gt; is the mirror&amp;nbsp;image: &lt;code&gt;jexec &amp;lt;jail&amp;gt; /bin/sh -c 'cat &amp;lt; src'&lt;/code&gt; writes into a staging file on the host (created&amp;nbsp;with &lt;code&gt;umask 077&lt;/code&gt;), which is then pulled over &lt;span class="caps"&gt;SFTP&lt;/span&gt; and&amp;nbsp;removed.&lt;/p&gt;
&lt;h2 id="what-changed-for-users"&gt;What changed for&amp;nbsp;users&lt;/h2&gt;
&lt;p&gt;Moving the transfer into the jail had consequences, which is why this became a major&amp;nbsp;release:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Transfers run&amp;nbsp;as &lt;code&gt;ansible_jail_user&lt;/code&gt; inside the jail&lt;/strong&gt;, not as root on the host. For the&amp;nbsp;default &lt;code&gt;ansible_jail_user=root&lt;/code&gt; nothing changes. If you use a non-root jail user and copy into root-owned paths,&amp;nbsp;add &lt;code&gt;become: true&lt;/code&gt; to those&amp;nbsp;tasks.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The host-side privilege footprint shrank&amp;nbsp;to &lt;code&gt;jexec&lt;/code&gt;.&lt;/strong&gt; In my original article I recommended&amp;nbsp;a &lt;code&gt;doas.conf&lt;/code&gt; that allowed the &lt;span class="caps"&gt;SSH&lt;/span&gt; user to&amp;nbsp;run &lt;code&gt;jls&lt;/code&gt;, &lt;code&gt;jexec&lt;/code&gt;, &lt;code&gt;mkdir&lt;/code&gt;, &lt;code&gt;mv&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;rm&lt;/code&gt; as root. An unrestricted&amp;nbsp;root &lt;code&gt;mv&lt;/code&gt; is, in hindsight, a root shell with extra steps. You can delete everything except&amp;nbsp;the &lt;code&gt;jexec&lt;/code&gt; rule:&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;permit nopass ansible as root cmd jexec
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible_jail_root&lt;/code&gt; is deprecated and ignored.&lt;/strong&gt; The plugin no longer needs to know where the jail lives on the host, because it never goes there.&amp;nbsp;The &lt;code&gt;jls&lt;/code&gt; probe is gone as&amp;nbsp;well.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;While I was in there, 2.0.0 also fixed a handful of correctness bugs: the default jail name is now really the inventory hostname&amp;nbsp;(not &lt;code&gt;ansible_host&lt;/code&gt;), &lt;code&gt;ansible_jail_user&lt;/code&gt; is passed&amp;nbsp;as &lt;code&gt;jexec -U&lt;/code&gt; so it is looked up in the jail&amp;#8217;s password database instead of the host&amp;#8217;s, and plugin-internal commands no longer wait for a become prompt that never comes. The &lt;a href="https://codeberg.org/Larvitz/ansible_jailexec/src/branch/main/CHANGELOG.md"&gt;changelog&lt;/a&gt; has the&amp;nbsp;details.&lt;/p&gt;
&lt;p&gt;The release was tested with the unit suite and end-to-end against a real FreeBSD 15.0 jail host. The integration smoke test now round-trips a file&amp;nbsp;through &lt;code&gt;put_file&lt;/code&gt;, &lt;code&gt;slurp&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;fetch_file&lt;/code&gt;, so the transfer path is exercised against a real jail on every&amp;nbsp;run.&lt;/p&gt;
&lt;h2 id="release-and-disclosure"&gt;Release and&amp;nbsp;disclosure&lt;/h2&gt;
&lt;p&gt;For a project with one maintainer, &amp;#8220;coordinated disclosure&amp;#8221; mostly means coordinating with yourself. The timeline on 2026-06-10 (&lt;span class="caps"&gt;UTC&lt;/span&gt;):&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Time&lt;/th&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;before 08:50&lt;/td&gt;
&lt;td&gt;Issue identified while reviewing the transfer code, fix developed and tested&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;08:50&lt;/td&gt;
&lt;td&gt;GitHub security advisory drafted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;08:59&lt;/td&gt;
&lt;td&gt;Fix, &lt;code&gt;SECURITY.md&lt;/code&gt;, and advisory document committed and merged&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;09:01&lt;/td&gt;
&lt;td&gt;Advisory published&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;09:06&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ansible-jailexec 2.0.0&lt;/code&gt; uploaded to PyPI&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Alongside that, 2.0.0 was pushed and tagged on both Codeberg and GitHub, with the advisory text also committed to the repository&amp;nbsp;as &lt;code&gt;docs/security/advisory-2.0.0-jail-escape.md&lt;/code&gt; and a&amp;nbsp;new &lt;code&gt;SECURITY.md&lt;/code&gt; explaining how to report issues privately. GitHub, acting as the &lt;span class="caps"&gt;CVE&lt;/span&gt; Numbering Authority, assigned &lt;strong&gt;&lt;span class="caps"&gt;CVE&lt;/span&gt;-2026-55074&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;The part that does not show up in any feed was contacting people directly. I knew of exactly two people who had re-packaged the plugin into their own Ansible collections. I notified both of them and asked them to pull in&amp;nbsp;2.0.0.&lt;/p&gt;
&lt;p&gt;That is the uncomfortable part: two is the number I &lt;strong&gt;knew&lt;/strong&gt; about, not the number that exists. A plugin like this does not have a mailing list, and many installations are a single Python file copied into&amp;nbsp;someone&amp;#8217;s &lt;code&gt;connection_plugins/&lt;/code&gt; directory or vendored into a collection I have never heard of. Dependency scanners match package metadata, so a copied or vendored file never triggers an alert, and it does not change when PyPI does. For those users, the advisory and this article are the only channels I&amp;nbsp;have.&lt;/p&gt;
&lt;p&gt;If you maintain a collection or repository that bundles a copy&amp;nbsp;of &lt;code&gt;jailexec.py&lt;/code&gt;, please check its version and update&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;Since I had no evidence of anyone exploiting it and the fix was ready, I saw no value in an embargo. Publishing the advisory and the fixed release within minutes of each other kept the window between &amp;#8220;public&amp;#8221; and &amp;#8220;fixed&amp;#8221; as small as I could make&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="a-small-epilogue-about-signatures"&gt;A small epilogue about&amp;nbsp;signatures&lt;/h2&gt;
&lt;p&gt;In August, &lt;a href="https://github.com/chofstede/ansible_jailexec/issues/7"&gt;issue #7&lt;/a&gt; (thanks @geekobiloba) turned up an unrelated bug: Ansible rebuilds a connection plugin&amp;#8217;s options before every loop item, which reset the &lt;span class="caps"&gt;SSH&lt;/span&gt; target back to the jail name, so the second loop item went to the wrong host. That was&amp;nbsp;2.0.1.&lt;/p&gt;
&lt;p&gt;Then GitHub and Codeberg both flagged the 2.0.1 commit as having an invalid signature, because the committer email was not a &lt;span class="caps"&gt;UID&lt;/span&gt; on my signing key. Fixing that meant rewriting the commit, and GitHub&amp;#8217;s immutable releases had already reserved&amp;nbsp;the &lt;code&gt;v2.0.1&lt;/code&gt; tag permanently. So there is now a 2.0.2 that is byte-identical except for the version string and a changelog entry explaining why it exists. After a security release, I would rather ship a boring extra version number than a release whose provenance cannot be&amp;nbsp;verified.&lt;/p&gt;
&lt;h2 id="lessons-learned"&gt;Lessons&amp;nbsp;learned&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Resolve paths on the correct side of the boundary.&lt;/strong&gt; If a privileged process on the outside touches a path the inside controls, the inside controls the privileged process. Do the work inside, with the inside&amp;#8217;s privileges, and pass data across the boundary as a stream rather than as a&amp;nbsp;path.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;String validation is not filesystem validation.&lt;/strong&gt;&amp;nbsp;Rejecting &lt;code&gt;..&lt;/code&gt; felt like security. It protected against the attack I had imagined and did nothing against the one that&amp;nbsp;existed.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Least privilege on the host matters.&lt;/strong&gt; The&amp;nbsp;old &lt;code&gt;doas.conf&lt;/code&gt; handed&amp;nbsp;out &lt;code&gt;mv&lt;/code&gt; and &lt;code&gt;mkdir&lt;/code&gt; as root without restriction. Even without this bug, that was more than the plugin needed. Now it is one&amp;nbsp;line.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Write the security section last.&lt;/strong&gt; Or at least reread it after every refactor. Mine described the vulnerable mechanism as a security&amp;nbsp;feature.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Small projects still deserve a real advisory.&lt;/strong&gt; The whole process, from &lt;span class="caps"&gt;GHSA&lt;/span&gt; draft to &lt;span class="caps"&gt;CVE&lt;/span&gt;, is free and took far less time than fixing the bug. People who depend on your code, even through a copied Python file, deserve a proper identifier they can search for, scan for, and reference in their own change&amp;nbsp;tickets.&lt;/p&gt;
&lt;p&gt;If you run jailexec, please upgrade. If you find something&amp;nbsp;else, &lt;code&gt;SECURITY.md&lt;/code&gt; explains how to reach me privately. I will take the report seriously, and apparently I also write articles about&amp;nbsp;it.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/chofstede/ansible_jailexec/security/advisories/GHSA-cxgv-hp74-jj7r"&gt;&lt;span class="caps"&gt;GHSA&lt;/span&gt;-cxgv-hp74-jj7r / &lt;span class="caps"&gt;CVE&lt;/span&gt;-2026-55074&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://codeberg.org/Larvitz/ansible_jailexec"&gt;jailexec on&amp;nbsp;Codeberg&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/chofstede/ansible_jailexec"&gt;jailexec on&amp;nbsp;GitHub&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pypi.org/project/ansible-jailexec/"&gt;ansible-jailexec on&amp;nbsp;PyPI&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;Original article: Managing FreeBSD Jails with&amp;nbsp;Ansible&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://cwe.mitre.org/data/definitions/59.html"&gt;&lt;span class="caps"&gt;CWE&lt;/span&gt;-59: Improper Link Resolution Before File&amp;nbsp;Access&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=jexec&amp;amp;sektion=8"&gt;jexec(8)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="ansible"/><category term="jails"/><category term="security"/><category term="cve"/><category term="symlinks"/><category term="disclosure"/></entry><entry><title>bhyve Inside a Jail: The FreeBSD Feature Nobody Uses</title><link href="https://hofstede.it/bhyve-inside-a-jail-the-freebsd-feature-nobody-uses/" rel="alternate"/><published>2026-08-27T00:00:00+02:00</published><updated>2026-08-27T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-08-27:/bhyve-inside-a-jail-the-freebsd-feature-nobody-uses/</id><summary type="html">&lt;p&gt;A jail shares the host kernel, so sooner or later you need a real virtual machine. bhyve has been able to run inside a jail since FreeBSD 12.0, and almost nobody does it. This is the full build from my production hypervisor: the devfs ruleset, the four things that have to line up, the two-tier bridge topology, and an honest account of what allow.vmm does and does not buy&amp;nbsp;you.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A hypervisor running virtual machines inside a container" src="https://hofstede.it/images/2026-08-27-bhyve-in-a-jail.webp" title="bhyve running inside a FreeBSD jail"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-bhyve-is-and-what-it-is-not"&gt;What bhyve Is, and What It Is Not&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#loading-the-modules"&gt;Loading the&amp;nbsp;modules&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#raw-bhyve-once"&gt;Raw bhyve,&amp;nbsp;Once&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#vm-bhyve-as-the-daily-interface"&gt;vm-bhyve as the Daily&amp;nbsp;Interface&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#storage-zvol-or-raw-image"&gt;Storage: zvol or Raw&amp;nbsp;Image&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#networking-and-the-diagram-that-explains-everything"&gt;Networking, and the Diagram That Explains&amp;nbsp;Everything&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-actual-subject-bhyve-inside-a-jail"&gt;The Actual Subject: bhyve Inside a Jail&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#why-bother"&gt;Why&amp;nbsp;bother&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-four-things-that-have-to-line-up"&gt;The four things that have to line&amp;nbsp;up&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-fifth-thing-which-is-deliberately-absent"&gt;The fifth thing, which is deliberately&amp;nbsp;absent&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-full-jail-configuration"&gt;The full jail&amp;nbsp;configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#bringing-vm-bhyve-up-inside-it"&gt;Bringing vm-bhyve up inside&amp;nbsp;it&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-this-does-not-buy-you"&gt;What this does not buy&amp;nbsp;you&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#failure-modes"&gt;Failure&amp;nbsp;Modes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-i-would-keep"&gt;What I Would&amp;nbsp;Keep&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;I have written about jails on this blog often enough that the position is not in doubt. Jails are cheap, they are fast, they are the right default, and the vast majority of what people run in virtual machines would be happier in a jail. That is still what I&amp;nbsp;believe.&lt;/p&gt;
&lt;p&gt;It is also, occasionally, wrong. A jail shares the host kernel. That single sentence is the whole boundary of the technology. The moment your workload needs a kernel that is not the host&amp;#8217;s kernel, no amount of enthusiasm for jails will help you, and you need a&amp;nbsp;hypervisor.&lt;/p&gt;
&lt;p&gt;I hit that wall four times in the last year, and every one of them has already shown up on this blog in&amp;nbsp;passing:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/my-home-assistant-setup-bhyve-freebsd-jails-zigbee-homematic-and-matter-over-thread/"&gt;Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt;&lt;/a&gt; ships as an appliance image. It is not going into a&amp;nbsp;jail.&lt;/li&gt;
&lt;li&gt;The &lt;a href="https://hofstede.it/joining-dn42-a-mikrotik-border-three-wireguard-peerings-and-a-freebsd-jail-in-the-hobbyist-internet/"&gt;&lt;span class="caps"&gt;DN42&lt;/span&gt; border router&lt;/a&gt; runs MikroTik &lt;span class="caps"&gt;CHR&lt;/span&gt;, which is somebody else&amp;#8217;s routing stack and definitely somebody else&amp;#8217;s&amp;nbsp;kernel.&lt;/li&gt;
&lt;li&gt;Testing a 16-&lt;span class="caps"&gt;CURRENT&lt;/span&gt; kernel without volunteering the host as the test&amp;nbsp;subject.&lt;/li&gt;
&lt;li&gt;The occasional vendor tool that only exists for Linux or&amp;nbsp;Windows.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In every one of those&amp;nbsp;articles, &lt;code&gt;bhyve&lt;/code&gt; appeared in a sentence or two and then the article moved on to its actual subject. It has never had a piece of its own here. This is that piece, and the part I actually want to write about is the part that gets skipped everywhere else: the hypervisor does not have to run on the host. It can run inside a&amp;nbsp;jail.&lt;/p&gt;
&lt;h2 id="what-bhyve-is-and-what-it-is-not"&gt;What bhyve Is, and What It Is&amp;nbsp;Not&lt;/h2&gt;
&lt;p&gt;bhyve is a hypervisor in the FreeBSD base system. There is no port to install for the core functionality, no subscription, and no daemon. You need hardware virtualization extensions (on amd64, which is what this article is about, that means &lt;span class="caps"&gt;VT&lt;/span&gt;-x with &lt;span class="caps"&gt;EPT&lt;/span&gt; on Intel or &lt;span class="caps"&gt;AMD&lt;/span&gt;-V with &lt;span class="caps"&gt;RVI&lt;/span&gt; on &lt;span class="caps"&gt;AMD&lt;/span&gt;; bhyve also runs on arm64 and &lt;span class="caps"&gt;RISC&lt;/span&gt;-V these days),&amp;nbsp;the &lt;code&gt;vmm&lt;/code&gt; kernel module, and that is essentially the&amp;nbsp;list.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# kldstat | grep vmm
 7    1 0xffffffff82584000   380270 vmm.ko
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Each running guest gets a device&amp;nbsp;under &lt;code&gt;/dev/vmm/&amp;lt;name&amp;gt;&lt;/code&gt;, and serial consoles are handled&amp;nbsp;by &lt;code&gt;nmdm&lt;/code&gt;, the null-modem driver, which gives you a pair of pseudo-terminals wired back to&amp;nbsp;back.&lt;/p&gt;
&lt;p&gt;What bhyve is &lt;em&gt;not&lt;/em&gt; is a virtualization platform. bhyve itself provides no live migration, clustering, &lt;span class="caps"&gt;HA&lt;/span&gt;, web interface, or concept of a resource pool. If you want the thing where a &lt;span class="caps"&gt;VM&lt;/span&gt; moves between hosts because one of them got sad, &lt;a href="https://hofstede.it/hetzner-proxmox-cluster/"&gt;that is what the Proxmox cluster article was about&lt;/a&gt;, and bhyve is not competing for that job. bhyve runs virtual machines on one box, extremely well, with almost no moving parts. Judge it on&amp;nbsp;that.&lt;/p&gt;
&lt;p&gt;One naming detail that will trip you up, because it tripped me up. On 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; the userland control utility is&amp;nbsp;still &lt;code&gt;bhyvectl&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# ls -l /usr/sbin/bhyve*
-r-xr-xr-x  1 root wheel 505672 Jun 24 14:53 /usr/sbin/bhyve
-r-xr-xr-x  1 root wheel  55216 Jun 24 14:53 /usr/sbin/bhyvectl
-r-xr-xr-x  1 root wheel  18744 Jun 24 14:53 /usr/sbin/bhyveload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There is&amp;nbsp;no &lt;code&gt;/usr/sbin/vmmctl&lt;/code&gt;.&amp;nbsp;But &lt;code&gt;/dev/vmmctl&lt;/code&gt; very much exists, and it is the control device the kernel side uses to create and destroy &lt;span class="caps"&gt;VM&lt;/span&gt; instances. So when you&amp;nbsp;see &lt;code&gt;vmmctl&lt;/code&gt; in a devfs ruleset later in this article, that is a device node, not a command. The names invite exactly the wrong&amp;nbsp;assumption.&lt;/p&gt;
&lt;h3 id="loading-the-modules"&gt;Loading the&amp;nbsp;modules&lt;/h3&gt;
&lt;p&gt;While we are in the basement, one thing worth doing correctly from the start. My hypervisor&amp;nbsp;host&amp;#8217;s &lt;code&gt;/boot/loader.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# --- Bhyve &amp;amp; Virtualization Modules ---&lt;/span&gt;
&lt;span class="na"&gt;vmm_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="c1"&gt;# The bhyve hypervisor kernel module&lt;/span&gt;
&lt;span class="na"&gt;nmdm_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="c1"&gt;# Null modem for VM serial consoles&lt;/span&gt;
&lt;span class="na"&gt;if_bridge_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# Network bridging&lt;/span&gt;
&lt;span class="na"&gt;if_epair_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="c1"&gt;# Virtual ethernet pairs (for VNET jails)&lt;/span&gt;
&lt;span class="na"&gt;if_tap_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# Virtual ethernet tap devices (for bhyve)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There are two separate concerns here. For ordinary virtio&amp;nbsp;guests, &lt;code&gt;vmm&lt;/code&gt; can be loaded later&amp;nbsp;with &lt;code&gt;kldload&lt;/code&gt;, and putting it&amp;nbsp;in &lt;code&gt;kld_list&lt;/code&gt; is valid. The &lt;a href="https://man.freebsd.org/cgi/man.cgi?query=vmm&amp;amp;sektion=4&amp;amp;manpath=FreeBSD+15.1-RELEASE"&gt;&lt;code&gt;vmm(4)&lt;/code&gt; manual&lt;/a&gt; requires boot-time loading only when &lt;span class="caps"&gt;PCI&lt;/span&gt; devices must be reserved for passthrough, before their normal host drivers claim&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;This setup deliberately does not use passthrough. I still keep the five modules together&amp;nbsp;in &lt;code&gt;loader.conf&lt;/code&gt; so they are present before the jail service starts; in particular,&amp;nbsp;the &lt;code&gt;allow.vmm&lt;/code&gt; jail parameter does not exist&amp;nbsp;until &lt;code&gt;vmm.ko&lt;/code&gt; is loaded. If you&amp;nbsp;prefer &lt;code&gt;kld_list&lt;/code&gt;, make&amp;nbsp;sure &lt;code&gt;vmm&lt;/code&gt; is loaded before the jail&amp;nbsp;starts.&lt;/p&gt;
&lt;h2 id="raw-bhyve-once"&gt;Raw bhyve,&amp;nbsp;Once&lt;/h2&gt;
&lt;p&gt;I am going to show the manual invocation exactly once, because everything after this hides it behind a tool, and it is worth seeing what is being&amp;nbsp;hidden.&lt;/p&gt;
&lt;p&gt;A minimal FreeBSD guest, booted&amp;nbsp;with &lt;code&gt;bhyveload&lt;/code&gt; rather than &lt;span class="caps"&gt;UEFI&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bhyveload&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1024&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;/dev/zvol/zroot/vm/example/disk0&lt;span class="w"&gt; &lt;/span&gt;example

bhyve&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1024&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;,hostbridge&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;,virtio-blk,/dev/zvol/zroot/vm/example/disk0&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;,virtio-net,tap0&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;31&lt;/span&gt;,lpc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;com1,/dev/nmdm-example.1A&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;example
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two commands, and you have a virtual machine.&amp;nbsp;The &lt;code&gt;-s&lt;/code&gt; flags are &lt;span class="caps"&gt;PCI&lt;/span&gt; slot assignments: a host bridge, a virtio block device backed by a zvol, a virtio &lt;span class="caps"&gt;NIC&lt;/span&gt; backed by a tap interface, and an &lt;span class="caps"&gt;LPC&lt;/span&gt; bridge carrying the serial console out to&amp;nbsp;an &lt;code&gt;nmdm&lt;/code&gt; device.&lt;/p&gt;
&lt;p&gt;When a guest dies badly, the &lt;span class="caps"&gt;VM&lt;/span&gt; instance survives it, and the next start fails because the name is&amp;nbsp;taken:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bhyvectl&lt;span class="w"&gt; &lt;/span&gt;--destroy&lt;span class="w"&gt; &lt;/span&gt;--vm&lt;span class="o"&gt;=&lt;/span&gt;example
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the entire mental&amp;nbsp;model. &lt;code&gt;vm-bhyve&lt;/code&gt;, which is what I actually use, is a shell script that writes those command lines for you from a config file. Nothing more mysterious than&amp;nbsp;that.&lt;/p&gt;
&lt;h2 id="vm-bhyve-as-the-daily-interface"&gt;vm-bhyve as the Daily&amp;nbsp;Interface&lt;/h2&gt;
&lt;p&gt;One thing to hold in mind for the next three sections: on my box every command here runs &lt;em&gt;inside&lt;/em&gt; the jail, not on the host. The host has&amp;nbsp;no &lt;code&gt;vm-bhyve&lt;/code&gt; package,&amp;nbsp;no &lt;code&gt;vm_enable&lt;/code&gt;, and no datastore of its own. I am showing the setup without that complication first because it is the same either way, and coming back to the jail-specific half once the jail itself has been&amp;nbsp;built.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;vm-bhyve
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vm_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vm_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;zfs:zroot/vm&amp;quot;&lt;/span&gt;
vm&lt;span class="w"&gt; &lt;/span&gt;init
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;zfs:&lt;/code&gt; prefix&amp;nbsp;on &lt;code&gt;vm_dir&lt;/code&gt; is the part that matters. It tells vm-bhyve that the datastore is a &lt;span class="caps"&gt;ZFS&lt;/span&gt; dataset rather than a directory, and from that point on every guest is a dataset of its own. Which means every guest inherits your snapshot policy for free. Mine looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;NAME                       USED  RECSIZE
zroot/vm                  28.3G     128K
zroot/vm/bsdmirror        19.5G     128K
zroot/vm/bsdmirror/disk0  19.5G        -
zroot/vm/bsdtest          7.45G     128K
zroot/vm/bsdtest/disk0    7.45G        -
zroot/vm/smallpox         68.9M     128K
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;zroot/vm&lt;/code&gt; is covered by the same sanoid policy as everything else on the box, so there are daily and weekly snapshots of every guest sitting there without my having configured anything &lt;span class="caps"&gt;VM&lt;/span&gt;-specific. Snapshot before a guest upgrade, roll back when the guest upgrade goes the way guest upgrades go. This is the &lt;a href="https://hofstede.it/freebsd-foundationals-zfs-the-last-filesystem-youll-ever-need/"&gt;FreeBSD Foundationals: &lt;span class="caps"&gt;ZFS&lt;/span&gt;&lt;/a&gt; argument applied to virtual machines, and it is most of the reason to put the datastore on &lt;span class="caps"&gt;ZFS&lt;/span&gt; in the first&amp;nbsp;place.&lt;/p&gt;
&lt;p&gt;Day to day, the interface is four&amp;nbsp;commands: &lt;code&gt;vm create&lt;/code&gt;, &lt;code&gt;vm install&lt;/code&gt;, &lt;code&gt;vm console&lt;/code&gt;, &lt;code&gt;vm list&lt;/code&gt;.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;NAME       DATASTORE  LOADER     CPU  MEMORY  VNC  AUTO     STATE
bsdmirror  default    bhyveload  2    1024    -    No       Running (4162)
bsdtest    default    bhyveload  1    256M    -    No       Stopped
smallpox   default    uefi       1    256m    -    Yes [1]  Running (9704)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note&amp;nbsp;the &lt;code&gt;LOADER&lt;/code&gt; column: &lt;code&gt;bhyveload&lt;/code&gt; for the FreeBSD&amp;nbsp;guests, &lt;code&gt;uefi&lt;/code&gt; for the one that needs firmware. Both paths coexist happily on the same host, which is worth knowing before you go looking for a global&amp;nbsp;setting.&lt;/p&gt;
&lt;h2 id="storage-zvol-or-raw-image"&gt;Storage: zvol or Raw&amp;nbsp;Image&lt;/h2&gt;
&lt;p&gt;Both of my storage styles are visible in that output, so let me use the real thing rather than a&amp;nbsp;hypothetical.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;bsdmirror&lt;/code&gt; and &lt;code&gt;bsdtest&lt;/code&gt; are backed by zvols, requested in the guest config with one&amp;nbsp;line:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;disk0_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;virtio-blk&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;disk0_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;disk0&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;disk0_dev&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;sparse-zvol&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Which&amp;nbsp;produces:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;NAME                      VOLSIZE  VOLBLOCK   USED  COMPRESS
zroot/vm/bsdmirror/disk0      50G       16K  19.5G  on
zroot/vm/bsdtest/disk0        20G       16K  7.45G  on
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Sparse, so a 50G guest disk costs 19.5G of pool.&amp;nbsp;16K &lt;code&gt;volblocksize&lt;/code&gt;, compression&amp;nbsp;on.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;smallpox&lt;/code&gt; uses the default instead, a raw image file on the&amp;nbsp;dataset:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;disk0_type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;virtio-blk&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;disk0_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;disk0.img&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The tradeoff is not dramatic and I would not lose sleep over either choice. zvols give you a block device with its own properties, snapshots, and tuning knobs, and they are the natural fit when the guest is going to be long-lived. Actual performance depends on the workload and the block-size choices at both layers. Raw images are a single file, which means that, with the guest stopped, you can copy one out&amp;nbsp;with &lt;code&gt;cp&lt;/code&gt;, hand it to somebody, or move it between datastores without thinking about &lt;span class="caps"&gt;ZFS&lt;/span&gt; at&amp;nbsp;all.&lt;/p&gt;
&lt;p&gt;The one trap worth naming: if the guest runs its own copy-on-write filesystem, and for FreeBSD guests it usually does, you now have &lt;span class="caps"&gt;ZFS&lt;/span&gt; on top of &lt;span class="caps"&gt;ZFS&lt;/span&gt;. Both layers will try to be clever about block sizes, and the interaction is rarely an improvement.&amp;nbsp;Match &lt;code&gt;volblocksize&lt;/code&gt; to what the guest actually writes rather than leaving it at whatever you first typed, and resist the urge to enable every feature on both&amp;nbsp;layers.&lt;/p&gt;
&lt;h2 id="networking-and-the-diagram-that-explains-everything"&gt;Networking, and the Diagram That Explains&amp;nbsp;Everything&lt;/h2&gt;
&lt;p&gt;This is where the jail story starts, so it is worth being&amp;nbsp;precise.&lt;/p&gt;
&lt;p&gt;On a conventional bhyve host, the topology is short: a bridge, a tap per guest &lt;span class="caps"&gt;NIC&lt;/span&gt;, done. On my setup the hypervisor lives inside a jail,&amp;nbsp;called &lt;code&gt;tardis&lt;/code&gt;, and the result is a two-tier arrangement that took me longer to hold in my head than I would like to admit. That jail is the subject of the rest of this article; for now all you need is its name, because it turns up in every interface name below. Here is one &lt;span class="caps"&gt;NIC&lt;/span&gt;, end to&amp;nbsp;end:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;host bridge0
   |
   +-- e0a_tardis        (epair, &amp;quot;a&amp;quot; side, stays on the host)
         ~~~~~~~~~~
       e0b_tardis        (epair, &amp;quot;b&amp;quot; side, handed into the jail,
                          then renamed to vnet0 by the jail&amp;#39;s rc.conf)
                              |
                          vm-public   (vm-bhyve switch &amp;quot;public&amp;quot;,
                                       a bridge inside the jail)
                              |
                            tap0  -&amp;gt;  guest virtio-net
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Three of those, in&amp;nbsp;parallel: &lt;code&gt;bridge0&lt;/code&gt; to &lt;code&gt;vm-public&lt;/code&gt;, &lt;code&gt;bridge1&lt;/code&gt; to &lt;code&gt;vm-wan&lt;/code&gt;, &lt;code&gt;bridge2&lt;/code&gt; to &lt;code&gt;vm-lan&lt;/code&gt;. Management, &lt;span class="caps"&gt;WAN&lt;/span&gt;, and &lt;span class="caps"&gt;LAN&lt;/span&gt;, which is the same three-way separation I use everywhere else and the same one described in the &lt;a href="https://hofstede.it/running-your-own-as-direct-hetzner-peering-a-fourth-edge-and-bringing-the-home-lan-into-the-fabric/"&gt;&lt;span class="caps"&gt;AS201379&lt;/span&gt; peering articles&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Two details in there cost me real&amp;nbsp;time.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The interface is renamed explicitly.&lt;/strong&gt; The jail config names the b side of each&amp;nbsp;epair &lt;code&gt;e0b_tardis&lt;/code&gt;, &lt;code&gt;e1b_tardis&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;e2b_tardis&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;vnet.interface&lt;/code&gt; parameter moves them into the jail with those names intact. These lines in the&amp;nbsp;jail&amp;#8217;s &lt;code&gt;/etc/rc.conf&lt;/code&gt; perform the rename when its network service&amp;nbsp;starts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;ifconfig_e0b_tardis_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vnet0&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;ifconfig_e1b_tardis_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vnet1&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;ifconfig_e2b_tardis_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vnet2&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is why the interfaces&amp;nbsp;are &lt;code&gt;vnet0&lt;/code&gt;, &lt;code&gt;vnet1&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;vnet2&lt;/code&gt; by the time vm-bhyve sees them. If you omit these lines, configure the vm-bhyve switches with the&amp;nbsp;original &lt;code&gt;e0b_tardis&lt;/code&gt; names instead. The rename is a policy choice, not something &lt;span class="caps"&gt;VNET&lt;/span&gt; does&amp;nbsp;automatically.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The taps are invisible from the host.&lt;/strong&gt; This is the payoff of the entire arrangement, and you can see it in two commands. From the&amp;nbsp;host:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# ifconfig -l
bge0 bge1 lo0 pflog0 bridge0 bridge1 bridge2
e0a_tardis e1a_tardis e2a_tardis
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Bridges, and the host side of some epairs. No taps, no &lt;span class="caps"&gt;VM&lt;/span&gt; plumbing, nothing hypervisor-shaped at all. From inside the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# jexec tardis ifconfig -l
lo0 pflog0 vnet0 vnet1 vnet2 vm-public vm-wan vm-lan tap0 tap1 tap2 tap3 tap4
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Everything. And vm-bhyve labels the taps helpfully, so you never have to guess which one belongs to&amp;nbsp;what:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;tap0: description: vmnet/smallpox/0/public
tap1: description: vmnet/smallpox/1/wan
tap2: description: vmnet/smallpox/2/lan
tap3: description: vmnet/bsdmirror/0/public
tap4: description: vmnet/bsdmirror/1/lan
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Five taps for two running guests: three NICs&amp;nbsp;on &lt;code&gt;smallpox&lt;/code&gt;, two&amp;nbsp;on &lt;code&gt;bsdmirror&lt;/code&gt;. &lt;code&gt;bsdtest&lt;/code&gt; is stopped and therefore adds no tap&amp;nbsp;interfaces.&lt;/p&gt;
&lt;p&gt;If you already run pf on the host, and after the &lt;a href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/"&gt;pf guide&lt;/a&gt; I assume some of you do, be aware that bridged traffic and pf interact&amp;nbsp;through &lt;code&gt;net.link.bridge.pfil_bridge&lt;/code&gt; and its neighbours. The defaults have changed over the years and the failure mode is &amp;#8220;traffic silently does or does not get filtered, differently from what you assumed&amp;#8221;. Check the sysctls rather than trusting your memory of&amp;nbsp;them.&lt;/p&gt;
&lt;h2 id="the-actual-subject-bhyve-inside-a-jail"&gt;The Actual Subject: bhyve Inside a&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;Here is the thing that made me want to write&amp;nbsp;this. &lt;code&gt;allow.vmm&lt;/code&gt; is not new. It was &lt;a href="https://reviews.freebsd.org/D16057"&gt;committed as r337023&lt;/a&gt; on 1 August 2018 and shipped in FreeBSD 12.0. It has been sitting in the base system for eight years, it is documented&amp;nbsp;in &lt;code&gt;jail(8)&lt;/code&gt;, and in all that time I have seen it discussed approximately&amp;nbsp;never.&lt;/p&gt;
&lt;h3 id="why-bother"&gt;Why&amp;nbsp;bother&lt;/h3&gt;
&lt;p&gt;Because it turns the hypervisor into a normal&amp;nbsp;service.&lt;/p&gt;
&lt;p&gt;Everything the &lt;span class="caps"&gt;VM&lt;/span&gt; host needs, the zvols, the taps, the consoles, its own routing table and its own firewall, lives inside one&amp;nbsp;jail. &lt;code&gt;service jail restart tardis&lt;/code&gt; restarts the entire virtualization&amp;nbsp;stack. &lt;code&gt;zfs send&lt;/code&gt; of the jail plus its datastore moves it. The host underneath stays a boring machine whose only job is running jails, with no &lt;span class="caps"&gt;VM&lt;/span&gt;-shaped special case in its&amp;nbsp;configuration.&lt;/p&gt;
&lt;p&gt;If you have spent any time keeping a hypervisor host tidy, you will recognise the appeal. The host stops being a&amp;nbsp;pet.&lt;/p&gt;
&lt;h3 id="the-four-things-that-have-to-line-up"&gt;The four things that have to line&amp;nbsp;up&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;One: &lt;code&gt;allow.vmm = 1&lt;/code&gt;.&lt;/strong&gt; This is the parameter that lets a process inside the jail&amp;nbsp;open &lt;code&gt;vmm&lt;/code&gt; at all. &lt;a href="https://man.freebsd.org/cgi/man.cgi?query=jail&amp;amp;sektion=8&amp;amp;manpath=FreeBSD+15.1-RELEASE"&gt;&lt;code&gt;jail(8)&lt;/code&gt;&lt;/a&gt; puts it&amp;nbsp;plainly:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;allow.vmm - The jail may access vmm(4). This flag is only available when the vmm(4) kernel module is&amp;nbsp;loaded.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Note the second sentence, because it produces a confusing failure:&amp;nbsp;if &lt;code&gt;vmm.ko&lt;/code&gt; is not loaded on the host, the parameter does not merely have no effect, it does not exist, and your jail fails to start with a complaint about an unknown&amp;nbsp;parameter.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Two: a devfs ruleset that unhides the right nodes.&lt;/strong&gt; This is the piece people get wrong, and it is&amp;nbsp;why &lt;code&gt;vm console&lt;/code&gt; works for some and not others. My ruleset&amp;nbsp;10:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[devfsrules_bhyve_jail=10]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path fuse unhide
add path zfs unhide
add path pf unhide
add path &amp;#39;vmm*&amp;#39; unhide
add path &amp;#39;vmm/*&amp;#39; unhide
add path &amp;#39;vmm.io*&amp;#39; unhide
add path &amp;#39;vmm.io/*&amp;#39; unhide
add path &amp;#39;vmmctl&amp;#39; unhide
add path &amp;#39;tap*&amp;#39; unhide
add path &amp;#39;nmdm*&amp;#39; unhide
add path &amp;#39;zvol*&amp;#39; unhide
add path &amp;#39;zvol/*&amp;#39; unhide
add path &amp;#39;zvol/*/*&amp;#39; unhide
add path &amp;#39;zvol/*/*/*&amp;#39; unhide
add path &amp;#39;zvol/*/*/*/*&amp;#39; unhide
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Working through&amp;nbsp;it:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;It starts&amp;nbsp;from &lt;code&gt;devfsrules_hide_all&lt;/code&gt; and then&amp;nbsp;unhides &lt;code&gt;basic&lt;/code&gt; and &lt;code&gt;login&lt;/code&gt;. Start from nothing and add, never the&amp;nbsp;reverse.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;vmm*&lt;/code&gt;, &lt;code&gt;vmm/*&lt;/code&gt;, &lt;code&gt;vmm.io*&lt;/code&gt;, &lt;code&gt;vmm.io/*&lt;/code&gt;, &lt;code&gt;vmmctl&lt;/code&gt; are the hypervisor devices, including the per-guest instances that appear&amp;nbsp;under &lt;code&gt;/dev/vmm/&lt;/code&gt; and the control device discussed&amp;nbsp;earlier.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;tap*&lt;/code&gt; is guest NICs. Without it the guest starts and has no&amp;nbsp;network.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;nmdm*&lt;/code&gt; is guest serial consoles. Without&amp;nbsp;it &lt;code&gt;vm console&lt;/code&gt; is dead and you will assume the guest failed to boot when it is running perfectly and simply cannot talk to&amp;nbsp;you.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;zvol*&lt;/code&gt; and the nested globs are block-backed guests. The nesting matters: the path&amp;nbsp;is &lt;code&gt;zvol/&amp;lt;pool&amp;gt;/&amp;lt;dataset&amp;gt;/&amp;lt;disk&amp;gt;&lt;/code&gt;, so a single level of glob is not enough, and how deep you need to go depends on how deep your datasets are nested. Mine goes to five levels because I would rather not revisit&amp;nbsp;this.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;zfs&lt;/code&gt; and &lt;code&gt;pf&lt;/code&gt; let the jail manage its own datasets and its own&amp;nbsp;firewall.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;fuse&lt;/code&gt; is, on my system, a&amp;nbsp;leftover. &lt;code&gt;fusefs&lt;/code&gt; is not loaded and nothing in the jail mounts anything through it. I am leaving it in the printed ruleset because it is what is actually deployed, but it is doing nothing, and if you are building this fresh you can skip&amp;nbsp;it.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The result is&amp;nbsp;a &lt;code&gt;/dev&lt;/code&gt; that stays genuinely&amp;nbsp;small:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# jexec tardis ls /dev
fd  nmdm-bsdmirror.1A  nmdm-bsdmirror.1B  nmdm-smallpox.1A
nmdm-smallpox.1B  null  pf  pts  random  stderr  stdin  stdout
tap0  tap1  tap2  tap3  tap4  urandom  vmm  vmm.io  vmmctl
zero  zfs  zvol
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Three: &lt;span class="caps"&gt;ZFS&lt;/span&gt; delegation.&lt;/strong&gt; vm-bhyve needs to create datasets and zvols, from inside the jail, in a dataset the host owns. That takes a matching pair. On the host side, the dataset is marked jailed and handed over as the jail is&amp;nbsp;created:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;jailed&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;on&lt;span class="w"&gt; &lt;/span&gt;zroot/vm
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;exec.created += &amp;quot;zfs jail $name zroot/vm&amp;quot;;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And in the jail config, permission to actually mount&amp;nbsp;things:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;allow.mount;
allow.mount.devfs;
allow.mount.zfs;
enforce_statfs = 1;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Miss &lt;code&gt;jailed=on&lt;/code&gt; and &lt;code&gt;vm create&lt;/code&gt; fails with a permissions error that points at the dataset and does not tell you which of the two halves you forgot. The result, once it works, is that the jail mounts its own&amp;nbsp;datastore:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# jexec tardis mount
zroot/jails/tardis on / (zfs, local, noatime, nfsv4acls)
zroot/vm on /zroot/vm (zfs, local, noatime, nfsv4acls)
zroot/vm/smallpox on /zroot/vm/smallpox (zfs, local, noatime, nfsv4acls)
zroot/vm/bsdtest on /zroot/vm/bsdtest (zfs, local, noatime, nfsv4acls)
zroot/vm/bsdmirror on /zroot/vm/bsdmirror (zfs, local, noatime, nfsv4acls)
devfs on /dev (devfs)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Four: &lt;code&gt;allow.mlock = 1&lt;/code&gt;.&lt;/strong&gt; bhyve wires guest memory rather than letting it be paged. A jail cannot do that by default. Leave this out and the guest refuses to start, with an error that describes a memory allocation failure and does not obviously say &amp;#8220;you forgot a jail&amp;nbsp;parameter&amp;#8221;.&lt;/p&gt;
&lt;h3 id="the-fifth-thing-which-is-deliberately-absent"&gt;The fifth thing, which is deliberately&amp;nbsp;absent&lt;/h3&gt;
&lt;p&gt;Ask the running jail what it is allowed to do and one entry stands&amp;nbsp;out:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# jls -j tardis -n
...
allow.vmm
allow.novmm_ppt
...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;allow.vmm_ppt&lt;/code&gt; is off. &lt;span class="caps"&gt;PCI&lt;/span&gt; passthrough is a &lt;em&gt;separate&lt;/em&gt; capability from access to vmm, and this is new: it was added in &lt;a href="https://www.freebsd.org/releases/15.1R/relnotes/"&gt;FreeBSD 15.1&lt;/a&gt; as &lt;a href="https://cgit.freebsd.org/src/commit/?id=94066f9a07ad"&gt;commit 94066f9a07ad&lt;/a&gt;, sponsored by the FreeBSD Foundation and Klara. The release notes describe it as a knob &amp;#8220;to control &lt;span class="caps"&gt;PCI&lt;/span&gt; passthrough access in bhyve(8)&amp;nbsp;jails&amp;#8221;.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;jail(8)&lt;/code&gt; is unusually direct about why you want it&amp;nbsp;off:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;This allows privileged users inside the jail to manipulate physical devices claimed by the ppt driver, and thus must not be configured in untrusted&amp;nbsp;jails.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;So a jailed hypervisor runs virtio guests all day and cannot hand a physical device to a guest. For me that is the correct trade, and it is also the honest answer to the question people will ask after the &lt;a href="https://hofstede.it/building-my-own-tv-station-ersatztv-in-a-freebsd-jail-with-intel-hardware-transcoding/"&gt;ErsatzTV article&lt;/a&gt;: no, you cannot do the &lt;span class="caps"&gt;GPU&lt;/span&gt; passthrough thing this way. If you need passthrough, you are running bhyve on the host, or you are turning the knob on and accepting&amp;nbsp;what &lt;code&gt;jail(8)&lt;/code&gt; just told&amp;nbsp;you.&lt;/p&gt;
&lt;h3 id="the-full-jail-configuration"&gt;The full jail&amp;nbsp;configuration&lt;/h3&gt;
&lt;p&gt;For reference, the whole thing, with the three-&lt;span class="caps"&gt;NIC&lt;/span&gt; vnet&amp;nbsp;plumbing:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;tardis {
    devfs_ruleset = 10;
    exec.clean;
    exec.consolelog = /var/log/jails/tardis_console.log;
    exec.start = &amp;#39;/bin/sh /etc/rc&amp;#39;;
    exec.stop = &amp;#39;/bin/sh /etc/rc.shutdown&amp;#39;;
    host.hostname = &amp;quot;tardis&amp;quot;;
    mount.devfs;
    path = &amp;quot;/zroot/jails/$name&amp;quot;;

    # --- Virtualization &amp;amp; ZFS Permissions ---
    allow.vmm = 1;
    # allow.vmm_ppt is deliberately omitted, and therefore off
    allow.mount;
    allow.mount.devfs;
    allow.mount.zfs;
    enforce_statfs = 1;
    allow.mlock = 1;
    allow.chflags;   # not bhyve-related, see below

    vnet = new;
    vnet.interface = &amp;quot;e0b_tardis&amp;quot;, &amp;quot;e1b_tardis&amp;quot;, &amp;quot;e2b_tardis&amp;quot;;

    # NIC 0 (Management -&amp;gt; bridge0)
    exec.prestart += &amp;quot;epair0=\$(ifconfig epair create) &amp;amp;&amp;amp; ifconfig \${epair0} up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e0a_tardis &amp;amp;&amp;amp; ifconfig \${epair0%a}b up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e0b_tardis&amp;quot;;
    exec.prestart += &amp;quot;ifconfig bridge0 addm e0a_tardis&amp;quot;;
    exec.prestart += &amp;quot;ifconfig e0a_tardis description \&amp;quot;vnet0 mgmt for tardis\&amp;quot;&amp;quot;;

    # NIC 1 (WAN/BGP -&amp;gt; bridge1)
    exec.prestart += &amp;quot;epair1=\$(ifconfig epair create) &amp;amp;&amp;amp; ifconfig \${epair1} up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e1a_tardis &amp;amp;&amp;amp; ifconfig \${epair1%a}b up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e1b_tardis&amp;quot;;
    exec.prestart += &amp;quot;ifconfig bridge1 addm e1a_tardis&amp;quot;;
    exec.prestart += &amp;quot;ifconfig e1a_tardis description \&amp;quot;vnet1 WAN/BGP for tardis\&amp;quot;&amp;quot;;

    # NIC 2 (LAN -&amp;gt; bridge2)
    exec.prestart += &amp;quot;epair2=\$(ifconfig epair create) &amp;amp;&amp;amp; ifconfig \${epair2} up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e2a_tardis &amp;amp;&amp;amp; ifconfig \${epair2%a}b up -txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro name e2b_tardis&amp;quot;;
    exec.prestart += &amp;quot;ifconfig bridge2 addm e2a_tardis&amp;quot;;
    exec.prestart += &amp;quot;ifconfig e2a_tardis description \&amp;quot;vnet2 LAN for tardis\&amp;quot;&amp;quot;;

    # Teardown
    exec.poststop += &amp;quot;ifconfig e0a_tardis destroy&amp;quot;;
    exec.poststop += &amp;quot;ifconfig e1a_tardis destroy&amp;quot;;
    exec.poststop += &amp;quot;ifconfig e2a_tardis destroy&amp;quot;;

    exec.created += &amp;quot;zfs jail $name zroot/vm&amp;quot;;
    exec.jail_user = root;
    exec.system_user = root;
    persist;
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;exec.prestart&lt;/code&gt; lines each create an epair, disable every offload on both halves, rename both ends to something a human can read, and add the host side to a bridge. The offload flags&amp;nbsp;(&lt;code&gt;-txcsum -rxcsum -txcsum6 -rxcsum6 -tso -lro&lt;/code&gt;) are not superstition. Checksum offload and &lt;span class="caps"&gt;TSO&lt;/span&gt; across an epair, through a bridge, into a virtio &lt;span class="caps"&gt;NIC&lt;/span&gt; in a guest is a well-known way to get packets that are subtly wrong rather than absent, which is a far worse debugging experience than no connectivity at all. Turn them off and move on with your&amp;nbsp;life.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;allow.chflags&lt;/code&gt; is not part of the bhyve story. It lets privileged users in the jail clear system file flags such&amp;nbsp;as &lt;code&gt;schg&lt;/code&gt;, which matters for jails that maintain their own userland and not at all for running virtual machines. It is in my config because it is in my other jail configs, and I am printing what is deployed. Nothing here needs&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;exec.poststop&lt;/code&gt; destroys the host-side interfaces, so restarting the jail does not leave a trail of orphaned epairs&amp;nbsp;behind. &lt;code&gt;persist&lt;/code&gt; keeps the jail alive with no processes running in it, which matters because a vnet jail that exits takes its network configuration with&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="bringing-vm-bhyve-up-inside-it"&gt;Bringing vm-bhyve up inside&amp;nbsp;it&lt;/h3&gt;
&lt;p&gt;Now the half I deferred earlier. With the jail running, the setup from the vm-bhyve section happens inside it, and nowhere&amp;nbsp;else:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;jexec&lt;span class="w"&gt; &lt;/span&gt;tardis&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;vm-bhyve
jexec&lt;span class="w"&gt; &lt;/span&gt;tardis&lt;span class="w"&gt; &lt;/span&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vm_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
jexec&lt;span class="w"&gt; &lt;/span&gt;tardis&lt;span class="w"&gt; &lt;/span&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vm_dir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;zfs:zroot/vm&amp;quot;&lt;/span&gt;
jexec&lt;span class="w"&gt; &lt;/span&gt;tardis&lt;span class="w"&gt; &lt;/span&gt;vm&lt;span class="w"&gt; &lt;/span&gt;init
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;vm_enable&lt;/code&gt; and &lt;code&gt;vm_dir&lt;/code&gt; go in the &lt;em&gt;jail&amp;#8217;s&lt;/em&gt; &lt;code&gt;/etc/rc.conf&lt;/code&gt;. The&amp;nbsp;host&amp;#8217;s &lt;code&gt;rc.conf&lt;/code&gt; has neither, and that is the point of the whole exercise: the host does not know it is a&amp;nbsp;hypervisor.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;vm_dir&lt;/code&gt; is the delegated dataset by its &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;nbsp;name, &lt;code&gt;zroot/vm&lt;/code&gt;, not by a path. That name is identical inside and outside the jail, which is convenient, and it is the same dataset&amp;nbsp;that &lt;code&gt;zfs jail&lt;/code&gt; handed over&amp;nbsp;at &lt;code&gt;exec.created&lt;/code&gt;. The jail mounts it itself at startup, which is what&amp;nbsp;the &lt;code&gt;mount&lt;/code&gt; output above shows,&amp;nbsp;and &lt;code&gt;vm init&lt;/code&gt; then creates the datastore layout inside&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The vm-bhyve switches are created in the jail too, against the renamed vnet&amp;nbsp;interfaces:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;jexec&lt;span class="w"&gt; &lt;/span&gt;tardis&lt;span class="w"&gt; &lt;/span&gt;vm&lt;span class="w"&gt; &lt;/span&gt;switch&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;public
jexec&lt;span class="w"&gt; &lt;/span&gt;tardis&lt;span class="w"&gt; &lt;/span&gt;vm&lt;span class="w"&gt; &lt;/span&gt;switch&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;vnet0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Repeat&amp;nbsp;for &lt;code&gt;wan&lt;/code&gt; on &lt;code&gt;vnet1&lt;/code&gt; and &lt;code&gt;lan&lt;/code&gt; on &lt;code&gt;vnet2&lt;/code&gt;, and that produces&amp;nbsp;the &lt;code&gt;vm-public&lt;/code&gt;, &lt;code&gt;vm-wan&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;vm-lan&lt;/code&gt; bridges from the diagram. From&amp;nbsp;here &lt;code&gt;vm create&lt;/code&gt;, &lt;code&gt;vm install&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;vm console&lt;/code&gt; behave exactly as they do on a bare host, which is the whole reason this arrangement is worth the&amp;nbsp;setup.&lt;/p&gt;
&lt;h3 id="what-this-does-not-buy-you"&gt;What this does not buy&amp;nbsp;you&lt;/h3&gt;
&lt;p&gt;I want to be clear about this, because &amp;#8220;hypervisor in a jail&amp;#8221; sounds like it should be a security improvement, and it is&amp;nbsp;not.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;allow.vmm&lt;/code&gt; hands the jail a large and complicated piece of kernel attack surface. The vmm code is not a small, carefully minimised interface, and a root process inside that jail is talking to it directly. Confining the hypervisor to a jail is an &lt;em&gt;operational&lt;/em&gt; win: it packages the thing, it makes it movable and restartable, it keeps the host clean. It is not a containment upgrade, and a bhyve-capable jail is not somewhere to put a workload you do not trust.&amp;nbsp;Turning &lt;code&gt;allow.vmm_ppt&lt;/code&gt; on as well would make that considerably more&amp;nbsp;true.&lt;/p&gt;
&lt;p&gt;Treat the jail boundary here as an organisational tool, not a security boundary, and you will not be disappointed by&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="failure-modes"&gt;Failure&amp;nbsp;Modes&lt;/h2&gt;
&lt;p&gt;The ones I actually hit, in roughly the order I hit&amp;nbsp;them:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The console works from the host but not from the jail.&lt;/strong&gt;&amp;nbsp;Missing &lt;code&gt;nmdm*&lt;/code&gt; in the devfs ruleset. The guest is fine. You just cannot talk to&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;vm create&lt;/code&gt; fails with a permissions error on the dataset.&lt;/strong&gt; Half of the &lt;span class="caps"&gt;ZFS&lt;/span&gt; delegation is&amp;nbsp;missing. &lt;code&gt;jailed=on&lt;/code&gt; on the host side&amp;nbsp;and &lt;code&gt;zfs jail&lt;/code&gt; at &lt;code&gt;exec.created&lt;/code&gt; are two separate steps and you need&amp;nbsp;both.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The guest refuses to start with a memory&amp;nbsp;error.&lt;/strong&gt; &lt;code&gt;allow.mlock&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The jail refuses to start, complaining about an unknown&amp;nbsp;parameter.&lt;/strong&gt; &lt;code&gt;vmm.ko&lt;/code&gt; is not loaded on the host,&amp;nbsp;so &lt;code&gt;allow.vmm&lt;/code&gt; does not exist yet.&amp;nbsp;Put &lt;code&gt;vmm_load="YES"&lt;/code&gt; in &lt;code&gt;loader.conf&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Networking that half works.&lt;/strong&gt; Offloads on the epairs. Small packets fine, large transfers corrupt or&amp;nbsp;stalled.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A guest will not restart after an unclean shutdown.&lt;/strong&gt; Stale instance&amp;nbsp;under &lt;code&gt;/dev/vmm/&lt;/code&gt;. &lt;code&gt;bhyvectl --destroy --vm=&amp;lt;name&amp;gt;&lt;/code&gt; and try&amp;nbsp;again.&lt;/p&gt;
&lt;h2 id="what-i-would-keep"&gt;What I Would&amp;nbsp;Keep&lt;/h2&gt;
&lt;p&gt;All of it, which is not something I say often after a year of running&amp;nbsp;something.&lt;/p&gt;
&lt;p&gt;Jails remain the answer for almost everything. bhyve is for the handful of workloads that genuinely need their own kernel, and there are fewer of those than people assume. But wrapping the hypervisor in a jail is the part I would do again without hesitating, because it moved virtual machines from &amp;#8220;special thing the host does&amp;#8221; into &amp;#8220;one more jail on a host that only runs jails&amp;#8221;. The host has no &lt;span class="caps"&gt;VM&lt;/span&gt;-specific configuration beyond five module loads&amp;nbsp;in &lt;code&gt;loader.conf&lt;/code&gt;. Everything else is&amp;nbsp;inside &lt;code&gt;tardis&lt;/code&gt;, snapshotted nightly along with everything else, and restartable with one&amp;nbsp;command.&lt;/p&gt;
&lt;p&gt;Eight years this has been in the base system. It deserves more use than it&amp;nbsp;gets.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="bhyve"/><category term="virtualization"/><category term="jails"/><category term="vm-bhyve"/><category term="zfs"/><category term="vnet"/><category term="networking"/></entry><entry><title>Three Test Jails on a 512 MB Free-Tier VPS</title><link href="https://hofstede.it/three-test-jails-on-a-512-mb-free-tier-vps/" rel="alternate"/><published>2026-08-04T00:00:00+02:00</published><updated>2026-08-04T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-08-04:/three-test-jails-on-a-512-mb-free-tier-vps/</id><summary type="html">&lt;p&gt;A bug report against my jailexec connection plugin needed a real FreeBSD jail to reproduce. An idle free-tier &lt;span class="caps"&gt;VPS&lt;/span&gt; with 512 &lt;span class="caps"&gt;MB&lt;/span&gt; of &lt;span class="caps"&gt;RAM&lt;/span&gt; turned out to be plenty: pkgbase provides a 33 &lt;span class="caps"&gt;MB&lt;/span&gt; base userland, persistent jails need no idle processes, and three Python-equipped test targets added only 1.1 &lt;span class="caps"&gt;GB&lt;/span&gt; of disk use. This is the build, including the two sharp edges that cost me twenty&amp;nbsp;minutes.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A compact server hosting three lightweight test environments" src="https://hofstede.it/images/2026-08-04-pkgbase-test-jails-free-tier-vps.webp" title="Three lightweight FreeBSD test jails on a small server"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-box"&gt;The&amp;nbsp;Box&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-pkgbase-makes-test-jails-cheap"&gt;Why pkgbase Makes Test Jails Cheap&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#sharp-edge-one-the-fingerprints-live-inside-the-rootdir"&gt;Sharp edge one: the fingerprints live inside the&amp;nbsp;rootdir&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#sharp-edge-two-nobody-runs-ldconfig-for-you"&gt;Sharp edge two: nobody runs ldconfig for&amp;nbsp;you&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-smallest-jailconf-that-does-anything"&gt;The Smallest jail.conf That Does&amp;nbsp;Anything&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#three-jails-three-code-paths"&gt;Three Jails, Three Code Paths&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-gotcha-that-will-bite-anyone-running-ansible-against-a-jail"&gt;The gotcha that will bite anyone running Ansible against a&amp;nbsp;jail&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#does-the-test-actually-catch-the-bug"&gt;Does the Test Actually Catch the&amp;nbsp;Bug?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-it-cost"&gt;What It&amp;nbsp;Cost&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#takeaways"&gt;Takeaways&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;Someone opened &lt;a href="https://github.com/chofstede/ansible_jailexec/issues/7"&gt;an issue against ansible-jailexec&lt;/a&gt;, the &lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;Ansible connection plugin for FreeBSD jails&lt;/a&gt; I wrote about at the end of last year, with an admirably small reproducer:&amp;nbsp;a &lt;code&gt;copy&lt;/code&gt; task works, the&amp;nbsp;same &lt;code&gt;copy&lt;/code&gt; task with&amp;nbsp;a &lt;code&gt;loop&lt;/code&gt; breaks the play. First item fine, everything after it&amp;nbsp;dead.&lt;/p&gt;
&lt;p&gt;I found the cause by&amp;nbsp;reading &lt;code&gt;ansible-core&lt;/code&gt; source. Ansible rebuilds a connection plugin&amp;#8217;s option dictionary before every task — and inside a loop, before every single item. My plugin redirects &lt;span class="caps"&gt;SSH&lt;/span&gt; at the jail &lt;em&gt;host&lt;/em&gt; by overriding&amp;nbsp;the &lt;code&gt;host&lt;/code&gt; option exactly once,&amp;nbsp;in &lt;code&gt;_connect()&lt;/code&gt;. The option reload wiped that&amp;nbsp;redirect, &lt;code&gt;_connect()&lt;/code&gt; short-circuits on an established connection so nothing put it back, and from item two onward Ansible dutifully tried to &lt;span class="caps"&gt;SSH&lt;/span&gt; to the jail name&amp;nbsp;itself.&lt;/p&gt;
&lt;p&gt;The unit tests confirmed the fix. That is not the same as knowing it works. The whole point of a connection plugin is the part that talks to a real machine, and my mocks are exactly as correct as my understanding&amp;nbsp;of &lt;code&gt;ansible-core&lt;/code&gt;, which had just been demonstrated to be&amp;nbsp;incomplete.&lt;/p&gt;
&lt;p&gt;So I needed test jails. Real&amp;nbsp;ones.&lt;/p&gt;
&lt;h2 id="the-box"&gt;The&amp;nbsp;Box&lt;/h2&gt;
&lt;p&gt;What I had lying around was a Vultr free-tier instance that has been running nothing&amp;nbsp;but &lt;code&gt;sshd&lt;/code&gt; and a WireGuard interface for 51&amp;nbsp;days:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;CPU:    1 × Intel Xeon (Skylake, IBRS)
RAM:    496 MB physical, 512 MB swap
Disk:   9.2 GB UFS, 1.3 GB used
OS:     FreeBSD 15.1-RELEASE, installed from pkgbase
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Half a gigabyte of &lt;span class="caps"&gt;RAM&lt;/span&gt; and a single core. Twelve years ago this was a respectable server; today it is below the minimum system requirements of several &amp;#8220;lightweight&amp;#8221; container platforms. It is, however, entirely sufficient for running three jails that do nothing but wait to have commands executed in&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;The install was pkgbase, which turned out to be the detail that made the whole exercise&amp;nbsp;pleasant.&lt;/p&gt;
&lt;h2 id="why-pkgbase-makes-test-jails-cheap"&gt;Why pkgbase Makes Test Jails&amp;nbsp;Cheap&lt;/h2&gt;
&lt;p&gt;The traditional way to populate a jail is to&amp;nbsp;fetch &lt;code&gt;base.txz&lt;/code&gt; and extract it. That gives you the full base system — every binary, every&amp;nbsp;library, &lt;code&gt;/usr/share/doc&lt;/code&gt;, the lot — for roughly 1 &lt;span class="caps"&gt;GB&lt;/span&gt; per jail before you install anything of your&amp;nbsp;own.&lt;/p&gt;
&lt;p&gt;With pkgbase, the base system is a set of ordinary packages, and you can install a subset of them into a&amp;nbsp;directory:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# A minimal pkgbase install ships neither of these.&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-jail&lt;span class="w"&gt; &lt;/span&gt;doas

mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/jails/smokeroot
pkg&lt;span class="w"&gt; &lt;/span&gt;--rootdir&lt;span class="w"&gt; &lt;/span&gt;/jails/smokeroot&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-base&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-runtime
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;FreeBSD-runtime&lt;/code&gt; pulls in twelve packages and produces&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;33M     /jails/smokeroot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Thirty-three megabytes for a jail with a&amp;nbsp;working &lt;code&gt;/bin/sh&lt;/code&gt;, &lt;code&gt;cat&lt;/code&gt;, &lt;code&gt;ls&lt;/code&gt;, &lt;code&gt;ps&lt;/code&gt;, &lt;code&gt;hostname&lt;/code&gt;, &lt;code&gt;pw&lt;/code&gt; and the rest of the essential userland. No documentation, no debugging symbols,&amp;nbsp;no &lt;code&gt;sendmail&lt;/code&gt;, no games. For a jail whose entire purpose is to be the far end of&amp;nbsp;a &lt;code&gt;jexec&lt;/code&gt; call, that is precisely the right amount of operating&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;My plugin needs a Python interpreter in the jail as well, because that is where Ansible modules actually&amp;nbsp;run:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;--rootdir&lt;span class="w"&gt; &lt;/span&gt;/jails/smokeroot&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;python3
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the expensive part, and it is worth seeing the two numbers next to each&amp;nbsp;other:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Content&lt;/th&gt;
&lt;th style="text-align: right;"&gt;Size&lt;/th&gt;
&lt;th style="text-align: right;"&gt;Packages&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;FreeBSD-runtime&lt;/code&gt; (the entire base userland the jail needs)&lt;/td&gt;
&lt;td style="text-align: right;"&gt;33 &lt;span class="caps"&gt;MB&lt;/span&gt;&lt;/td&gt;
&lt;td style="text-align: right;"&gt;12&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;The above&amp;nbsp;plus &lt;code&gt;python3&lt;/code&gt; and its dependencies&lt;/td&gt;
&lt;td style="text-align: right;"&gt;389 &lt;span class="caps"&gt;MB&lt;/span&gt;&lt;/td&gt;
&lt;td style="text-align: right;"&gt;20&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The base system is 8 % of the jail. The Python stack is the other 92 %. I find that funny every&amp;nbsp;time.&lt;/p&gt;
&lt;h3 id="sharp-edge-one-the-fingerprints-live-inside-the-rootdir"&gt;Sharp edge one: the fingerprints live inside the&amp;nbsp;rootdir&lt;/h3&gt;
&lt;p&gt;The&amp;nbsp;first &lt;code&gt;pkg --rootdir&lt;/code&gt; run&amp;nbsp;failed:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg: Error opening the trusted directory /usr/share/keys/pkgbase-15/trusted
pkg: Error loading trusted certificates
pkg: Repository FreeBSD-base cannot be opened. &amp;#39;pkg update&amp;#39; required
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The path in that message is misleading. The keys exist&amp;nbsp;at &lt;code&gt;/usr/share/keys/pkgbase-15/trusted&lt;/code&gt; on the host — that is how the host installed itself.&amp;nbsp;But &lt;code&gt;pkg&lt;/code&gt; resolves the fingerprint path &lt;em&gt;relative to the rootdir&lt;/em&gt;, so it was looking inside an empty jail directory and reporting the path without the prefix. Copy the keys in&amp;nbsp;first:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/jails/smokeroot/usr/share/keys
cp&lt;span class="w"&gt; &lt;/span&gt;-R&lt;span class="w"&gt; &lt;/span&gt;/usr/share/keys/&lt;span class="w"&gt; &lt;/span&gt;/jails/smokeroot/usr/share/keys/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="sharp-edge-two-nobody-runs-ldconfig-for-you"&gt;Sharp edge two: nobody runs ldconfig for&amp;nbsp;you&lt;/h3&gt;
&lt;p&gt;With the jails built and started, Ansible got as far as executing a module and&amp;nbsp;then:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ld-elf.so.1: Shared object &amp;quot;libpython3.12.so.1.0&amp;quot; not found, required by &amp;quot;python3&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;pkg --rootdir&lt;/code&gt; installs files. It does not run the post-install steps that would normally happen inside the system being modified,&amp;nbsp;and &lt;code&gt;ldconfig&lt;/code&gt; is one of them. The hints file in each jail was empty, so the dynamic linker had never heard&amp;nbsp;of &lt;code&gt;/usr/local/lib&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;jexec&lt;span class="w"&gt; &lt;/span&gt;smokeroot&lt;span class="w"&gt; &lt;/span&gt;/sbin/ldconfig&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;/usr/local/lib
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Both of these are obvious in hindsight and neither is documented anywhere near where you need it, which is my excuse for writing them&amp;nbsp;down.&lt;/p&gt;
&lt;h2 id="the-smallest-jailconf-that-does-anything"&gt;The Smallest jail.conf That Does&amp;nbsp;Anything&lt;/h2&gt;
&lt;p&gt;Here is the entire jail configuration for all three&amp;nbsp;jails:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;exec.clean;
mount.devfs;
persist;
path = &amp;quot;/jails/$name&amp;quot;;
host.hostname = &amp;quot;$name&amp;quot;;

smokeroot { }
smokeuser { }
smokealias { }
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;No &lt;code&gt;exec.start&lt;/code&gt;.&amp;nbsp;No &lt;code&gt;/etc/rc&lt;/code&gt;. No &lt;span class="caps"&gt;IP&lt;/span&gt; address,&amp;nbsp;no &lt;code&gt;vnet&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;ip4.addr&lt;/code&gt;, not even a loopback alias. These jails run &lt;strong&gt;zero processes&lt;/strong&gt; when&amp;nbsp;idle. &lt;code&gt;persist&lt;/code&gt; is what keeps them alive without a single running program in them, and that is&amp;nbsp;all &lt;code&gt;jexec&lt;/code&gt; requires: a jail that&amp;nbsp;exists.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;mount.devfs&lt;/code&gt; is the one line you cannot drop for these targets. Python&amp;nbsp;wants &lt;code&gt;/dev/urandom&lt;/code&gt; at startup, and without devfs the interpreter fails in a way that looks nothing like a missing device&amp;nbsp;node.&lt;/p&gt;
&lt;p&gt;Networking is absent by design. The packages are installed from the host&amp;nbsp;with &lt;code&gt;pkg --rootdir&lt;/code&gt;, so nothing inside the jails ever needs to reach the internet. A test target that cannot phone home is a&amp;nbsp;feature.&lt;/p&gt;
&lt;p&gt;Two more lines&amp;nbsp;in &lt;code&gt;/etc/rc.conf&lt;/code&gt; make them survive a&amp;nbsp;reboot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;jail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;jail_list&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;smokeroot smokeuser smokealias&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Incidentally, this box has&amp;nbsp;no &lt;code&gt;sysrc&lt;/code&gt; — a minimal pkgbase install does not include it. Appending&amp;nbsp;to &lt;code&gt;rc.conf&lt;/code&gt; with &lt;code&gt;grep -q || echo &amp;gt;&amp;gt;&lt;/code&gt; works fine and is a useful reminder of how thin &amp;#8220;minimal&amp;#8221; really&amp;nbsp;is.&lt;/p&gt;
&lt;h2 id="three-jails-three-code-paths"&gt;Three Jails, Three Code&amp;nbsp;Paths&lt;/h2&gt;
&lt;p&gt;Rather than one generic jail, I built three, each aimed at a specific thing the plugin can get&amp;nbsp;wrong:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Jail&lt;/th&gt;
&lt;th&gt;What it covers&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;smokeroot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Baseline. Commands run&amp;nbsp;as &lt;code&gt;root&lt;/code&gt; inside the jail.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;smokeuser&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Contains an&amp;nbsp;unprivileged &lt;code&gt;tester&lt;/code&gt; user,&amp;nbsp;so &lt;code&gt;ansible_jail_user=tester&lt;/code&gt; exercises &lt;code&gt;jexec -U&lt;/code&gt; resolving against the &lt;strong&gt;jail&amp;#8217;s&lt;/strong&gt; passwd database rather than the host&amp;#8217;s.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;smokealias&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Reached under a &lt;em&gt;different&lt;/em&gt; inventory hostname&amp;nbsp;via &lt;code&gt;ansible_jail_name&lt;/code&gt;,&amp;nbsp;with &lt;code&gt;ansible_host&lt;/code&gt; pointed at the&amp;nbsp;unroutable &lt;code&gt;192.0.2.99&lt;/code&gt;. If the plugin ever confuses &amp;#8220;the address of the jail host&amp;#8221; with &amp;#8220;the name of the jail&amp;#8221;, this jail fails loudly.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The third one deserves a note. Using a &lt;span class="caps"&gt;TEST&lt;/span&gt;-&lt;span class="caps"&gt;NET&lt;/span&gt; address as a tripwire is cheap and effective: without a deliberately configured local route, a connection&amp;nbsp;to &lt;code&gt;192.0.2.99&lt;/code&gt; cannot succeed. Any code path that reaches&amp;nbsp;for &lt;code&gt;ansible_host&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;ansible_jail_host&lt;/code&gt; therefore produces an immediate, unambiguous failure rather than a subtle wrong&amp;nbsp;answer.&lt;/p&gt;
&lt;p&gt;Building the second and third jail took no downloads at all&amp;nbsp;— &lt;code&gt;cp -a&lt;/code&gt; from the first is faster than re-fetching packages, and on this box, considerably&amp;nbsp;faster.&lt;/p&gt;
&lt;h3 id="the-gotcha-that-will-bite-anyone-running-ansible-against-a-jail"&gt;The gotcha that will bite anyone running Ansible against a&amp;nbsp;jail&lt;/h3&gt;
&lt;p&gt;The first full run&amp;nbsp;against &lt;code&gt;smokeuser&lt;/code&gt; failed like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Failed to create temporary directory ...
Failed command was: ( umask 77 &amp;amp;&amp;amp; mkdir -p &amp;quot;` echo ~chofstede/.ansible/tmp `&amp;quot; ...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Ansible builds its temporary directory path from the home directory of the &lt;strong&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt;&lt;/strong&gt; user. That user exists on the jail host. It does not exist inside the jail, and a&amp;nbsp;non-root &lt;code&gt;ansible_jail_user&lt;/code&gt; cannot create directories in the&amp;nbsp;jail&amp;#8217;s &lt;code&gt;/&lt;/code&gt;. The fix is one inventory&amp;nbsp;variable:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;ansible_remote_tmp&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/tmp/.ansible-tmp&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This has nothing to do with my plugin specifically — it applies to any setup where the connection lands somewhere the &lt;span class="caps"&gt;SSH&lt;/span&gt; user does not exist. It is now in the plugin&amp;#8217;s troubleshooting table, because I would rather nobody else spend twenty minutes on&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="does-the-test-actually-catch-the-bug"&gt;Does the Test Actually Catch the&amp;nbsp;Bug?&lt;/h2&gt;
&lt;p&gt;A regression test that has never seen the regression is decoration. So before declaring victory I took a copy of the plugin, stripped out the fix, and ran the loop against a real&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;changed: [smokeroot] =&amp;gt; (item=one)
failed:  [smokeroot] (item=two)   =&amp;gt; ssh: Could not resolve hostname smokeroot
failed:  [smokeroot] (item=three) =&amp;gt; ssh: Could not resolve hostname smokeroot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There it is. Item one succeeds while the redirect&amp;nbsp;from &lt;code&gt;_connect()&lt;/code&gt; is still in place; the option reload before item two throws it away; &lt;span class="caps"&gt;SSH&lt;/span&gt; goes looking for a host&amp;nbsp;called &lt;code&gt;smokeroot&lt;/code&gt; and finds nothing. Exactly the shape the reporter described, and exactly what reading&amp;nbsp;the &lt;code&gt;ansible-core&lt;/code&gt; source&amp;nbsp;predicted.&lt;/p&gt;
&lt;p&gt;With the fix restored, all three jails pass all twenty tasks of the smoke test, including the&amp;nbsp;loop.&lt;/p&gt;
&lt;h2 id="what-it-cost"&gt;What It&amp;nbsp;Cost&lt;/h2&gt;
&lt;p&gt;Before the build, the filesystem used 1.3 &lt;span class="caps"&gt;GB&lt;/span&gt;. After three jails with a Python interpreter&amp;nbsp;each:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Filesystem      Size    Used   Avail Capacity  Mounted on
/dev/vtbd0p2    9.2G    2.4G    6.0G    29%    /

Mem: 55M Active, 188M Inact, 148M Wired, 71M Free
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is roughly 1.1 &lt;span class="caps"&gt;GB&lt;/span&gt; for all three test targets together. More importantly, the idle jails add no processes at all. The host&amp;#8217;s process count rises and falls with &lt;span class="caps"&gt;SSH&lt;/span&gt; sessions, but the jails contribute nothing until a task runs in one; when a play does run, the peak load is one Python interpreter at a&amp;nbsp;time.&lt;/p&gt;
&lt;p&gt;There is a small irony in the privilege escalation, too. This host uses &lt;a href="https://hofstede.it/mdo-on-freebsd-15-base-system-privilege-delegation-with-mac_do/"&gt;&lt;code&gt;mdo(1)&lt;/code&gt;&lt;/a&gt;, which I have written about before and which has&amp;nbsp;replaced &lt;code&gt;sudo&lt;/code&gt; on most of my machines. My own plugin only&amp;nbsp;supports &lt;code&gt;doas&lt;/code&gt;, &lt;code&gt;sudo&lt;/code&gt; or nothing at all — so to test it I&amp;nbsp;installed &lt;code&gt;doas&lt;/code&gt; on a box that had been happily living without it.&amp;nbsp;Adding &lt;code&gt;mdo&lt;/code&gt; as a fourth choice is now on the&amp;nbsp;list.&lt;/p&gt;
&lt;h2 id="takeaways"&gt;Takeaways&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;pkgbase makes disposable jails genuinely disposable.&lt;/strong&gt; 33 &lt;span class="caps"&gt;MB&lt;/span&gt; and twelve packages for a working userland changes what is worth spinning&amp;nbsp;up.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A jail does not need to run&amp;nbsp;anything.&lt;/strong&gt; &lt;code&gt;persist&lt;/code&gt; plus &lt;code&gt;mount.devfs&lt;/code&gt; and a path is a complete, useful test&amp;nbsp;target. &lt;code&gt;exec.start&lt;/code&gt; and &lt;code&gt;/etc/rc&lt;/code&gt; are for jails that provide&amp;nbsp;services.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;pkg --rootdir&lt;/code&gt; does two things you must do yourself:&lt;/strong&gt; copy the repository fingerprints into the rootdir, and&amp;nbsp;run &lt;code&gt;ldconfig&lt;/code&gt; afterwards.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Free-tier hardware is underrated for test infrastructure.&lt;/strong&gt; Half a gig of &lt;span class="caps"&gt;RAM&lt;/span&gt; is not enough for many modern service stacks; it is more than enough to hold three jails still while something else pokes at&amp;nbsp;them.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Test the test.&lt;/strong&gt; Reverting the fix and watching the regression reappear took two minutes and is the only reason I can claim the bug is actually&amp;nbsp;fixed.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The plugin fix shipped as &lt;a href="https://github.com/chofstede/ansible_jailexec/releases/tag/v2.0.2"&gt;2.0.2&lt;/a&gt;. The jails are still running, costing nothing, waiting for the next bug&amp;nbsp;report.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="jails"/><category term="pkgbase"/><category term="ansible"/><category term="jailexec"/><category term="testing"/><category term="vps"/><category term="automation"/></entry><entry><title>Building My Own TV Station: ErsatzTV in a FreeBSD Jail, With Intel Hardware Transcoding</title><link href="https://hofstede.it/building-my-own-tv-station-ersatztv-in-a-freebsd-jail-with-intel-hardware-transcoding/" rel="alternate"/><published>2026-07-26T00:00:00+02:00</published><updated>2026-07-26T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-07-26:/building-my-own-tv-station-ersatztv-in-a-freebsd-jail-with-intel-hardware-transcoding/</id><summary type="html">&lt;p&gt;Being ill for a week taught me that a large Jellyfin library is not entertainment, it is homework. ErsatzTV turns a media library back into linear television with real channels and schedules, but it only ships for Linux, Windows, and macOS. It is a .&lt;span class="caps"&gt;NET&lt;/span&gt; application driving ffmpeg, so I built it in a FreeBSD Bastille jail instead. One missing conditional was all that stood between FreeBSD and Intel &lt;span class="caps"&gt;VAAPI&lt;/span&gt; hardware&amp;nbsp;acceleration.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="The ErsatzTV web interface showing a channel preview with a Star Trek: The Next Generation episode playing" src="https://hofstede.it/images/2026-07-26-ersatztv-freebsd-jail.webp" title="My sci-fi channel, transcoded by an Intel iGPU inside a FreeBSD jail"&gt;&lt;/p&gt;
&lt;p&gt;I spent most of last week horizontal. Not dramatically ill, just the sort of illness where the day consists of tea, bed, and a laptop balanced on a duvet at an angle no ergonomics guide would&amp;nbsp;endorse.&lt;/p&gt;
&lt;p&gt;This is exactly the situation a large media library is supposed to solve. Mine holds an embarrassing number of series and films, all neatly indexed by Jellyfin, all one click away. And every single time I opened it, I scrolled. Then I scrolled some more. Then I closed it again and stared at the ceiling, which at least did not require me to make a&amp;nbsp;decision.&lt;/p&gt;
&lt;p&gt;That is the failure mode nobody warns you about when you build a media server. A library is not entertainment. A library is a catalogue, and a catalogue asks you a question every time you open it: &lt;em&gt;what do you want?&lt;/em&gt; When you are ill, tired, or simply out of decision budget for the day, that question is surprisingly&amp;nbsp;expensive.&lt;/p&gt;
&lt;p&gt;Old-fashioned television never asked. You turned it on, something was already happening, and you either watched it or you did not. The scheduling was somebody else&amp;#8217;s&amp;nbsp;problem.&lt;/p&gt;
&lt;p&gt;So I built myself a television&amp;nbsp;station.&lt;/p&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#what-ersatztv-actually-does"&gt;What ErsatzTV Actually&amp;nbsp;Does&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-this-was-worth-attempting-at-all"&gt;Why This Was Worth Attempting At&amp;nbsp;All&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-jail"&gt;The&amp;nbsp;Jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#letting-a-jail-see-the-gpu"&gt;Letting a Jail See the &lt;span class="caps"&gt;GPU&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#building-ersatztv-on-freebsd"&gt;Building ErsatzTV on&amp;nbsp;FreeBSD&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-one-line-problem"&gt;The One-Line&amp;nbsp;Problem&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#running-it"&gt;Running&amp;nbsp;It&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-station"&gt;The&amp;nbsp;Station&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#feeding-it-back-into-jellyfin"&gt;Feeding It Back Into&amp;nbsp;Jellyfin&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-this-says-about-porting"&gt;What This Says About&amp;nbsp;Porting&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="what-ersatztv-actually-does"&gt;What ErsatzTV Actually&amp;nbsp;Does&lt;/h2&gt;
&lt;p&gt;&lt;a href="https://ersatztv.org/"&gt;ErsatzTV&lt;/a&gt; takes a media library and turns it back into linear channels. You define a channel, attach collections of content to it, describe how those should be ordered and scheduled, and ErsatzTV produces a continuous stream. It exposes that stream as an &lt;span class="caps"&gt;M3U&lt;/span&gt; playlist with an &lt;span class="caps"&gt;XMLTV&lt;/span&gt; guide, which means anything that speaks &lt;span class="caps"&gt;IPTV&lt;/span&gt; can consume it, including Jellyfin&amp;nbsp;itself.&lt;/p&gt;
&lt;p&gt;It supports the things that make a channel feel like a channel: schedules with time blocks, filler content between programmes, watermarks and channel logos in the corner, and per-channel transcoding profiles. You can genuinely end up with something that looks like a broadcaster, complete with a small logo in the top right, rather than a playlist with delusions of&amp;nbsp;grandeur.&lt;/p&gt;
&lt;p&gt;Crucially for a home server, it is lazy in the right way. ErsatzTV does not transcode continuously into the void. The stream starts when a client connects and stops when the last one disconnects. A channel that nobody is watching costs nothing beyond the scheduler knowing what &lt;em&gt;would&lt;/em&gt; be playing. On a machine that is on all day anyway, that difference matters. I have no interest in heating my flat by broadcasting Star Trek to an empty&amp;nbsp;room.&lt;/p&gt;
&lt;p&gt;There was one obstacle. ErsatzTV ships binaries and containers for Linux, Windows, and macOS. My media&amp;nbsp;server, &lt;code&gt;voyager&lt;/code&gt;, runs&amp;nbsp;FreeBSD.&lt;/p&gt;
&lt;h2 id="why-this-was-worth-attempting-at-all"&gt;Why This Was Worth Attempting At&amp;nbsp;All&lt;/h2&gt;
&lt;p&gt;The usual reaction to &amp;#8220;the vendor does not support my operating system&amp;#8221; is to give up gracefully, and quite often that is correct. This case was different for two&amp;nbsp;reasons.&lt;/p&gt;
&lt;p&gt;First, ErsatzTV is a .&lt;span class="caps"&gt;NET&lt;/span&gt; application, and FreeBSD has a usable .&lt;span class="caps"&gt;NET&lt;/span&gt; toolchain in ports. Second, it does not implement video processing itself. All the heavy lifting is delegated&amp;nbsp;to &lt;code&gt;ffmpeg&lt;/code&gt; and &lt;code&gt;ffprobe&lt;/code&gt;, which FreeBSD has had for approximately forever, including &lt;span class="caps"&gt;VAAPI&lt;/span&gt; support for Intel&amp;nbsp;graphics.&lt;/p&gt;
&lt;p&gt;So the interesting question was never &amp;#8220;can FreeBSD decode this video&amp;#8221;. FreeBSD was always going to be fine at that part. The question was whether the application layer contained any genuinely Linux-specific assumptions, or merely Linux-shaped&amp;nbsp;ones.&lt;/p&gt;
&lt;p&gt;That distinction turns out to be the whole&amp;nbsp;story.&lt;/p&gt;
&lt;h2 id="the-jail"&gt;The&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;I build these experiments in a Bastille jail, because a jail is cheap, disposable, and honest about what it can reach. If the experiment&amp;nbsp;fails, &lt;code&gt;bastille destroy&lt;/code&gt; removes every trace of the .&lt;span class="caps"&gt;NET&lt;/span&gt; &lt;span class="caps"&gt;SDK&lt;/span&gt;, the NuGet caches, and my&amp;nbsp;optimism.&lt;/p&gt;
&lt;p&gt;The jail gets exactly two things from the host: the media, and the &lt;span class="caps"&gt;GPU&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;Media comes in as a read-only nullfs mount. ErsatzTV needs to read the same files Jellyfin reads, but it has absolutely no business writing to&amp;nbsp;them:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@voyager:~ # cat /usr/local/bastille/jails/ersatztv-test/fstab
/usr/local/bastille/releases/15.1-RELEASE /usr/local/bastille/jails/ersatztv-test/root/.bastille nullfs ro 0 0
/media_zfs/Series/                /usr/local/bastille/jails/ersatztv-test/root/media/Series            nullfs  ro      0       0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The first line is the standard thin-jail base mount that Bastille creates. The second is&amp;nbsp;mine. &lt;code&gt;/media_zfs/Series&lt;/code&gt; is the same dataset Jellyfin scans, mounted&amp;nbsp;at &lt;code&gt;/media/Series&lt;/code&gt; inside the jail and&amp;nbsp;marked &lt;code&gt;ro&lt;/code&gt;. Two applications now read the same library, one of them cannot modify it, and there is no second copy of&amp;nbsp;anything.&lt;/p&gt;
&lt;p&gt;If you want the full background on &lt;span class="caps"&gt;VNET&lt;/span&gt;, thin jails, and the isolation model, my &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;FreeBSD Foundationals article on Jails&lt;/a&gt; covers the mechanics properly. This article assumes the jail already&amp;nbsp;exists.&lt;/p&gt;
&lt;h2 id="letting-a-jail-see-the-gpu"&gt;Letting a Jail See the &lt;span class="caps"&gt;GPU&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;The more interesting half is hardware&amp;nbsp;acceleration. &lt;code&gt;voyager&lt;/code&gt; has Intel integrated graphics driven&amp;nbsp;by &lt;code&gt;i915kms&lt;/code&gt; from &lt;code&gt;drm-kmod&lt;/code&gt;, which gives the&amp;nbsp;host &lt;code&gt;/dev/dri/renderD128&lt;/code&gt; and friends. Software transcoding several simultaneous 1080p channels on a home server &lt;span class="caps"&gt;CPU&lt;/span&gt; is technically possible and practically unpleasant, so I wanted the iGPU inside the&amp;nbsp;jail.&lt;/p&gt;
&lt;p&gt;Jails do not get the&amp;nbsp;host&amp;#8217;s &lt;code&gt;/dev&lt;/code&gt; verbatim. They get&amp;nbsp;a &lt;code&gt;devfs&lt;/code&gt; view filtered by a ruleset, and the default jail ruleset quite deliberately hides everything interesting. So I defined my own ruleset for &lt;span class="caps"&gt;GPU&lt;/span&gt;-enabled jails&amp;nbsp;in &lt;code&gt;/etc/devfs.rules&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@voyager:~ # cat /etc/devfs.rules
[devfsrules_jail_gpu=20]
add include $devfsrules_hide_all
add include $devfsrules_unhide_basic
add include $devfsrules_unhide_login
add path &amp;#39;dri*&amp;#39; unhide
add path &amp;#39;dri/*&amp;#39; unhide
add path &amp;#39;drm*&amp;#39; unhide
add path &amp;#39;drm/*&amp;#39; unhide
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The structure here is worth reading carefully,&amp;nbsp;because &lt;code&gt;devfs&lt;/code&gt; rules are one of those subsystems that look like a firewall and behave like one too. The first line hides everything. The next two restore the basic device nodes any process expects&amp;nbsp;(&lt;code&gt;null&lt;/code&gt;, &lt;code&gt;zero&lt;/code&gt;, &lt;code&gt;random&lt;/code&gt;, and so on) plus the login-related ones. Only then do I selectively unhide the &lt;span class="caps"&gt;DRI&lt;/span&gt; and &lt;span class="caps"&gt;DRM&lt;/span&gt; device nodes. The result is a jail that can see the &lt;span class="caps"&gt;GPU&lt;/span&gt; and essentially nothing&amp;nbsp;else.&lt;/p&gt;
&lt;p&gt;Both the directories and their contents need unhiding, which is why each path appears twice, once bare and once&amp;nbsp;with &lt;code&gt;/*&lt;/code&gt;. Unhiding a directory alone gets you an empty directory and a confusing&amp;nbsp;afternoon.&lt;/p&gt;
&lt;p&gt;Attach the ruleset to the jail, restart it, and the render node shows up&amp;nbsp;inside:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;devfs_system_ruleset&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;20&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;devfs&lt;span class="w"&gt; &lt;/span&gt;restart
bastille&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;ersatztv-test&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;devfs_ruleset&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;20&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;ersatztv-test
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@ersatztv-test:/usr/local/ersatztv # ls -l /dev/dri
total 0
lrwxr-xr-x  1 root wheel  8 Jul 26 12:24 card0 -&amp;gt; ../drm/0
lrwxr-xr-x  1 root wheel 10 Jul 26 12:24 renderD128 -&amp;gt; ../drm/128
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note&amp;nbsp;that &lt;code&gt;/dev/dri&lt;/code&gt; on FreeBSD consists of symlinks&amp;nbsp;into &lt;code&gt;/dev/drm&lt;/code&gt;, which is why the ruleset has to unhide both trees.&amp;nbsp;Unhiding &lt;code&gt;dri&lt;/code&gt; alone gets you two dangling symlinks and a&amp;nbsp;puzzled &lt;code&gt;ffmpeg&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;At this point the jail has media and a &lt;span class="caps"&gt;GPU&lt;/span&gt;. Now for the&amp;nbsp;application.&lt;/p&gt;
&lt;h2 id="building-ersatztv-on-freebsd"&gt;Building ErsatzTV on&amp;nbsp;FreeBSD&lt;/h2&gt;
&lt;p&gt;Inside the jail, the prerequisites are&amp;nbsp;unremarkable:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;dotnet&lt;span class="w"&gt; &lt;/span&gt;ffmpeg&lt;span class="w"&gt; &lt;/span&gt;git
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;FreeBSD&amp;#8217;s &lt;code&gt;ffmpeg&lt;/code&gt; package is built with &lt;span class="caps"&gt;VAAPI&lt;/span&gt; support, which is the piece that&amp;nbsp;matters. &lt;code&gt;dotnet&lt;/code&gt; gives us the &lt;span class="caps"&gt;SDK&lt;/span&gt; and&amp;nbsp;runtime.&lt;/p&gt;
&lt;p&gt;Then clone the source and build it. This is where FreeBSD asks for a few concessions, all of which are about MSBuild rather than about&amp;nbsp;ErsatzTV:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@ersatztv-test:/usr/local/ersatztv # env \
  DOTNET_SYSTEM_NET_DISABLEIPV6=1 \
  DOTNET_CLI_USE_MSBUILD_SERVER=0 \
  MSBUILDDISABLENODEREUSE=1 \
  dotnet publish ErsatzTV/ErsatzTV.csproj \
    -c Release \
    --self-contained false \
    --no-restore \
    -m:1 \
    -nr:false \
    -o /usr/local/ersatztv
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Three of those knobs deserve an&amp;nbsp;explanation:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;DOTNET_SYSTEM_NET_DISABLEIPV6=1&lt;/code&gt; tells .&lt;span class="caps"&gt;NET&lt;/span&gt;&amp;#8217;s socket layer to stay on IPv4. The .&lt;span class="caps"&gt;NET&lt;/span&gt; runtime&amp;#8217;s dual-stack socket handling is not entirely at home on FreeBSD, and this avoids a class of startup and binding oddities that have nothing to do with the application&amp;nbsp;logic.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;DOTNET_CLI_USE_MSBUILD_SERVER=0&lt;/code&gt;, &lt;code&gt;MSBUILDDISABLENODEREUSE=1&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;-nr:false&lt;/code&gt; between them disable MSBuild&amp;#8217;s persistent build server and worker node reuse. Those are performance optimisations for developer workstations that rebuild the same solution forty times an hour. In a jail doing exactly one build, they add complexity and long-lived processes I do not&amp;nbsp;want.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;-m:1&lt;/code&gt; limits the build to a single node. Slower, but far easier to read when something goes&amp;nbsp;wrong.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;code&gt;--self-contained false&lt;/code&gt; keeps the output as a framework-dependent build. It relies on the&amp;nbsp;FreeBSD &lt;code&gt;dotnet&lt;/code&gt; runtime already installed rather than trying to bundle a runtime for a platform Microsoft does not publish one for. That is precisely the right call here: FreeBSD ports maintain the runtime, so let&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;The build completed without a single patch. That was already more than I&amp;nbsp;expected.&lt;/p&gt;
&lt;h2 id="the-one-line-problem"&gt;The One-Line&amp;nbsp;Problem&lt;/h2&gt;
&lt;p&gt;It ran, too. The web interface came up, it&amp;nbsp;found &lt;code&gt;ffmpeg&lt;/code&gt;, it scanned my media. Everything looked correct until I tried to enable hardware acceleration on an FFmpeg profile, and the &lt;span class="caps"&gt;VAAPI&lt;/span&gt; options simply were not there. No error, no warning, just an interface that quietly declined to offer me the&amp;nbsp;feature.&lt;/p&gt;
&lt;p&gt;The cause turned out to be exactly the kind of assumption I was hoping to&amp;nbsp;find:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@ersatztv-test:/usr/local/src/ersatztv # git diff ErsatzTV/Pages/FFmpegEditor.razor
diff --git a/ErsatzTV/Pages/FFmpegEditor.razor b/ErsatzTV/Pages/FFmpegEditor.razor
index a95719ba..e57054f8 100644
--- a/ErsatzTV/Pages/FFmpegEditor.razor
+++ b/ErsatzTV/Pages/FFmpegEditor.razor
@@ -154,7 +154,7 @@
                 }
             &amp;lt;/MudSelect&amp;gt;
         &amp;lt;/MudStack&amp;gt;
-        @if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux))
+        @if (RuntimeInformation.IsOSPlatform(OSPlatform.Linux) || RuntimeInformation.IsOSPlatform(OSPlatform.FreeBSD))
         {
             @if (_model.HardwareAcceleration is HardwareAccelerationKind.Vaapi)
             {
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the entire patch. One conditional in one Blazor&amp;nbsp;page.&lt;/p&gt;
&lt;p&gt;The logic is not wrong, exactly. It is asking &amp;#8220;is this a platform where &lt;span class="caps"&gt;VAAPI&lt;/span&gt; exists?&amp;#8221; and using &amp;#8220;is this Linux?&amp;#8221; as the test, because at the time that code was written those were the same question. &lt;span class="caps"&gt;VAAPI&lt;/span&gt; is not a Linux kernel feature though. It is a userspace &lt;span class="caps"&gt;API&lt;/span&gt;, &lt;code&gt;libva&lt;/code&gt; plus a driver, and FreeBSD supports it perfectly well&amp;nbsp;through &lt;code&gt;drm-kmod&lt;/code&gt; and the same Intel media driver. The check conflated a capability with an operating&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;.&lt;span class="caps"&gt;NET&lt;/span&gt; already knows the&amp;nbsp;difference. &lt;code&gt;OSPlatform.FreeBSD&lt;/code&gt; has been part of the runtime for years. Nothing else needed changing: not the ffmpeg argument construction, not the device path handling, not the capability probing. Once the interface was willing to &lt;em&gt;offer&lt;/em&gt; &lt;span class="caps"&gt;VAAPI&lt;/span&gt;, every layer below it worked on the first attempt, because those layers were talking&amp;nbsp;to &lt;code&gt;ffmpeg&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;ffmpeg&lt;/code&gt; had never had a problem in the first&amp;nbsp;place.&lt;/p&gt;
&lt;p&gt;This is my favourite category of porting bug. There is no missing functionality, no unavailable syscall, no incompatible library. There is a piece of code that asked the wrong question and got a technically accurate&amp;nbsp;answer.&lt;/p&gt;
&lt;h2 id="running-it"&gt;Running&amp;nbsp;It&lt;/h2&gt;
&lt;p&gt;With the patch applied and rebuilt, startup is uneventful in the best possible&amp;nbsp;way:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@ersatztv-test:/usr/local/ersatztv # env \
  DOTNET_SYSTEM_NET_DISABLEIPV6=1 \
  LIBVA_DRIVER_NAME=iHD \
  dotnet ErsatzTV.dll
[12:25:55 INF] ErsatzTV version develop
[12:25:55 WRN] Give feedback at https://github.com/ErsatzTV/ErsatzTV or https://discord.ersatztv.org
[12:25:55 INF] Database is at /root/.local/share/ersatztv/ersatztv.sqlite3
[12:25:55 INF] Transcode folder is /root/.local/share/etv-transcode
[12:25:55 INF] Using Lucene (embedded) search index backend
[12:25:55 INF] Server will listen on streaming port 8409, UI port 8409 - try UI at http://localhost:8409
[12:25:55 INF] Worker service started
[12:25:55 INF] FFmpeg worker service started
[12:25:55 INF] Applying database migrations
[12:25:58 INF] Initializing database
[12:25:59 INF] Done applying database migrations
[12:25:59 INF] Cleaning database
[12:25:59 INF] Search index worker service started
[12:25:59 INF] Emptying transcode cache folder
[12:25:59 INF] Done emptying transcode cache folder
[12:25:59 INF] Cleaning channel cache
[12:25:59 INF] Located ffmpeg at /usr/local/bin/ffmpeg
[12:25:59 INF] Located ffprobe at /usr/local/bin/ffprobe
[12:25:59 INF] Done cleaning database
[12:25:59 INF] Initializing search index
[12:25:59 INF] Done initializing search index
[12:26:00 INF] Search index is already version 49
[12:26:00 INF] Scanner service started
[12:26:00 INF] Emby service started; secrets are at /root/.local/share/ersatztv/emby-secrets.json
[12:26:00 INF] Scheduler service started
[12:26:00 INF] Jellyfin service started; secrets are at /root/.local/share/ersatztv/jellyfin-secrets.json
[12:26:00 INF] Plex service started; secrets are at /root/.local/share/ersatztv/plex-secrets.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There is nothing in that log that admits to running on an unsupported platform. It finds SQLite, ffmpeg, ffprobe, its transcode folder, and starts every worker service it expects to&amp;nbsp;start.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;LIBVA_DRIVER_NAME=iHD&lt;/code&gt; selects the Intel Media Driver rather than&amp;nbsp;letting &lt;code&gt;libva&lt;/code&gt; guess. On a machine with exactly one &lt;span class="caps"&gt;GPU&lt;/span&gt; that is usually unnecessary, but being explicit costs nothing and removes an entire category of &amp;#8220;why is it using software encoding&amp;#8221; confusion&amp;nbsp;later.&lt;/p&gt;
&lt;p&gt;One honest caveat: I am starting this by hand, from a shell, with the environment variables inline. For a real service that is not good enough. A production deployment wants a&amp;nbsp;proper &lt;code&gt;rc.d&lt;/code&gt; script so the thing can be enabled&amp;nbsp;with &lt;code&gt;sysrc ersatztv_enable=YES&lt;/code&gt;, controlled&amp;nbsp;with &lt;code&gt;service ersatztv start|stop|restart&lt;/code&gt;, run as an unprivileged user rather than root, and brought up automatically when the jail boots. This was an experiment to find out whether ErsatzTV runs on FreeBSD at all, so I have not written that script yet. It is the obvious next&amp;nbsp;step.&lt;/p&gt;
&lt;p&gt;The relevant part is what happens when a channel actually starts. The transcoding pipeline picks up the render node&amp;nbsp;in &lt;code&gt;/dev/dri&lt;/code&gt;, hands the work to the iGPU, and the &lt;span class="caps"&gt;CPU&lt;/span&gt; graph stays flat. Which is the point of the whole&amp;nbsp;exercise.&lt;/p&gt;
&lt;h2 id="the-station"&gt;The&amp;nbsp;Station&lt;/h2&gt;
&lt;p&gt;With the infrastructure working, I got to do the fun part: programming a television&amp;nbsp;channel.&lt;/p&gt;
&lt;p&gt;Mine is a science fiction channel, assembled from what my library already&amp;nbsp;contained:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Series&lt;/th&gt;
&lt;th&gt;Why it earns its slot&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Star Trek: &lt;span class="caps"&gt;TOS&lt;/span&gt;, &lt;span class="caps"&gt;TNG&lt;/span&gt;, &lt;span class="caps"&gt;DS9&lt;/span&gt;, &lt;span class="caps"&gt;VOY&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;Several hundred episodes, near-uniform runtime, no continuity obligations&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stargate &lt;span class="caps"&gt;SG&lt;/span&gt;-1&lt;/td&gt;
&lt;td&gt;Reliably episodic, tonally consistent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Doctor Who&lt;/td&gt;
&lt;td&gt;Wildly variable, which is its own kind of variety&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Farscape&lt;/td&gt;
&lt;td&gt;The one that keeps the mix from getting too comfortable&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The characteristics that make a series good television-station material are not the same ones that make it good binge material. What you want is a large episode count, consistent runtimes, and stories that mostly resolve within the hour. Ninety-eight percent of classic Trek qualifies. A tightly serialised prestige drama would be terrible for this, because arriving in the middle of episode six of a nine-part arc is not relaxing, it is homework with better production&amp;nbsp;values.&lt;/p&gt;
&lt;p&gt;ErsatzTV then handles the ordering. I can shuffle within a collection, play a series in order, block out time ranges, and let the scheduler produce something that feels curated rather than randomised. The channel logo in the corner is a small touch that does a disproportionate amount of work for the&amp;nbsp;illusion.&lt;/p&gt;
&lt;h2 id="feeding-it-back-into-jellyfin"&gt;Feeding It Back Into&amp;nbsp;Jellyfin&lt;/h2&gt;
&lt;p&gt;The last step closes the loop nicely. ErsatzTV publishes an &lt;span class="caps"&gt;M3U&lt;/span&gt; playlist and an &lt;span class="caps"&gt;XMLTV&lt;/span&gt; guide, and Jellyfin has a Live &lt;span class="caps"&gt;TV&lt;/span&gt; feature that consumes exactly&amp;nbsp;those.&lt;/p&gt;
&lt;p&gt;So the channel is added to Jellyfin as a tuner. The same interface that was making me scroll endlessly now has a Live &lt;span class="caps"&gt;TV&lt;/span&gt; section with a small set of channels that are simply &lt;em&gt;on&lt;/em&gt;. I do not browse them. I open one, and something is already&amp;nbsp;happening.&lt;/p&gt;
&lt;p&gt;The data path is pleasantly&amp;nbsp;circular:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;   ZFS dataset  /media_zfs/Series
        |
        +--&amp;gt; Jellyfin jail          (library, on-demand browsing)
        |
        +--&amp;gt; ErsatzTV jail (ro)     (scheduling + VAAPI transcoding)
                  |
                  | M3U + XMLTV
                  v
             Jellyfin Live TV       (channels that are just on)
                  |
                  v
              TV / phone / laptop
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;One dataset, read by two applications, presented in two different ways: a catalogue for the times when I know what I want, and a channel for the times when I do&amp;nbsp;not.&lt;/p&gt;
&lt;p&gt;And because ErsatzTV only transcodes while somebody is watching, the second mode costs nothing when I am not using it. Turn the &lt;span class="caps"&gt;TV&lt;/span&gt; off and the station goes dark. That is a far better default than the real broadcast industry ever&amp;nbsp;managed.&lt;/p&gt;
&lt;h2 id="what-this-says-about-porting"&gt;What This Says About&amp;nbsp;Porting&lt;/h2&gt;
&lt;p&gt;The technically interesting result here is how little there was to&amp;nbsp;do.&lt;/p&gt;
&lt;p&gt;FreeBSD had the runtime, in ports. FreeBSD&amp;nbsp;had &lt;code&gt;ffmpeg&lt;/code&gt; with &lt;span class="caps"&gt;VAAPI&lt;/span&gt;, in ports. FreeBSD had the Intel graphics driver,&amp;nbsp;in &lt;code&gt;drm-kmod&lt;/code&gt;. FreeBSD&amp;#8217;s jail&amp;nbsp;and &lt;code&gt;devfs&lt;/code&gt; machinery gave me a cleaner device isolation story than a container would have. Every hard piece of this problem was solved before I started, by people who did the work years&amp;nbsp;ago.&lt;/p&gt;
&lt;p&gt;The only thing standing between all of that and a working application was a&amp;nbsp;single &lt;code&gt;if&lt;/code&gt; that asked whether the operating system was Linux when it meant to ask whether &lt;span class="caps"&gt;VAAPI&lt;/span&gt; was available. That is not a criticism of ErsatzTV, which is a genuinely lovely piece of software and generous with its debug output. It is just what happens when a capability is common enough on one platform that the platform becomes shorthand for the&amp;nbsp;capability.&lt;/p&gt;
&lt;p&gt;It is worth remembering when your own code reaches for a platform&amp;nbsp;check. &lt;code&gt;IsOSPlatform(Linux)&lt;/code&gt; is rarely the actual question. The actual question is usually &amp;#8220;does this device node exist&amp;#8221;, &amp;#8220;is this library present&amp;#8221;, or &amp;#8220;does this &lt;span class="caps"&gt;API&lt;/span&gt; return something sensible&amp;#8221;. Those questions have the pleasant property of being true on platforms nobody thought to&amp;nbsp;test.&lt;/p&gt;
&lt;p&gt;I was ill, bored, and mildly annoyed at my own media library. A week later I have a science fiction channel that runs on an operating system the project does not support, on hardware acceleration the interface was not willing to admit existed, in a jail that can see precisely one &lt;span class="caps"&gt;GPU&lt;/span&gt; and one read-only&amp;nbsp;directory.&lt;/p&gt;
&lt;p&gt;I still do not know what is on tonight. That is entirely the&amp;nbsp;point. &lt;code&gt;^_^&lt;/code&gt;&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="bastille"/><category term="jails"/><category term="ersatztv"/><category term="jellyfin"/><category term="dotnet"/><category term="ffmpeg"/><category term="vaapi"/><category term="i915kms"/><category term="self-hosting"/><category term="iptv"/></entry><entry><title>My Home Assistant Setup: bhyve, FreeBSD Jails, Zigbee, Homematic, and Matter over Thread</title><link href="https://hofstede.it/my-home-assistant-setup-bhyve-freebsd-jails-zigbee-homematic-and-matter-over-thread/" rel="alternate"/><published>2026-07-18T00:00:00+02:00</published><updated>2026-07-18T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-07-18:/my-home-assistant-setup-bhyve-freebsd-jails-zigbee-homematic-and-matter-over-thread/</id><summary type="html">&lt;p&gt;My Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt; installation runs as a bhyve &lt;span class="caps"&gt;VM&lt;/span&gt; on a FreeBSD home server, while Zigbee2MQTT and Mosquitto live in a &lt;span class="caps"&gt;VNET&lt;/span&gt; jail with direct access to a ConBee &lt;span class="caps"&gt;II&lt;/span&gt; &lt;span class="caps"&gt;USB&lt;/span&gt; adapter. Homematic comes through a &lt;span class="caps"&gt;CCU3&lt;/span&gt;, and three recent &lt;span class="caps"&gt;IKEA&lt;/span&gt; Matter-over-Thread devices use the Nanoleaf Shapes controller I already owned as their border&amp;nbsp;router.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="My Home Assistant dashboard, with weather and heating on the left, climate data in the middle, and lights and energy on the right" src="https://hofstede.it/images/2026-07-18-home-assistant-freebsd-dashboard.webp" title="One dashboard hiding three radio ecosystems and a slightly unusual FreeBSD architecture"&gt;&lt;/p&gt;
&lt;p&gt;The useful part of a home automation dashboard is that it hides the plumbing. The screenshot above shows weather, heating, doors, windows, temperature, humidity, air quality, lights, and energy consumption. A light is a light, a thermostat is a thermostat, and a water sensor is either dry or it is about to make my evening substantially&amp;nbsp;worse.&lt;/p&gt;
&lt;p&gt;Behind that dashboard, however, are three quite different device&amp;nbsp;ecosystems.&lt;/p&gt;
&lt;p&gt;Most of the apartment uses Zigbee devices from Philips, &lt;span class="caps"&gt;IKEA&lt;/span&gt;, Xiaomi, Sonoff, and a few others. The heating and several door and presence sensors are Homematic or Homematic &lt;span class="caps"&gt;IP&lt;/span&gt; devices connected to an eQ-3 &lt;span class="caps"&gt;CCU3&lt;/span&gt;. The newest additions are &lt;span class="caps"&gt;IKEA&lt;/span&gt; Matter-over-Thread devices which, somewhat unexpectedly, use my Nanoleaf Shapes controller as their Thread border&amp;nbsp;router.&lt;/p&gt;
&lt;p&gt;Home Assistant ties all of this together. The slightly unusual part is where each component runs: Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt; lives in a bhyve virtual machine on my FreeBSD home server, while Zigbee2MQTT and Mosquitto run in a separate &lt;span class="caps"&gt;VNET&lt;/span&gt; jail on the host because bhyve does not offer the &lt;span class="caps"&gt;USB&lt;/span&gt; passthrough I would need for the ConBee &lt;span class="caps"&gt;II&lt;/span&gt;&amp;nbsp;adapter.&lt;/p&gt;
&lt;p&gt;This is not the architecture I would draw if every hypervisor feature were available. It is the architecture that works, survives upgrades, and has turned out to be pleasantly easy to reason&amp;nbsp;about.&lt;/p&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-shape-of-the-setup"&gt;The Shape of the&amp;nbsp;Setup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#home-assistant-os-in-bhyve"&gt;Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt; in&amp;nbsp;bhyve&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-usb-problem-becomes-a-jail"&gt;The &lt;span class="caps"&gt;USB&lt;/span&gt; Problem Becomes a&amp;nbsp;Jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#zigbee-is-still-the-workhorse"&gt;Zigbee Is Still the&amp;nbsp;Workhorse&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#homematic-the-older-very-reliable-side"&gt;Homematic: The Older, Very Reliable&amp;nbsp;Side&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#matter-over-thread-via-a-nanoleaf-controller-apparently"&gt;Matter over Thread via a Nanoleaf Controller,&amp;nbsp;Apparently&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-dashboard-is-an-interface-not-an-inventory"&gt;The Dashboard Is an Interface, Not an&amp;nbsp;Inventory&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#automations-small-things-that-remove-small-irritations"&gt;Automations: Small Things That Remove Small&amp;nbsp;Irritations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-i-would-keep-if-i-rebuilt-it"&gt;What I Would Keep If I Rebuilt&amp;nbsp;It&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-shape-of-the-setup"&gt;The Shape of the&amp;nbsp;Setup&lt;/h2&gt;
&lt;p&gt;At a high level, the system looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                                +----------------------------+
                                | Home Assistant OS          |
                                | bhyve VM, 2 vCPU, 2 GiB    |
                                |                            |
                                | Home Assistant Core        |
                                | Matter Server              |
                                | dashboards + automations   |
                                +-----+-------------+--------+
                                      |             |
                                   MQTT|             |LAN
                                      |             |
            +-------------------------+             +------------------+
            |                                                            |
            v                                                            v
+-----------------------------+                         +-----------------------------+
| FreeBSD VNET jail           |                         | eQ-3 CCU3                   |
|                             |                         |                             |
| Mosquitto                   |                         | Homematic + Homematic IP    |
| Zigbee2MQTT                 |                         | BidCoS-RF + HmIP-RF         |
| ConBee II via host USB      |                         +-----------------------------+
+-------------+---------------+
              |
              | Zigbee
              v
  lights, switches, sensors,
  plugs, remotes, repeater

              IKEA Matter-over-Thread devices
                            |
                            | Thread
                            v
              +-----------------------------+
              | Nanoleaf Shapes controller  |
              | Thread border router        |
              +-----------------------------+
                            |
                            | IPv6 over LAN
                            +-----------------------&amp;gt; Home Assistant
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There is no single &amp;#8220;smart home protocol&amp;#8221; here. Zigbee, Homematic, classic BidCoS-&lt;span class="caps"&gt;RF&lt;/span&gt;, Homematic &lt;span class="caps"&gt;IP&lt;/span&gt;, Matter, Thread, &lt;span class="caps"&gt;MQTT&lt;/span&gt;, and normal &lt;span class="caps"&gt;IP&lt;/span&gt; networking all participate. Home Assistant&amp;#8217;s job is to turn those into one coherent set of entities and&amp;nbsp;actions.&lt;/p&gt;
&lt;p&gt;The current installation has 474 entities across seven areas. That number includes the usual supporting entities such as battery levels, firmware updates, event channels, radio diagnostics, and device configuration. The dashboard deliberately exposes only the useful&amp;nbsp;surface.&lt;/p&gt;
&lt;h2 id="home-assistant-os-in-bhyve"&gt;Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt; in&amp;nbsp;bhyve&lt;/h2&gt;
&lt;p&gt;Home Assistant itself runs as Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt; rather than as a Python environment or a manually managed container. On the FreeBSD host, the &lt;span class="caps"&gt;VM&lt;/span&gt; is intentionally&amp;nbsp;small:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@voyager:~ # vm list | grep kes
kes       default    uefi       2    2G      -    Yes [3]  Running (6256)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is two virtual CPUs and 2 GiB of memory. At the time of writing it runs Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt; 18.1, Home Assistant Core 2026.7.2, and the 2026.07 supervisor. Home Assistant correctly identifies the virtualization platform as&amp;nbsp;bhyve.&lt;/p&gt;
&lt;p&gt;I like Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt; for this role because it gives the application the environment it expects. Core, the supervisor, the Matter server, backups, and upgrades all remain on the supported path. FreeBSD only has to provide a reliable &lt;span class="caps"&gt;VM&lt;/span&gt;, storage, and networking. I do not have to reproduce the Home Assistant appliance model out of hand-built&amp;nbsp;containers.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;VM&lt;/span&gt; is also a useful failure boundary. Home automation software changes much faster than the FreeBSD host below it. Home Assistant can update on its own cadence without turning the home server into an experimental Python environment, while &lt;span class="caps"&gt;ZFS&lt;/span&gt; and the host&amp;#8217;s normal backup strategy cover the &lt;span class="caps"&gt;VM&lt;/span&gt; from&amp;nbsp;below.&lt;/p&gt;
&lt;p&gt;There is one important limitation: &lt;span class="caps"&gt;USB&lt;/span&gt;.&lt;/p&gt;
&lt;h2 id="the-usb-problem-becomes-a-jail"&gt;The &lt;span class="caps"&gt;USB&lt;/span&gt; Problem Becomes a&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;My Zigbee network uses a Dresden Elektronik ConBee &lt;span class="caps"&gt;II&lt;/span&gt; &lt;span class="caps"&gt;USB&lt;/span&gt; adapter. Passing that adapter directly into a bhyve guest is not straightforward, but exposing &lt;span class="caps"&gt;USB&lt;/span&gt; devices to a FreeBSD jail through devfs is. The practical solution is to move the part which needs the &lt;span class="caps"&gt;USB&lt;/span&gt; device out of the &lt;span class="caps"&gt;VM&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;z2m&lt;/code&gt; &lt;span class="caps"&gt;VNET&lt;/span&gt; jail sees the adapter&amp;nbsp;directly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@voyager:~ # bastille cmd z2m usbconfig | grep -i con

[z2m]:
ugen0.3: &amp;lt;ZigBee gateway [ConBee II] Dresden Elektronik&amp;gt; at usbus0,
         cfg=0 md=HOST spd=FULL (12Mbps) pwr=ON (100mA)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Zigbee2MQTT talks to the ConBee &lt;span class="caps"&gt;II&lt;/span&gt; inside that jail. Mosquitto runs beside it, and Home Assistant connects to the broker over the &lt;span class="caps"&gt;LAN&lt;/span&gt;. The radio never needs to enter the &lt;span class="caps"&gt;VM&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ConBee II -&amp;gt; FreeBSD devfs -&amp;gt; Zigbee2MQTT -&amp;gt; Mosquitto -&amp;gt; Home Assistant
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The relevant processes are exactly what I expect to&amp;nbsp;see:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;nobody  mosquitto -c /usr/local/etc/mosquitto/mosquitto.conf -d
z2m     daemon: /usr/local/bin/zigbee2mqtt-start
z2m     /usr/local/bin/node index.js
root    /usr/local/bin/caddy run --config /usr/local/etc/caddy
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The jail has its own network stack, address, services, logs, and package lifecycle. From Home Assistant&amp;#8217;s perspective, none of this is unusual: it sees a normal &lt;span class="caps"&gt;MQTT&lt;/span&gt; broker and devices discovered by Zigbee2MQTT. From FreeBSD&amp;#8217;s perspective, the untrusted and comparatively fast-moving Node.js application is contained in a jail with only the &lt;span class="caps"&gt;USB&lt;/span&gt; and network access it actually&amp;nbsp;needs.&lt;/p&gt;
&lt;p&gt;This workaround turned into a design I now prefer. The radio layer does not have to share the Home Assistant lifecycle. Restarting or upgrading the &lt;span class="caps"&gt;VM&lt;/span&gt; does not detach the coordinator from the host, and Zigbee2MQTT remains an independently inspectable service. If Home Assistant disappeared tomorrow, the broker and Zigbee network would still be understandable from ordinary &lt;span class="caps"&gt;MQTT&lt;/span&gt;&amp;nbsp;topics.&lt;/p&gt;
&lt;h2 id="zigbee-is-still-the-workhorse"&gt;Zigbee Is Still the&amp;nbsp;Workhorse&lt;/h2&gt;
&lt;p&gt;Zigbee makes up the majority of the installation. Home Assistant currently reports 24 devices through the &lt;span class="caps"&gt;MQTT&lt;/span&gt; integration, including the Zigbee2MQTT bridge and two Zigbee&amp;nbsp;groups.&lt;/p&gt;
&lt;p&gt;The device mix is exactly why I prefer an open coordinator over a stack of vendor&amp;nbsp;bridges:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Philips Hue bulbs and dimmer&amp;nbsp;switches&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;IKEA&lt;/span&gt; lights, a smart plug, motion sensor, remote, and signal&amp;nbsp;repeater&lt;/li&gt;
&lt;li&gt;Xiaomi temperature and humidity&amp;nbsp;sensors&lt;/li&gt;
&lt;li&gt;Sonoff temperature and humidity&amp;nbsp;sensors&lt;/li&gt;
&lt;li&gt;the ConBee &lt;span class="caps"&gt;II&lt;/span&gt; coordinator&amp;nbsp;itself&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The three filament bulbs over the living-room ceiling are a Zigbee2MQTT group, as are the office lights. Home Assistant therefore gets one useful light entity for the room while the individual bulbs remain available when I need them. Brightness controls in the dashboard work the same way whether the tile targets one bulb or a&amp;nbsp;group.&lt;/p&gt;
&lt;p&gt;The mesh has also been boring in the best sense of the word. Mains-powered bulbs, plugs, and the &lt;span class="caps"&gt;IKEA&lt;/span&gt; repeater provide routing; battery-powered sensors and remotes can stay asleep most of the time. Zigbee2MQTT exposes firmware, battery, link, effect, and power-on behaviour entities without turning the normal dashboard into a device-management&amp;nbsp;console.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;MQTT&lt;/span&gt; is the seam that makes the FreeBSD split work. It is also a very good debugging boundary. When a device misbehaves, I can determine whether the radio event reached Zigbee2MQTT, whether the corresponding &lt;span class="caps"&gt;MQTT&lt;/span&gt; message exists, and whether Home Assistant consumed it. &amp;#8220;The light does not work&amp;#8221; becomes three small questions instead of one mysterious&amp;nbsp;stack.&lt;/p&gt;
&lt;h2 id="homematic-the-older-very-reliable-side"&gt;Homematic: The Older, Very Reliable&amp;nbsp;Side&lt;/h2&gt;
&lt;p&gt;The second device family is eQ-3 Homematic. I use both classic Homematic and Homematic &lt;span class="caps"&gt;IP&lt;/span&gt; devices, connected to a &lt;span class="caps"&gt;CCU3&lt;/span&gt; base station and integrated locally into Home&amp;nbsp;Assistant.&lt;/p&gt;
&lt;p&gt;Home Assistant currently sees 17 devices through the Homematic &lt;span class="caps"&gt;IP&lt;/span&gt; Local integration. Some are the &lt;span class="caps"&gt;CCU3&lt;/span&gt; and its virtual receiver devices, while the physical collection&amp;nbsp;includes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;radiator thermostats in the living room, bedroom, and&amp;nbsp;office&lt;/li&gt;
&lt;li&gt;door and window&amp;nbsp;contacts&lt;/li&gt;
&lt;li&gt;an office presence&amp;nbsp;sensor&lt;/li&gt;
&lt;li&gt;metering and switching plugs for the office computer, air conditioner, and&amp;nbsp;television&lt;/li&gt;
&lt;li&gt;an older remote&amp;nbsp;control&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is the unglamorous part of the smart home, and arguably the most important. Door contacts should report a door. Heating should continue to be heating. I do not need either category to become&amp;nbsp;exciting.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;CCU3&lt;/span&gt; is a separate appliance because that is what the Homematic ecosystem expects, and I am fine with that. Home Assistant talks to it locally, turns its devices into the&amp;nbsp;same &lt;code&gt;binary_sensor&lt;/code&gt;, &lt;code&gt;climate&lt;/code&gt;, &lt;code&gt;switch&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;sensor&lt;/code&gt; entities used everywhere else, and leaves the radio-specific details at the integration&amp;nbsp;boundary.&lt;/p&gt;
&lt;p&gt;That normalization is visible on the dashboard. The heating tiles and door/window tiles do not advertise that they come from a completely different base station than the Zigbee lights next to them. They simply expose the controls and states I care&amp;nbsp;about.&lt;/p&gt;
&lt;h2 id="matter-over-thread-via-a-nanoleaf-controller-apparently"&gt;Matter over Thread via a Nanoleaf Controller,&amp;nbsp;Apparently&lt;/h2&gt;
&lt;p&gt;The newest part of the system is also the one I did not originally&amp;nbsp;plan.&lt;/p&gt;
&lt;p&gt;I already had a Nanoleaf Shapes installation in the living room. At some point I discovered that its controller can act as a Thread border router. That meant I could add Matter-over-Thread devices without buying another dedicated radio or attaching another &lt;span class="caps"&gt;USB&lt;/span&gt; adapter to the Home Assistant&amp;nbsp;machine.&lt;/p&gt;
&lt;p&gt;The current Matter collection is three &lt;span class="caps"&gt;IKEA&lt;/span&gt;&amp;nbsp;devices:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Device&lt;/th&gt;
&lt;th&gt;&lt;span class="caps"&gt;IKEA&lt;/span&gt; model&lt;/th&gt;
&lt;th&gt;What Home Assistant gets&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Office air-quality sensor&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;ALPSTUGA&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;temperature, humidity, &lt;span class="caps"&gt;CO2&lt;/span&gt;, &lt;span class="caps"&gt;PM2&lt;/span&gt;.5, air quality, and diagnostics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Desk switch&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;BILRESA&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;two button events, battery state, and Thread diagnostics&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kitchen water-leak sensor&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;KLIPPBOK&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;leak state, battery state, and Thread diagnostics&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Home Assistant&amp;#8217;s Matter integration and Matter Server manage the devices, while the Nanoleaf controller provides the path from the Thread mesh to the normal IPv6 network. There is no ConBee &lt;span class="caps"&gt;II&lt;/span&gt; involved and no &lt;span class="caps"&gt;MQTT&lt;/span&gt; translation layer. Thread carries IPv6, the border router forwards it, and Matter provides the application&amp;nbsp;protocol.&lt;/p&gt;
&lt;p&gt;I expected this to be the fragile, standards-committee-shaped corner of the setup. So far it has worked remarkably well. The &lt;span class="caps"&gt;ALPSTUGA&lt;/span&gt; data feeds the office air-quality section of the dashboard. The &lt;span class="caps"&gt;BILRESA&lt;/span&gt; switch controls the office ceiling light. The &lt;span class="caps"&gt;KLIPPBOK&lt;/span&gt; sensor sits in the kitchen and has a dedicated water-leak alarm&amp;nbsp;automation.&lt;/p&gt;
&lt;p&gt;Most importantly, I did not have to replace the Nanoleaf system or buy a new border router. A controller already hanging on the wall turned out to have a second, genuinely useful job. Yes, that actually&amp;nbsp;works. &lt;code&gt;^_^&lt;/code&gt;&lt;/p&gt;
&lt;p&gt;This does not make me ready to replace every Zigbee device with Thread. Zigbee is mature, cheap, and already deployed throughout the apartment. Matter over Thread is simply another useful option, especially for devices which expose a good native Matter&amp;nbsp;implementation.&lt;/p&gt;
&lt;h2 id="the-dashboard-is-an-interface-not-an-inventory"&gt;The Dashboard Is an Interface, Not an&amp;nbsp;Inventory&lt;/h2&gt;
&lt;p&gt;The &amp;#8220;Zuhause Neu&amp;#8221; dashboard is a storage-mode Lovelace dashboard using the modern sections layout. It has one overview with three&amp;nbsp;columns:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;weather, heating, doors and windows, the kitchen water sensor, and rubbish&amp;nbsp;collection&lt;/li&gt;
&lt;li&gt;temperature, humidity, 24-hour history, and office air&amp;nbsp;quality&lt;/li&gt;
&lt;li&gt;lights and seven-day energy&amp;nbsp;consumption&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The cards are mostly built-in heading, tile, grid, history graph, weather forecast, and statistics graph cards. I use tile features for thermostat set points, heating modes, and light brightness. The result is not exotic, but it is responsive, easy to edit in Home Assistant, and does not depend on a large pile of custom frontend&amp;nbsp;code.&lt;/p&gt;
&lt;p&gt;More importantly, the dashboard does not attempt to display all 474 entities. A device might contribute a battery voltage, firmware update entity, Thread channel, restart counter, identify button, hardware error flag, and half a dozen configuration values. Those are useful when diagnosing the device. They are noise when I want to know whether the kitchen is&amp;nbsp;leaking.&lt;/p&gt;
&lt;p&gt;The main view therefore answers ordinary&amp;nbsp;questions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Is anybody&amp;nbsp;home?&lt;/li&gt;
&lt;li&gt;Is a door or window&amp;nbsp;open?&lt;/li&gt;
&lt;li&gt;What is the temperature and humidity in each&amp;nbsp;room?&lt;/li&gt;
&lt;li&gt;Is the office air still&amp;nbsp;good?&lt;/li&gt;
&lt;li&gt;What are the lights&amp;nbsp;doing?&lt;/li&gt;
&lt;li&gt;How much energy did the air conditioner and computer&amp;nbsp;use?&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Device administration remains in Home Assistant&amp;#8217;s device and entity pages. The dashboard is allowed to be an&amp;nbsp;interface.&lt;/p&gt;
&lt;h2 id="automations-small-things-that-remove-small-irritations"&gt;Automations: Small Things That Remove Small&amp;nbsp;Irritations&lt;/h2&gt;
&lt;p&gt;The installation currently has a dozen automations. They are not an attempt to make the apartment sentient. Most solve one narrow&amp;nbsp;problem:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;turn the hall light on when the motion sensor fires, then turn it off after the hall is&amp;nbsp;empty&lt;/li&gt;
&lt;li&gt;turn the office light on and off based on&amp;nbsp;presence&lt;/li&gt;
&lt;li&gt;let the Matter desk switch control the Zigbee office ceiling&amp;nbsp;light&lt;/li&gt;
&lt;li&gt;switch the heating off at&amp;nbsp;night&lt;/li&gt;
&lt;li&gt;mirror the living-room ceiling state to the Nanoleaf&amp;nbsp;Shapes&lt;/li&gt;
&lt;li&gt;send notifications for weak&amp;nbsp;batteries&lt;/li&gt;
&lt;li&gt;raise an alarm when the kitchen water sensor detects a&amp;nbsp;leak&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That desk-switch automation is a good example of what Home Assistant buys me. A Matter-over-Thread button from &lt;span class="caps"&gt;IKEA&lt;/span&gt; controls a Zigbee group of Philips bulbs, with a Nanoleaf controller providing the Thread border router and a ConBee &lt;span class="caps"&gt;II&lt;/span&gt; in a FreeBSD jail providing the Zigbee coordinator. None of those vendors designed the complete path. Home Assistant makes it one event and one light&amp;nbsp;action.&lt;/p&gt;
&lt;p&gt;The useful smart home is mostly made of connections like that: small, local, comprehensible, and easy to override with a physical&amp;nbsp;switch.&lt;/p&gt;
&lt;h2 id="what-i-would-keep-if-i-rebuilt-it"&gt;What I Would Keep If I Rebuilt&amp;nbsp;It&lt;/h2&gt;
&lt;p&gt;If I started again, I would keep the important&amp;nbsp;boundaries:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Home Assistant &lt;span class="caps"&gt;OS&lt;/span&gt; in a &lt;span class="caps"&gt;VM&lt;/span&gt;.&lt;/strong&gt; It gets the supported appliance environment; FreeBSD gets a clean guest&amp;nbsp;boundary.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The &lt;span class="caps"&gt;USB&lt;/span&gt; radio on the FreeBSD side.&lt;/strong&gt; A jail is a natural place for a host-attached coordinator when the hypervisor cannot pass it&amp;nbsp;through.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;MQTT&lt;/span&gt; between Zigbee2MQTT and Home Assistant.&lt;/strong&gt; It is an open, observable interface rather than an internal implementation&amp;nbsp;detail.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The &lt;span class="caps"&gt;CCU3&lt;/span&gt; as the Homematic radio authority.&lt;/strong&gt; It is reliable, local, and already owns that&amp;nbsp;ecosystem.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The existing Thread border router.&lt;/strong&gt; If the Nanoleaf controller already provides a healthy Thread network, adding another border router is not automatically an&amp;nbsp;improvement.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A deliberately selective dashboard.&lt;/strong&gt; Device capabilities belong in the registry; daily decisions belong on the&amp;nbsp;overview.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I would also keep the system local. Cloud services may add optional features, but lights, heating, sensors, dashboards, and automations should not depend on an external account remaining reachable. Every important integration in this article communicates on the local&amp;nbsp;network.&lt;/p&gt;
&lt;p&gt;The setup is not minimal. It contains a &lt;span class="caps"&gt;VM&lt;/span&gt;, a &lt;span class="caps"&gt;VNET&lt;/span&gt; jail, a &lt;span class="caps"&gt;USB&lt;/span&gt; coordinator, an &lt;span class="caps"&gt;MQTT&lt;/span&gt; broker, a &lt;span class="caps"&gt;CCU3&lt;/span&gt;, a Matter server, and a Nanoleaf border router. But each component has one clear responsibility, and each boundary uses a protocol I can&amp;nbsp;inspect.&lt;/p&gt;
&lt;p&gt;That is a trade I will happily make. The dashboard stays simple because the architecture underneath it is explicit. Three radio ecosystems, several vendors, and 474 entities end up looking like one home rather than a collection of incompatible product&amp;nbsp;demos.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="home-assistant"/><category term="bhyve"/><category term="jails"/><category term="zigbee2mqtt"/><category term="mqtt"/><category term="homematic"/><category term="matter"/><category term="thread"/><category term="iot"/></entry><entry><title>Hosting DokuWiki on FreeBSD with Bastille, nginx, and Caddy</title><link href="https://hofstede.it/hosting-dokuwiki-on-freebsd-with-bastille-nginx-and-caddy/" rel="alternate"/><published>2026-07-10T00:00:00+02:00</published><updated>2026-07-10T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-07-10:/hosting-dokuwiki-on-freebsd-with-bastille-nginx-and-caddy/</id><summary type="html">&lt;p&gt;A small, database-free DokuWiki deployment in a FreeBSD 15.1 Bastille jail, with nginx and &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt; inside the jail and Caddy handling &lt;span class="caps"&gt;TLS&lt;/span&gt; and access policy at the&amp;nbsp;edge.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A DokuWiki page in the dark theme showing a MikroTik RouterOS BGP debugging cheatsheet" src="https://hofstede.it/images/2026-07-10-hosting-dokuwiki-freebsd-bastille.webp" title="The DokuWiki instance running in its Bastille jail"&gt;&lt;/p&gt;
&lt;p&gt;Some services do not need PostgreSQL, Redis, an object store, three containers, and a deployment manifest long enough to qualify as literature. Sometimes I need a wiki: a place for notes, runbooks, snippets, and the sort of documentation that is useful precisely because it is not&amp;nbsp;public.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.dokuwiki.org/dokuwiki"&gt;DokuWiki&lt;/a&gt; fits that job unusually well. It stores pages as plain text files, needs no database, has a mature &lt;span class="caps"&gt;ACL&lt;/span&gt; system, and is available as a regular FreeBSD package. Put it in a Bastille jail and the entire application becomes a small, inspectable service: nginx, &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt;, and a directory tree I can back up with ordinary &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;nbsp;tooling.&lt;/p&gt;
&lt;p&gt;This follows the same pattern as my &lt;a href="https://hofstede.it/hosting-a-static-blog-on-freebsd-with-bastille-jails-and-automated-deployment/"&gt;blog infrastructure&lt;/a&gt; and &lt;a href="https://hofstede.it/self-hosted-cryptpad-on-freebsd-with-vnet-jails-and-caddy/"&gt;CryptPad deployment&lt;/a&gt;: the application lives in its own jail, while a separate Caddy jail is the only public web frontend. If the networking below feels too compressed, the &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;FreeBSD Foundationals article on Jails&lt;/a&gt; covers &lt;span class="caps"&gt;VNET&lt;/span&gt;, epairs, bridges, and the isolation model in&amp;nbsp;detail.&lt;/p&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-shape-of-the-setup"&gt;The Shape of the&amp;nbsp;Setup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#creating-the-jail"&gt;Creating the&amp;nbsp;Jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#installing-dokuwiki-from-packages"&gt;Installing DokuWiki from&amp;nbsp;Packages&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#nginx-inside-the-jail"&gt;nginx Inside the&amp;nbsp;Jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#caddy-at-the-edge"&gt;Caddy at the&amp;nbsp;Edge&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#initial-dokuwiki-setup"&gt;Initial DokuWiki&amp;nbsp;Setup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#updates-and-backups"&gt;Updates and&amp;nbsp;Backups&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-this-setup-buys-me"&gt;What This Setup Buys&amp;nbsp;Me&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#further-reading"&gt;Further&amp;nbsp;Reading&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-shape-of-the-setup"&gt;The Shape of the&amp;nbsp;Setup&lt;/h2&gt;
&lt;p&gt;The hostnames and addresses below are deliberately&amp;nbsp;sanitized. &lt;code&gt;example.com&lt;/code&gt;, &lt;code&gt;192.0.2.0/24&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;2001:db8::/32&lt;/code&gt; are reserved for documentation; they describe the topology without publishing the production&amp;nbsp;inventory.&lt;/p&gt;
&lt;p&gt;The running system has two relevant thin&amp;nbsp;jails:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[ Internet ]
     |
     | HTTPS :443
     v
+-------------------------------+
| caddy jail                    |
| 192.0.2.10 (public ingress)   |
| 2001:db8:1000::10 (public)    |
| TLS + GeoIP/access policy     |
+---------------+---------------+
                |
                | HTTP over host-local IPv6
                | -&amp;gt; [2001:db8:1000::26]:80
                v
+-------------------------------+
| wiki jail                     |
| 10.0.0.26                     |
| 2001:db8:1000::26             |
| nginx -&amp;gt; PHP-FPM -&amp;gt; DokuWiki  |
+-------------------------------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On the host, Bastille shows the wiki as a FreeBSD 15.1 thin jail with both private IPv4 and routed&amp;nbsp;IPv6:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@server.example.com:~ # bastille list wiki
 JID  Name  Boot  Prio  State  Type  IP Address               Published Ports  Release
 8    wiki  on    99    Up     thin  10.0.0.26                 -                15.1-RELEASE-p1
                                     2001:db8:1000::26
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There are no published ports. That column being empty is important: DokuWiki is not exposed through a&amp;nbsp;Bastille &lt;code&gt;rdr&lt;/code&gt; (port redirection) rule. Caddy reaches nginx directly over IPv6, and the host firewall decides which traffic may reach the jail network. For the broader &lt;span class="caps"&gt;PF&lt;/span&gt; design behind these hosts, see my &lt;a href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/"&gt;practical &lt;span class="caps"&gt;PF&lt;/span&gt; guide&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;This is not the smallest possible layout. Caddy and DokuWiki could share a jail, but the separate frontend is an intentional part of the architecture. Caddy is the central ingress for every web application on this host. Each application gets its own jail, but only the Caddy jail is reachable from the outside. That gives the machine a single public entry point where &lt;span class="caps"&gt;TLS&lt;/span&gt; termination, access logging, headers, and ingress policy are handled consistently. The application jails remain backends: isolated from each other, invisible to the public network, and only expected to serve requests from&amp;nbsp;Caddy.&lt;/p&gt;
&lt;h2 id="creating-the-jail"&gt;Creating the&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;I am assuming that Bastille is already installed,&amp;nbsp;the &lt;code&gt;15.1-RELEASE&lt;/code&gt; bootstrap exists, and&amp;nbsp;the &lt;code&gt;bastille0&lt;/code&gt; bridge is configured. A basic &lt;span class="caps"&gt;VNET&lt;/span&gt; jail on that bridge starts like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-B&lt;span class="w"&gt; &lt;/span&gt;wiki&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.1-RELEASE&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.0.0.26&lt;span class="w"&gt; &lt;/span&gt;bastille0
bastille&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;wiki
bastille&lt;span class="w"&gt; &lt;/span&gt;console&lt;span class="w"&gt; &lt;/span&gt;wiki
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The routed IPv6 address is part of my existing Bastille network configuration. Exactly how you attach it depends on whether you route a prefix&amp;nbsp;to &lt;code&gt;bastille0&lt;/code&gt;, bridge a public segment, or use &lt;span class="caps"&gt;NAT66&lt;/span&gt;. I route a dedicated prefix to the jail network; there is no reason to hide IPv6 behind &lt;span class="caps"&gt;NAT&lt;/span&gt; when &lt;span class="caps"&gt;PF&lt;/span&gt; can filter it&amp;nbsp;properly.&lt;/p&gt;
&lt;p&gt;Inside the jail, the normal housekeeping applies: disable services I do not need, keep syslog from opening network sockets, and enable only the two application&amp;nbsp;daemons:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;syslogd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-ss&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sendmail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sendmail_submit_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sendmail_outbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sendmail_msp_queue_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;

sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;nginx_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;php_fpm_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The result is refreshingly boring. The entire application process tree is nginx, &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt;, and the standard jail&amp;nbsp;housekeeping:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root  php-fpm: master process (/usr/local/etc/php-fpm.conf)
www   php-fpm: pool www
www   php-fpm: pool www
root  nginx: master process /usr/local/sbin/nginx
www   nginx: worker process
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For a private documentation service, boring is a&amp;nbsp;feature.&lt;/p&gt;
&lt;h2 id="installing-dokuwiki-from-packages"&gt;Installing DokuWiki from&amp;nbsp;Packages&lt;/h2&gt;
&lt;p&gt;DokuWiki is in the FreeBSD package repository with &lt;span class="caps"&gt;PHP&lt;/span&gt; flavors. This jail uses &lt;span class="caps"&gt;PHP&lt;/span&gt;&amp;nbsp;8.5:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;dokuwiki-php85
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The package pulls in &lt;span class="caps"&gt;PHP&lt;/span&gt; and the extensions DokuWiki needs. On this system that&amp;nbsp;includes &lt;code&gt;ctype&lt;/code&gt;, &lt;code&gt;filter&lt;/code&gt;, &lt;code&gt;gd&lt;/code&gt;, &lt;code&gt;iconv&lt;/code&gt;, &lt;code&gt;mbstring&lt;/code&gt;, &lt;code&gt;session&lt;/code&gt;, &lt;code&gt;simplexml&lt;/code&gt;, &lt;code&gt;xml&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;zlib&lt;/code&gt;. &lt;span class="caps"&gt;LDAP&lt;/span&gt; and &lt;span class="caps"&gt;PDO&lt;/span&gt; are installed as well because I may want directory-backed authentication and other plugins later, but a basic DokuWiki does not need a database driver at&amp;nbsp;all.&lt;/p&gt;
&lt;p&gt;The package installs the application&amp;nbsp;below:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/usr/local/www/dokuwiki
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the whole attraction. There is no application checkout&amp;nbsp;under &lt;code&gt;/opt&lt;/code&gt;, no Composer build, and no database&amp;nbsp;bootstrap. &lt;code&gt;pkg&lt;/code&gt; owns the application code and upgrades it along with the rest of the&amp;nbsp;jail.&lt;/p&gt;
&lt;p&gt;Start from the production &lt;span class="caps"&gt;PHP&lt;/span&gt;&amp;nbsp;configuration:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cp&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/php.ini-production&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/php.ini
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt; master configuration can remain almost entirely at its packaged default. It includes the pool definitions&amp;nbsp;from &lt;code&gt;/usr/local/etc/php-fpm.d/*.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[global]&lt;/span&gt;
&lt;span class="na"&gt;pid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;run/php-fpm.pid&lt;/span&gt;
&lt;span class="na"&gt;include&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/etc/php-fpm.d/*.conf&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In&amp;nbsp;the &lt;code&gt;www&lt;/code&gt; pool, I use a Unix socket rather than opening another &lt;span class="caps"&gt;TCP&lt;/span&gt;&amp;nbsp;listener:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;; /usr/local/etc/php-fpm.d/www.conf&lt;/span&gt;
&lt;span class="k"&gt;[www]&lt;/span&gt;
&lt;span class="na"&gt;user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;www&lt;/span&gt;
&lt;span class="na"&gt;group&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;www&lt;/span&gt;

&lt;span class="na"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/var/run/php-fpm.sock&lt;/span&gt;
&lt;span class="na"&gt;listen.owner&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;www&lt;/span&gt;
&lt;span class="na"&gt;listen.group&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;www&lt;/span&gt;
&lt;span class="na"&gt;listen.mode&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0660&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;nginx runs its worker&amp;nbsp;as &lt;code&gt;www&lt;/code&gt;, so both processes can use the socket without making it&amp;nbsp;world-writable.&lt;/p&gt;
&lt;p&gt;If uploads should be larger than &lt;span class="caps"&gt;PHP&lt;/span&gt;&amp;#8217;s defaults, change both sides of the request path. Setting only nginx&amp;#8217;s limit does not make &lt;span class="caps"&gt;PHP&lt;/span&gt; accept a 32 &lt;span class="caps"&gt;MB&lt;/span&gt;&amp;nbsp;file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;; /usr/local/etc/php.ini&lt;/span&gt;
&lt;span class="na"&gt;upload_max_filesize&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;32M&lt;/span&gt;
&lt;span class="na"&gt;post_max_size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;32M&lt;/span&gt;
&lt;span class="na"&gt;cgi.fix_pathinfo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;post_max_size&lt;/code&gt; must be at least as large&amp;nbsp;as &lt;code&gt;upload_max_filesize&lt;/code&gt;. I also&amp;nbsp;disable &lt;code&gt;cgi.fix_pathinfo&lt;/code&gt;; nginx resolves the script explicitly, so &lt;span class="caps"&gt;PHP&lt;/span&gt; does not need to guess which file was&amp;nbsp;intended.&lt;/p&gt;
&lt;h2 id="nginx-inside-the-jail"&gt;nginx Inside the&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;nginx serves the DokuWiki tree, hands real &lt;span class="caps"&gt;PHP&lt;/span&gt; files to &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt;, and translates clean wiki URLs into DokuWiki&amp;#8217;s entry points. The complete configuration is short enough to keep in one&amp;nbsp;file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;worker_processes&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;events&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;worker_connections&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;http&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;include&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="s"&gt;mime.types&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;default_type&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;application/octet-stream&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;sendfile&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;keepalive_timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:80&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ipv6only=on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;wiki.example.com&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;root&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/usr/local/www/dokuwiki&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;index&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;index.php&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;index.html&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;doku.php&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;client_max_body_size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;32M&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;try_files&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$uri&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$uri/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@dokuwiki&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@dokuwiki&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;rewrite&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;^/_media/(.*)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/lib/exe/fetch.php?media=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;last&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;rewrite&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;^/_detail/(.*)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/lib/exe/detail.php?media=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;last&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;rewrite&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;^/_export/([^/]+)/(.*)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/doku.php?do=export_&lt;/span&gt;&lt;span class="nv"&gt;$1&amp;amp;id=$2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;last&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;rewrite&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;^/(.*)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/doku.php?id=&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;last&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sr"&gt;\.php$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;try_files&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$uri&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;404&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;fastcgi_pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;unix:/var/run/php-fpm.sock&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;fastcgi_index&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;index.php&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;fastcgi_params&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;fastcgi_param&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;SCRIPT_FILENAME&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$document_root$fastcgi_script_name&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;fastcgi_param&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;SCRIPT_NAME&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$fastcgi_script_name&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# DokuWiki stores both content and configuration below the webroot.&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sr"&gt;^/(data|conf|bin|inc|vendor)/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;deny&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;all&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sr"&gt;/\.ht&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;deny&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;all&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# Keep the installer unreachable after the initial setup.&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/install.php&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kn"&gt;deny&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;all&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There are five details here worth calling&amp;nbsp;out.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The rewrite rules preserve query arguments without spelling&amp;nbsp;out &lt;code&gt;$args&lt;/code&gt;.&lt;/strong&gt; nginx automatically appends the request&amp;#8217;s existing arguments when a rewrite target introduces new ones. A request such&amp;nbsp;as &lt;code&gt;/_media/photo.jpg?w=200&lt;/code&gt; therefore&amp;nbsp;reaches &lt;code&gt;fetch.php&lt;/code&gt; with&amp;nbsp;both &lt;code&gt;media=photo.jpg&lt;/code&gt; and &lt;code&gt;w=200&lt;/code&gt;;&amp;nbsp;adding &lt;code&gt;&amp;amp;$args&lt;/code&gt; manually would duplicate the original query&amp;nbsp;string.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;nginx listens on IPv6 only.&lt;/strong&gt; The jail has an &lt;span class="caps"&gt;RFC1918&lt;/span&gt; address because that is how the broader Bastille network is built, but this particular backend path&amp;nbsp;uses &lt;code&gt;2001:db8:1000::26&lt;/code&gt;. Keeping the listener IPv6-only makes that decision&amp;nbsp;explicit.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;try_files $uri =404&lt;/code&gt; comes before &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt;.&lt;/strong&gt; nginx will not pass an&amp;nbsp;arbitrary &lt;code&gt;.php&lt;/code&gt; path to the interpreter unless the file actually exists below the document&amp;nbsp;root.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The internal directories are denied at the web server.&lt;/strong&gt; DokuWiki stores pages&amp;nbsp;below &lt;code&gt;data/&lt;/code&gt; and configuration&amp;nbsp;below &lt;code&gt;conf/&lt;/code&gt;, both underneath the document root. DokuWiki ships its own protection files for Apache, but nginx does not&amp;nbsp;read &lt;code&gt;.htaccess&lt;/code&gt;. Without explicit deny rules, the flat-file design that makes backups simple can turn into a data&amp;nbsp;leak.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The installer is a one-time endpoint.&lt;/strong&gt;&amp;nbsp;Use &lt;code&gt;/install.php&lt;/code&gt; to create the initial configuration and administrator, then remove or deny it. I prefer the nginx denial because a later package operation cannot accidentally make the file reachable&amp;nbsp;again.&lt;/p&gt;
&lt;p&gt;Validate before starting&amp;nbsp;anything:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;php-fpm&lt;span class="w"&gt; &lt;/span&gt;-tt
nginx&lt;span class="w"&gt; &lt;/span&gt;-t
service&lt;span class="w"&gt; &lt;/span&gt;php-fpm&lt;span class="w"&gt; &lt;/span&gt;start
service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;At this point the backend can be tested from the Caddy jail without involving public &lt;span class="caps"&gt;DNS&lt;/span&gt; or &lt;span class="caps"&gt;TLS&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-I&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Host: wiki.example.com&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;http://&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8:1000::26&lt;span class="o"&gt;]&lt;/span&gt;/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That should return an &lt;span class="caps"&gt;HTTP&lt;/span&gt; response from nginx. Supplying the&amp;nbsp;expected &lt;code&gt;Host&lt;/code&gt; header also confirms that nginx selected the intended virtual host. If it&amp;nbsp;returns &lt;code&gt;502 Bad Gateway&lt;/code&gt;, check the &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt; socket and its ownership. If it returns DokuWiki&amp;#8217;s raw &lt;span class="caps"&gt;PHP&lt;/span&gt; source, stop immediately: the &lt;span class="caps"&gt;PHP&lt;/span&gt; location is not being selected and the site must not be&amp;nbsp;exposed.&lt;/p&gt;
&lt;h2 id="caddy-at-the-edge"&gt;Caddy at the&amp;nbsp;Edge&lt;/h2&gt;
&lt;p&gt;Caddy runs in a separate jail&amp;nbsp;at &lt;code&gt;2001:db8:1000::10&lt;/code&gt;. It obtains and renews the certificate, applies the public access policy, and proxies accepted requests to&amp;nbsp;nginx:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;wiki&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;requests&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neither&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;geolocated&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;DE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;nor&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;trusted&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;network&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;@blocked&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="ow"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;maxmind_geolocation&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;db_path&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;share&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;GeoIP&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;GeoLite2&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;Country&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;mmdb&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;allow_countries&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;DE&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="ow"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;remote_ip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;10.0.0.0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="nl"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="o"&gt;::/&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="nl"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="o"&gt;::/&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;abort&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;@blocked&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;reverse_proxy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;2001:db8:1000::26&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;header_up&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;Host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;&lt;span class="k"&gt;host&lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;header_up&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;X&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nc"&gt;Real&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;IP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;&lt;span class="n"&gt;client_ip&lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;header_up&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;X&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;Forwarded&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;For&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;&lt;span class="n"&gt;client_ip&lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;header_up&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;X&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;Forwarded&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;Proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;&lt;span class="n"&gt;scheme&lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;Strict&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;Transport&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;Security&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;max-age=31536000; includeSubdomains&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;}&lt;/span&gt;
&lt;span class="err"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The matcher logic is easy to misread. A request is blocked when it is &lt;strong&gt;neither&lt;/strong&gt; from Germany &lt;strong&gt;nor&lt;/strong&gt; from one of the explicitly trusted networks. In other words, German source addresses and my internal networks are allowed; everything else is aborted before it reaches&amp;nbsp;DokuWiki.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;maxmind_geolocation&lt;/code&gt; is not part of stock Caddy. The Caddy binary needs to be built or packaged with that module, and the GeoLite2 database needs to be kept current. More importantly, country filtering is not authentication. GeoIP data is imperfect and a &lt;span class="caps"&gt;VPN&lt;/span&gt; exits wherever its operator wants it to. DokuWiki&amp;#8217;s ACLs still decide who can read and edit the wiki; the Caddy rule merely removes a large amount of irrelevant internet&amp;nbsp;traffic.&lt;/p&gt;
&lt;p&gt;Validate and reload Caddy in its&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;caddy&lt;span class="w"&gt; &lt;/span&gt;validate&lt;span class="w"&gt; &lt;/span&gt;--config&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/caddy/Caddyfile
service&lt;span class="w"&gt; &lt;/span&gt;caddy&lt;span class="w"&gt; &lt;/span&gt;reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Caddy speaks plain &lt;span class="caps"&gt;HTTP&lt;/span&gt; to nginx. That is also deliberate. The packets cross the host-local Bastille bridge between two jails, but never leave the physical machine or enter a network I do not control. They are not literally loopback packets - &lt;span class="caps"&gt;VNET&lt;/span&gt; gives each jail its own network stack and connects it through epairs and the bridge - but from a trust-boundary perspective the path is entirely&amp;nbsp;local.&lt;/p&gt;
&lt;p&gt;Adding &lt;span class="caps"&gt;TLS&lt;/span&gt; between Caddy and every application jail would therefore encrypt traffic only while it moves inside the same kernel. It would add certificate management, another failure mode, and unnecessary encryption work without protecting the traffic on any external wire. The &lt;span class="caps"&gt;CPU&lt;/span&gt; cost would be small on modern hardware, but it would still be wasted cycles. &lt;span class="caps"&gt;TLS&lt;/span&gt; terminates once, at the central ingress where the untrusted network ends. If a backend ever moved to another host or crossed a network I did not fully control, I would enable &lt;span class="caps"&gt;TLS&lt;/span&gt; on that hop&amp;nbsp;immediately.&lt;/p&gt;
&lt;h2 id="initial-dokuwiki-setup"&gt;Initial DokuWiki&amp;nbsp;Setup&lt;/h2&gt;
&lt;p&gt;With the proxy in place, temporarily&amp;nbsp;allow &lt;code&gt;/install.php&lt;/code&gt;: comment out the&amp;nbsp;exact &lt;code&gt;location = /install.php&lt;/code&gt; block shown above,&amp;nbsp;run &lt;code&gt;nginx -t &amp;amp;&amp;amp; service nginx reload&lt;/code&gt;, and&amp;nbsp;visit:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;https://wiki.example.com/install.php
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The installer creates the wiki title, administrator, and initial &lt;span class="caps"&gt;ACL&lt;/span&gt; policy. For an internal wiki I start closed: no anonymous reads, no public registration, and explicit access for authenticated users. That is easier to loosen later than discovering that a runbook was indexed by a search&amp;nbsp;engine.&lt;/p&gt;
&lt;p&gt;After the installer&amp;nbsp;completes:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Restore the nginx denial&amp;nbsp;for &lt;code&gt;/install.php&lt;/code&gt;,&amp;nbsp;run &lt;code&gt;nginx -t&lt;/code&gt;, and reload&amp;nbsp;nginx.&lt;/li&gt;
&lt;li&gt;Log out and confirm that an anonymous browser cannot read a&amp;nbsp;page.&lt;/li&gt;
&lt;li&gt;Log in as a normal non-admin user and verify the intended namespace&amp;nbsp;permissions.&lt;/li&gt;
&lt;li&gt;Upload a file large enough to confirm the nginx and &lt;span class="caps"&gt;PHP&lt;/span&gt; limits&amp;nbsp;agree.&lt;/li&gt;
&lt;li&gt;Request &lt;code&gt;/data/pages/start.txt&lt;/code&gt; directly and confirm nginx&amp;nbsp;returns &lt;code&gt;403&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The last check matters. A working front page proves the application runs; it does not prove its private storage is&amp;nbsp;private.&lt;/p&gt;
&lt;h2 id="updates-and-backups"&gt;Updates and&amp;nbsp;Backups&lt;/h2&gt;
&lt;p&gt;The software side follows the normal FreeBSD package&amp;nbsp;workflow:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;update
pkg&lt;span class="w"&gt; &lt;/span&gt;audit&lt;span class="w"&gt; &lt;/span&gt;-F
pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade
service&lt;span class="w"&gt; &lt;/span&gt;php-fpm&lt;span class="w"&gt; &lt;/span&gt;restart
service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After an upgrade, I check the DokuWiki release notes, load a page, edit a test page, and exercise any non-core authentication plugin. &lt;span class="caps"&gt;PHP&lt;/span&gt; applications have a talent for looking healthy until the first request reaches a code path an extension no longer&amp;nbsp;supports.&lt;/p&gt;
&lt;p&gt;DokuWiki&amp;#8217;s lack of a database makes the backup boundary obvious. The important state is&amp;nbsp;below &lt;code&gt;/usr/local/www/dokuwiki&lt;/code&gt;,&amp;nbsp;especially:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;conf/              site configuration, users, ACLs
data/pages/        page content
data/media/        uploaded files
data/meta/         metadata
data/attic/        old page revisions
data/media_attic/  old media revisions
lib/plugins/       installed plugins
lib/tpl/           additional templates
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The simplest reliable policy is to snapshot and back up the entire jail dataset. That captures application state and configuration together, and it fits directly into the &lt;a href="https://hofstede.it/my-multi-stage-backup-strategy-zfs-proxmox-and-paranoia/"&gt;multi-stage &lt;span class="caps"&gt;ZFS&lt;/span&gt; backup strategy&lt;/a&gt; I use elsewhere. Plain files make recovery inspectable, but they do not make backups&amp;nbsp;optional.&lt;/p&gt;
&lt;p&gt;I still test a restore. I clone a recent snapshot into a throwaway jail on a disposable address, start &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt; and nginx there, and&amp;nbsp;point &lt;code&gt;curl&lt;/code&gt; at it with the&amp;nbsp;expected &lt;code&gt;Host&lt;/code&gt; header. Then I log in through a temporary Caddy route and verify a page edit, an old revision, and a media download before destroying the jail. A pile&amp;nbsp;of &lt;code&gt;start.txt&lt;/code&gt; files in an archive is not a restored wiki until nginx, &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt;, ACLs, plugins, and uploaded media all work together on a clean&amp;nbsp;jail.&lt;/p&gt;
&lt;h2 id="what-this-setup-buys-me"&gt;What This Setup Buys&amp;nbsp;Me&lt;/h2&gt;
&lt;p&gt;The end result is a deliberately modest&amp;nbsp;service:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;DokuWiki provides a capable wiki without a database or background&amp;nbsp;workers.&lt;/li&gt;
&lt;li&gt;The Bastille jail gives it a small filesystem, process, and network&amp;nbsp;boundary.&lt;/li&gt;
&lt;li&gt;nginx exposes only the application entry points and explicitly blocks DokuWiki&amp;#8217;s private&amp;nbsp;directories.&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt; communicates over a local Unix socket rather than another network&amp;nbsp;port.&lt;/li&gt;
&lt;li&gt;Caddy centralizes certificates, access policy, and public request&amp;nbsp;handling.&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots capture the complete service in one consistent backup&amp;nbsp;unit.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is the kind of workload where FreeBSD jails feel exactly right. The application does not need a virtual machine, and wrapping three daemons in an &lt;span class="caps"&gt;OCI&lt;/span&gt; stack would add more moving parts than it removes. One jail, two services, one directory of state, and a reverse proxy in front is&amp;nbsp;enough.&lt;/p&gt;
&lt;p&gt;The interesting part is not getting DokuWiki to render a&amp;nbsp;page. &lt;code&gt;pkg install&lt;/code&gt; handles most of that. The interesting part is drawing clean boundaries: no direct published port, no public &lt;span class="caps"&gt;PHP&lt;/span&gt;-&lt;span class="caps"&gt;FPM&lt;/span&gt; listener, no access&amp;nbsp;to &lt;code&gt;data/&lt;/code&gt; or &lt;code&gt;conf/&lt;/code&gt;, no assumption that GeoIP replaces authentication, and no backup plan that stops at “the files are somewhere on &lt;span class="caps"&gt;ZFS&lt;/span&gt;.” Once those details are explicit, the wiki becomes exactly what internal infrastructure should be: useful, quiet, and&amp;nbsp;unsurprising.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.dokuwiki.org/dokuwiki"&gt;DokuWiki&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook:&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/ports/"&gt;FreeBSD Handbook: Packages and&amp;nbsp;Ports&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastillebsd.org/"&gt;Bastille&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://caddyserver.com/docs/"&gt;Caddy&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/hosting-a-static-blog-on-freebsd-with-bastille-jails-and-automated-deployment/"&gt;Hosting a Static Blog on FreeBSD with Bastille Jails and Automated&amp;nbsp;Deployment&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/self-hosted-cryptpad-on-freebsd-with-vnet-jails-and-caddy/"&gt;Self-Hosted CryptPad on FreeBSD with &lt;span class="caps"&gt;VNET&lt;/span&gt; Jails and&amp;nbsp;Caddy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;FreeBSD Foundationals:&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="bastille"/><category term="jails"/><category term="dokuwiki"/><category term="nginx"/><category term="caddy"/><category term="php"/><category term="self-hosting"/><category term="ipv6"/></entry><entry><title>A Caching FreeBSD Mirror for DN42: nginx proxy_store, pf, and a Dual-Homed VM</title><link href="https://hofstede.it/a-caching-freebsd-mirror-for-dn42-nginx-proxy_store-pf-and-a-dual-homed-vm/" rel="alternate"/><published>2026-07-05T00:00:00+02:00</published><updated>2026-07-05T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-07-05:/a-caching-freebsd-mirror-for-dn42-nginx-proxy_store-pf-and-a-dual-homed-vm/</id><summary type="html">&lt;p&gt;Patching FreeBSD machines inside the &lt;span class="caps"&gt;DN42&lt;/span&gt; overlay network without giving them clearnet access - a dual-homed &lt;span class="caps"&gt;VM&lt;/span&gt; that lazily caches pkg.freebsd.org, update.freebsd.org, and release tarballs from ftp.freebsd.org with nginx proxy_store, follows &lt;span class="caps"&gt;CDN&lt;/span&gt; redirects server-side, and serves everything over IPv6 into the mesh. Now also available as a public &lt;span class="caps"&gt;DN42&lt;/span&gt; service at&amp;nbsp;bsdmirror.chofstede.dn42.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#at-a-glance"&gt;At a&amp;nbsp;Glance&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#architecture"&gt;Architecture&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-host"&gt;The&amp;nbsp;Host&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#pf-default-deny-both-directions"&gt;pf: Default Deny, Both&amp;nbsp;Directions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#nginx-a-lazy-mirror-with-proxy_store"&gt;nginx: A Lazy Mirror with proxy_store&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-pkg-mirror-port-8080"&gt;The pkg Mirror (Port&amp;nbsp;8080)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-freebsd-update-mirror-port-80"&gt;The freebsd-update Mirror (Port&amp;nbsp;80)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-release-mirror-port-8081"&gt;The Release Mirror (Port&amp;nbsp;8081)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-about-trust"&gt;What About&amp;nbsp;Trust?&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#client-configuration"&gt;Client&amp;nbsp;Configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-proof-150-to-151-over-the-mesh"&gt;The Proof: 15.0 to 15.1 Over the Mesh&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-full-jail-lifecycle"&gt;The Full Jail&amp;nbsp;Lifecycle&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#operational-notes"&gt;Operational&amp;nbsp;Notes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#public-dn42-service"&gt;Public &lt;span class="caps"&gt;DN42&lt;/span&gt;&amp;nbsp;Service&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#lessons-learned"&gt;Lessons&amp;nbsp;Learned&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;Since &lt;a href="https://hofstede.it/joining-dn42-a-mikrotik-border-three-wireguard-peerings-and-a-freebsd-jail-in-the-hobbyist-internet/"&gt;joining &lt;span class="caps"&gt;DN42&lt;/span&gt;&lt;/a&gt;, the number of FreeBSD machines living inside the mesh has quietly grown. They sit&amp;nbsp;on &lt;code&gt;fdce:73f7:a2dc::/48&lt;/code&gt;, they speak IPv6 to the rest of the hobbyist internet, and by design they have no route to the clearnet. That is exactly how I want them - right up until the moment a FreeBSD security advisory lands and every one of them needs to talk&amp;nbsp;to &lt;code&gt;update.freebsd.org&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The usual answers are all unsatisfying. Punching clearnet holes into isolated hosts defeats the point of isolating them. &lt;span class="caps"&gt;NAT&lt;/span&gt;-ing the whole &lt;span class="caps"&gt;DN42&lt;/span&gt; segment out through the home router reintroduces the coupling I was trying to avoid. And running a full FreeBSD mirror means rsyncing terabytes of distfiles I will never&amp;nbsp;touch.&lt;/p&gt;
&lt;p&gt;So I built the fourth option: a small dual-homed FreeBSD &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;nbsp;called &lt;code&gt;bsdmirror&lt;/code&gt; that faces the clearnet on one interface and &lt;span class="caps"&gt;DN42&lt;/span&gt; on the other, and lazily caches exactly the files my machines actually request - packages&amp;nbsp;from &lt;code&gt;pkg.freebsd.org&lt;/code&gt;, base system patches&amp;nbsp;from &lt;code&gt;update.freebsd.org&lt;/code&gt;, and release tarballs&amp;nbsp;from &lt;code&gt;ftp.freebsd.org&lt;/code&gt; for bootstrapping Bastille jails. The first client to ask for a file pays the upstream round trip; everyone after that is served from local &lt;span class="caps"&gt;ZFS&lt;/span&gt;. And since the thing exists anyway, it is now a public &lt;span class="caps"&gt;DN42&lt;/span&gt; service: any &lt;span class="caps"&gt;DN42&lt;/span&gt; participant can point their FreeBSD hosts&amp;nbsp;at &lt;code&gt;bsdmirror.chofstede.dn42&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This article walks through the whole thing: the dual-homed network setup, the strict pf policy, the three nginx server blocks (including the fun part - chasing &lt;span class="caps"&gt;CDN&lt;/span&gt; redirects server-side so the cache stays transparent), the client configuration, and a real 15.0 to 15.1 upgrade plus a Bastille jail bootstrap running entirely over the&amp;nbsp;mesh.&lt;/p&gt;
&lt;h2 id="at-a-glance"&gt;At a&amp;nbsp;Glance&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;A FreeBSD &lt;span class="caps"&gt;VM&lt;/span&gt; with two&amp;nbsp;interfaces: &lt;code&gt;vtnet0&lt;/code&gt; (clearnet, IPv4, egress only)&amp;nbsp;and &lt;code&gt;vtnet1&lt;/code&gt; (&lt;span class="caps"&gt;DN42&lt;/span&gt;, IPv6, ingress&amp;nbsp;only)&lt;/li&gt;
&lt;li&gt;pf with default deny in both directions - the box can only fetch from ports 80/443/53/123 outbound and only serves the mirror ports plus &lt;span class="caps"&gt;SSH&lt;/span&gt; inbound on the &lt;span class="caps"&gt;DN42&lt;/span&gt;&amp;nbsp;side&lt;/li&gt;
&lt;li&gt;nginx &lt;code&gt;proxy_store&lt;/code&gt; as a lazy, on-demand mirror: files land on disk in the same hierarchy as the upstream &lt;span class="caps"&gt;URL&lt;/span&gt;&amp;nbsp;space&lt;/li&gt;
&lt;li&gt;Three services: pkg repositories&amp;nbsp;on &lt;code&gt;:8080&lt;/code&gt;, &lt;code&gt;freebsd-update&lt;/code&gt; patches&amp;nbsp;on &lt;code&gt;:80&lt;/code&gt;, and release tarballs&amp;nbsp;(for &lt;code&gt;bastille bootstrap&lt;/code&gt; and friends)&amp;nbsp;on &lt;code&gt;:8081&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;A named-location trick to&amp;nbsp;follow &lt;code&gt;pkg.freebsd.org&lt;/code&gt; and &lt;code&gt;ftp.freebsd.org&lt;/code&gt; &lt;span class="caps"&gt;CDN&lt;/span&gt; redirects inside nginx, so clients never see a 302 pointing at a clearnet host they cannot&amp;nbsp;reach&lt;/li&gt;
&lt;li&gt;Signature files&amp;nbsp;(&lt;code&gt;latest.ssl&lt;/code&gt;, &lt;code&gt;pub.ssl&lt;/code&gt;) deliberately passed through uncached,&amp;nbsp;so &lt;code&gt;freebsd-update&lt;/code&gt; clients always see the current patch&amp;nbsp;level&lt;/li&gt;
&lt;li&gt;Clients keep verifying signatures against the official FreeBSD keys and fingerprints - the mirror never becomes a trusted&amp;nbsp;party&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="architecture"&gt;Architecture&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                         Clearnet
                            │
                    pkg.freebsd.org
                  update.freebsd.org
                    ftp.freebsd.org
                            │
                   80/443/53/123 out only
                            │
              ┌─────────────┴──────────────┐
              │  bsdmirror (FreeBSD VM)    │
              │                            │
              │  vtnet0  10.254.240.30     │  ← clearnet side (egress)
              │                            │
              │  nginx + proxy_store       │
              │  /var/cache/nginx (ZFS)    │
              │                            │
              │  vtnet1  fdce:...:d::30    │  ← DN42 side (ingress)
              └─────────────┬──────────────┘
                            │
   :8080 pkg / :80 freebsd-update / :8081 releases
                            │
              ┌─────────────┴──────────────┐
              │         DN42 Mesh          │
              │  fd00::/8 via fdce:...:d::1│
              │                            │
              │  my jails and VMs, plus    │
              │  anyone else on DN42       │
              └────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The separation is physical, not just a firewall policy: the clearnet interface carries only outbound fetches, and the &lt;span class="caps"&gt;DN42&lt;/span&gt; interface carries only inbound client traffic. There is no forwarding between the two&amp;nbsp;- &lt;code&gt;bsdmirror&lt;/code&gt; is not a router, it is a store-and-serve middlebox. A packet from &lt;span class="caps"&gt;DN42&lt;/span&gt; never transits to the clearnet; the only thing that crosses the boundary is an &lt;span class="caps"&gt;HTTP&lt;/span&gt; request that nginx chooses to make on its own&amp;nbsp;behalf.&lt;/p&gt;
&lt;h2 id="the-host"&gt;The&amp;nbsp;Host&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;rc.conf&lt;/code&gt; is short. Two interfaces, a default route per world, and a static route that sends all&amp;nbsp;of &lt;code&gt;fd00::/8&lt;/code&gt; back at the &lt;span class="caps"&gt;DN42&lt;/span&gt;&amp;nbsp;gateway:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;hostname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bsdmirror&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Network Configurations&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.254.240.30 netmask 0xffffff00&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 fdbd:a6b7:c88f::30 prefixlen 64&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.254.240.1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fdbd:a6b7:c88f::1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# DN42 interface&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet1_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 fdce:73f7:a2dc:d::30 prefixlen 64&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Static routes&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_static_routes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;dn42_static&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_route_dn42_static&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-6 fd00::/8 fdce:73f7:a2dc:d::1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Security Enhancements&lt;/span&gt;
&lt;span class="nv"&gt;clear_tmp_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;syslogd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-ss&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Core Services&lt;/span&gt;
&lt;span class="nv"&gt;zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;nginx_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pflog_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;sshd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;ntpd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ntpd_sync_on_start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ntpd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-g -N&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;fd00::/8&lt;/code&gt; static route is what makes the box reachable from the entire mesh, not just my own /48: replies to any &lt;span class="caps"&gt;DN42&lt;/span&gt; &lt;span class="caps"&gt;ULA&lt;/span&gt; address go back through the &lt;span class="caps"&gt;DN42&lt;/span&gt;-side gateway, while everything else (the upstream fetches) follows the clearnet default route. Routing does the traffic separation before pf even gets&amp;nbsp;involved.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;syslogd_flags="-ss"&lt;/code&gt; keeps syslogd from opening network sockets,&amp;nbsp;and &lt;code&gt;clear_tmp_enable&lt;/code&gt; is one of those defaults I set on every machine and forget&amp;nbsp;about.&lt;/p&gt;
&lt;h2 id="pf-default-deny-both-directions"&gt;pf: Default Deny, Both&amp;nbsp;Directions&lt;/h2&gt;
&lt;p&gt;Most firewall policies are strict inbound and permissive outbound. For a caching proxy the outbound side deserves the same treatment - this machine has exactly four reasons to open a connection to the internet, so those four are the whole egress&amp;nbsp;policy:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# --- Macros &amp;amp; Interfaces ---
ext_v4_if = &amp;quot;vtnet0&amp;quot;
cache_v6_if = &amp;quot;vtnet1&amp;quot;

# --- Options ---
set skip on lo0
set block-policy drop
set loginterface $cache_v6_if

# --- Scrubbing ---
scrub in all fragment reassemble

# --- Filtering Rules ---
# Strict Default Deny for all directions
block drop in log all
block drop out log all

# 1. Allow Egress Traffic safely (States allow replies back in)
pass out quick on $ext_v4_if proto tcp to any port { 80, 443 } keep state
pass out quick on $ext_v4_if proto udp to any port 53 keep state   # DNS
pass out quick on $ext_v4_if proto udp to any port 123 keep state  # NTP

# 2. Strict IPv6 Nginx Cache Ingress Rules on vtnet1
pass in quick on $cache_v6_if inet6 proto tcp to fdce:73f7:a2dc:d::30 port 8080 flags S/SA keep state # PKG
pass in quick on $cache_v6_if inet6 proto tcp to fdce:73f7:a2dc:d::30 port 80 flags S/SA keep state   # Updates
pass in quick on $cache_v6_if inet6 proto tcp to fdce:73f7:a2dc:d::30 port 8081 flags S/SA keep state # Releases

# 3. Essential Management
pass in quick on $cache_v6_if proto tcp from any to fdce:73f7:a2dc:d::30 port 22 flags S/SA keep state

# 4. Mandatory ICMPv6 Rules for network health/IPv6 Neighbor Discovery
pass in quick inet6 proto ipv6-icmp icmp6-type { echoreq, neighbrsol, neighbradv, toobig, timex }
pass out quick inet6 proto ipv6-icmp all
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few deliberate&amp;nbsp;choices:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;block drop out log all&lt;/code&gt; as the baseline.&lt;/strong&gt; If this box is ever compromised through the service it exposes to a hobbyist mesh, the attacker gets a machine that can talk &lt;span class="caps"&gt;HTTP&lt;/span&gt;, &lt;span class="caps"&gt;HTTPS&lt;/span&gt;, &lt;span class="caps"&gt;DNS&lt;/span&gt; and &lt;span class="caps"&gt;NTP&lt;/span&gt; to the clearnet and nothing else. No reverse shells on high ports, no outbound &lt;span class="caps"&gt;SSH&lt;/span&gt;, no exfiltration channels beyond what the box legitimately needs. Egress filtering is cheap on a single-purpose machine and expensive to retrofit everywhere else - single-purpose boxes are where you get to actually do&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The service rules bind to the &lt;span class="caps"&gt;DN42&lt;/span&gt; address, not the interface&amp;nbsp;alone.&lt;/strong&gt; &lt;code&gt;pass in ... to fdce:73f7:a2dc:d::30 port 8080&lt;/code&gt; will not accidentally start matching if the interface ever picks up an additional&amp;nbsp;address.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The ICMPv6 rules are not&amp;nbsp;optional.&lt;/strong&gt; &lt;code&gt;neighbrsol&lt;/code&gt;/&lt;code&gt;neighbradv&lt;/code&gt; are neighbor discovery - without them IPv6 on the segment simply dies.&amp;nbsp;And &lt;code&gt;toobig&lt;/code&gt; (Packet Too Big) matters more in &lt;span class="caps"&gt;DN42&lt;/span&gt; than almost anywhere else: the mesh is built from WireGuard tunnels with MTUs like 1360, so path &lt;span class="caps"&gt;MTU&lt;/span&gt; discovery is doing real work on nearly every transfer.&amp;nbsp;Drop &lt;code&gt;toobig&lt;/code&gt; and large package downloads will stall in exactly the way that takes an evening of tcpdump to&amp;nbsp;diagnose.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;set loginterface $cache_v6_if&lt;/code&gt;&lt;/strong&gt; gives me per-interface counters on the &lt;span class="caps"&gt;DN42&lt;/span&gt; side&amp;nbsp;via &lt;code&gt;pfctl -s info&lt;/code&gt; - a cheap way to watch how much the public service is actually being&amp;nbsp;used.&lt;/p&gt;
&lt;h2 id="nginx-a-lazy-mirror-with-proxy_store"&gt;nginx: A Lazy Mirror with&amp;nbsp;proxy_store&lt;/h2&gt;
&lt;p&gt;The interesting design decision&amp;nbsp;is &lt;code&gt;proxy_store&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;proxy_cache&lt;/code&gt;. They sound similar and behave very&amp;nbsp;differently:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;proxy_cache&lt;/code&gt;&lt;/strong&gt; is a real cache: hashed object names in a private directory layout, TTLs, revalidation, eviction. Great for accelerating a website, opaque on&amp;nbsp;disk.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;proxy_store&lt;/code&gt;&lt;/strong&gt; is a mirror-maker: it saves the fetched response as a plain file whose path matches the request &lt;span class="caps"&gt;URI&lt;/span&gt;,&amp;nbsp;permanently. &lt;code&gt;GET /FreeBSD:15:amd64/quarterly/data.pkg&lt;/code&gt; lands on disk&amp;nbsp;at &lt;code&gt;/var/cache/nginx/pkg/FreeBSD:15:amd64/quarterly/data.pkg&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For a package&amp;nbsp;mirror, &lt;code&gt;proxy_store&lt;/code&gt; is the right shape. The on-disk tree is the &lt;span class="caps"&gt;URL&lt;/span&gt; space, which&amp;nbsp;means &lt;code&gt;try_files&lt;/code&gt; can serve hits directly with zero proxy involvement, I can inspect the cache&amp;nbsp;with &lt;code&gt;ls&lt;/code&gt;, prune it&amp;nbsp;with &lt;code&gt;find&lt;/code&gt;, and the whole thing degrades gracefully into &amp;#8220;just a directory of files that nginx serves&amp;#8221;. After a while the cache tree looks like a partial, demand-driven copy of the&amp;nbsp;upstream:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/var/cache/nginx
├── freebsd-releases
│   └── pub/FreeBSD/releases/amd64/15.1-RELEASE/...  # base.txz &amp;amp; friends
├── freebsd-update
│   ├── 15.0-RELEASE/amd64/...
│   ├── 15.1-RELEASE/amd64/...
│   └── to-15.1-RELEASE/amd64/bp/...   # binary patches, fetched during upgrades
└── pkg
    ├── FreeBSD:13:amd64/
    ├── FreeBSD:14:powerpc64/
    ├── FreeBSD:15:amd64/
    │   ├── base_release_0/
    │   ├── kmods_quarterly_0/
    │   └── quarterly/
    └── FreeBSD:16:aarch64/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;FreeBSD:14:powerpc64&lt;/code&gt; directory is not mine - that is the public-service part of the story. Someone on &lt;span class="caps"&gt;DN42&lt;/span&gt; patches a powerpc64 machine through this box, and the cache grows a branch I will never use myself. That is the lazy mirror working exactly as&amp;nbsp;intended.&lt;/p&gt;
&lt;h3 id="the-pkg-mirror-port-8080"&gt;The pkg Mirror (Port&amp;nbsp;8080)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[fdce:73f7:a2dc:d::30]:8080&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ipv6only=on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;bsdmirror.chofstede.dn42&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# A resolver is required for dynamic upstream DNS queries&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;resolver&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="s"&gt;.8.8.8&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Define the local storage root&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;root&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/var/cache/nginx/pkg&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# Try serving the local file from disk first, fallback to upstream&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;try_files&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$uri&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@pkg_upstream&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@pkg_upstream&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store_access&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="s"&gt;user:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;group:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;all:r&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_pass&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="s"&gt;https://pkg.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="s"&gt;Host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;pkg.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_ssl_name&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="s"&gt;pkg.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_ssl_server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_intercept_errors&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;recursive_error_pages&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;error_page&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;301&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;302&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;307&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@pkg_redirect&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@pkg_redirect&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$redir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$upstream_http_location&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="c1"&gt;# the Location header&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_pass&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;$redir&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_ssl_server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store_access&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;user:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;group:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;all:r&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_temp_path&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;/var/cache/nginx/pkg_tmp&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The flow for a cache&amp;nbsp;miss: &lt;code&gt;try_files&lt;/code&gt; fails to find the file on disk, control falls through&amp;nbsp;to &lt;code&gt;@pkg_upstream&lt;/code&gt;, nginx fetches&amp;nbsp;from &lt;code&gt;pkg.freebsd.org&lt;/code&gt; over &lt;span class="caps"&gt;TLS&lt;/span&gt;, streams the response to the client,&amp;nbsp;and &lt;code&gt;proxy_store&lt;/code&gt; writes a copy into the tree. The next request for the same &lt;span class="caps"&gt;URI&lt;/span&gt; never leaves the&amp;nbsp;box.&lt;/p&gt;
&lt;p&gt;The part that took actual thought is the redirect&amp;nbsp;handling. &lt;code&gt;pkg.freebsd.org&lt;/code&gt; is not a file server - it frequently answers with an &lt;span class="caps"&gt;HTTP&lt;/span&gt; redirect to a &lt;span class="caps"&gt;CDN&lt;/span&gt; mirror. A naive proxy would forward that 302 to the client, and the client - a &lt;span class="caps"&gt;DN42&lt;/span&gt; host with no clearnet route - would then try to follow&amp;nbsp;a &lt;code&gt;Location:&lt;/code&gt; header pointing at a host it can never reach. Dead&amp;nbsp;end.&lt;/p&gt;
&lt;p&gt;The fix is&amp;nbsp;the &lt;code&gt;error_page 301 302 307 = @pkg_redirect&lt;/code&gt; construction.&amp;nbsp;With &lt;code&gt;proxy_intercept_errors on&lt;/code&gt;, nginx treats the upstream&amp;#8217;s redirect as an internal event instead of a response to relay. The named&amp;nbsp;location &lt;code&gt;@pkg_redirect&lt;/code&gt; grabs&amp;nbsp;the &lt;code&gt;Location&lt;/code&gt; header&amp;nbsp;via &lt;code&gt;$upstream_http_location&lt;/code&gt;, issues a second upstream request to wherever the redirect pointed, and stores &lt;em&gt;that&lt;/em&gt; response under the original &lt;span class="caps"&gt;URI&lt;/span&gt;. The client sees exactly one request and&amp;nbsp;one &lt;code&gt;200&lt;/code&gt; with file contents. The redirect chase happens entirely on the clearnet side of the&amp;nbsp;box.&lt;/p&gt;
&lt;p&gt;Two supporting details make this work:&amp;nbsp;the &lt;code&gt;resolver&lt;/code&gt; directive, because&amp;nbsp;a &lt;code&gt;proxy_pass&lt;/code&gt; with a variable target forces nginx to resolve hostnames at request time rather than config-load time;&amp;nbsp;and &lt;code&gt;proxy_ssl_server_name on&lt;/code&gt; in the redirect location, because the &lt;span class="caps"&gt;CDN&lt;/span&gt; endpoint needs a proper &lt;span class="caps"&gt;SNI&lt;/span&gt; handshake and its hostname is only known at&amp;nbsp;runtime.&lt;/p&gt;
&lt;h3 id="the-freebsd-update-mirror-port-80"&gt;The freebsd-update Mirror (Port&amp;nbsp;80)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[fdce:73f7:a2dc:d::30]:80&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;bsdmirror.chofstede.dn42&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;access_log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/var/log/nginx/update.access.log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;error_log&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;/var/log/nginx/update.error.log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Local storage for base system patches&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;root&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/var/cache/nginx/freebsd-update&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sr"&gt;^/[\w.-]+/[\w.-]+/(latest|pub)\.ssl$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;http://update.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;update.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;try_files&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$uri&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@update_upstream&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@update_upstream&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store_access&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="s"&gt;user:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;group:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;all:r&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_pass&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="s"&gt;http://update.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="s"&gt;Host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;update.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_redirect&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="no"&gt;off&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This one is simpler&amp;nbsp;- &lt;code&gt;update.freebsd.org&lt;/code&gt; serves plain &lt;span class="caps"&gt;HTTP&lt;/span&gt; without redirect games - but it has one rule that carries the whole design: the regex location&amp;nbsp;for &lt;code&gt;latest.ssl&lt;/code&gt; and &lt;code&gt;pub.ssl&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Those files are the signed pointers that&amp;nbsp;tell &lt;code&gt;freebsd-update&lt;/code&gt; what the current patch level is. Everything else in the update tree is content-addressed - metadata and patch files are named by their &lt;span class="caps"&gt;SHA256&lt;/span&gt; hashes, so a cached copy is either bit-identical to upstream or useless, and caching them forever is perfectly safe.&amp;nbsp;But &lt;code&gt;latest.ssl&lt;/code&gt; changes every time a new patch level is released. Cache it once&amp;nbsp;with &lt;code&gt;proxy_store&lt;/code&gt; (which has no concept of expiry) and every client behind the mirror is frozen at whatever patch level existed the day the cache was primed - silently, with valid signatures, forever. So the signature pointers are passed through uncached on every request, and only the immutable hash-named content gets&amp;nbsp;stored.&lt;/p&gt;
&lt;p&gt;That is also the answer to the obvious &amp;#8220;why port 8080 for pkg and port 80 for updates&amp;#8221; question, by the&amp;nbsp;way: &lt;code&gt;freebsd-update.conf&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s &lt;code&gt;ServerName&lt;/code&gt; directive accepts a hostname and nothing else - no port syntax. The update mirror must live on port 80. pkg&amp;#8217;s repository URLs are full URLs, so the pkg mirror can sit anywhere; 8080 keeps the services&amp;nbsp;separated.&lt;/p&gt;
&lt;h3 id="the-release-mirror-port-8081"&gt;The Release Mirror (Port&amp;nbsp;8081)&lt;/h3&gt;
&lt;p&gt;The third server block came later, when I wanted to bootstrap &lt;a href="https://bastillebsd.org/"&gt;Bastille&lt;/a&gt; jails on a &lt;span class="caps"&gt;DN42&lt;/span&gt;-only&amp;nbsp;host. &lt;code&gt;bastille bootstrap&lt;/code&gt; fetches release distribution sets&amp;nbsp;(&lt;code&gt;base.txz&lt;/code&gt;, &lt;code&gt;MANIFEST&lt;/code&gt;)&amp;nbsp;from &lt;code&gt;ftp.freebsd.org&lt;/code&gt;, which is yet another clearnet host the machine cannot reach - and yet another upstream that redirects to geolocated &lt;span class="caps"&gt;CDN&lt;/span&gt;&amp;nbsp;mirrors:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[fdce:73f7:a2dc:d::30]:8081&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ipv6only=on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;bsdmirror.chofstede.dn42&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;resolver&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="s"&gt;.8.8.8&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Local storage for base installation tarballs&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;root&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/var/cache/nginx/freebsd-releases&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;try_files&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$uri&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@releases_upstream&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@releases_upstream&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store_access&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="s"&gt;user:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;group:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;all:r&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_pass&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="s"&gt;https://ftp.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="s"&gt;Host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ftp.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_ssl_name&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="s"&gt;ftp.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_ssl_server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# Handle mirrors that redirect to geolocated CDN mirrors smoothly&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_intercept_errors&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;recursive_error_pages&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;error_page&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;301&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;302&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;307&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@releases_redirect&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@releases_redirect&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$redir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$upstream_http_location&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_pass&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nv"&gt;$redir&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_ssl_name&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="s"&gt;ftp.freebsd.org&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_ssl_server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_store_access&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;user:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;group:rw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;all:r&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Structurally this is the pkg mirror&amp;nbsp;again: &lt;code&gt;try_files&lt;/code&gt; for&amp;nbsp;hits, &lt;code&gt;proxy_store&lt;/code&gt; for misses, and the&amp;nbsp;same &lt;code&gt;error_page&lt;/code&gt;/&lt;code&gt;$upstream_http_location&lt;/code&gt; pair to absorb redirects server-side. Release tarballs are the best possible content for this pattern -&amp;nbsp;a &lt;code&gt;15.1-RELEASE/base.txz&lt;/code&gt; will never change for the rest of its existence, so a stored copy is valid forever, and at 156 &lt;span class="caps"&gt;MB&lt;/span&gt; apiece the bandwidth savings per repeat bootstrap are the largest of the three&amp;nbsp;services.&lt;/p&gt;
&lt;h3 id="what-about-trust"&gt;What About&amp;nbsp;Trust?&lt;/h3&gt;
&lt;p&gt;A caching proxy in the middle of your update path sounds like a place where supply-chain nightmares are born, so it is worth being precise about what the mirror can and cannot&amp;nbsp;do.&lt;/p&gt;
&lt;p&gt;It cannot&amp;nbsp;tamper. &lt;code&gt;freebsd-update&lt;/code&gt; verifies every metadata file against signatures made with FreeBSD&amp;#8217;s release keys, and the content files are verified against &lt;span class="caps"&gt;SHA256&lt;/span&gt; hashes from that signed&amp;nbsp;metadata. &lt;code&gt;pkg&lt;/code&gt; is configured&amp;nbsp;with &lt;code&gt;signature_type: "fingerprints"&lt;/code&gt; against the fingerprints shipped in the FreeBSD base system&amp;nbsp;(&lt;code&gt;/usr/share/keys/pkg&lt;/code&gt;). Bastille validates each downloaded distribution set against the &lt;span class="caps"&gt;SHA256&lt;/span&gt; in the&amp;nbsp;release &lt;code&gt;MANIFEST&lt;/code&gt;. In every case the trust anchor lives on the client, and the mirror is just an untrusted byte transport. If it corrupts or manipulates a file, verification fails and the client refuses&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;What the mirror &lt;em&gt;can&lt;/em&gt; do is serve stale data - which is exactly&amp;nbsp;why &lt;code&gt;latest.ssl&lt;/code&gt; and &lt;code&gt;pub.ssl&lt;/code&gt; bypass the cache, and why pkg&amp;#8217;s mutable repository metadata needs the housekeeping described&amp;nbsp;below.&lt;/p&gt;
&lt;h2 id="client-configuration"&gt;Client&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;On a &lt;span class="caps"&gt;DN42&lt;/span&gt;-connected FreeBSD machine, three files change. For the base&amp;nbsp;system, &lt;code&gt;/etc/freebsd-update.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ServerName bsdmirror.chofstede.dn42
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For packages, a repository override. I&amp;nbsp;use &lt;code&gt;/etc/pkg/FreeBSD.conf&lt;/code&gt; variants pointing at the mirror, keeping the fingerprint verification exactly as&amp;nbsp;stock:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;FreeBSD-ports: {
  url: &amp;quot;http://bsdmirror.chofstede.dn42:8080/${ABI}/quarterly&amp;quot;,
  mirror_type: &amp;quot;none&amp;quot;,
  signature_type: &amp;quot;fingerprints&amp;quot;,
  fingerprints: &amp;quot;/usr/share/keys/pkg&amp;quot;,
  enabled: yes
}
FreeBSD-ports-kmods: {
  url: &amp;quot;http://bsdmirror.chofstede.dn42:8080/${ABI}/kmods_quarterly_${VERSION_MINOR}&amp;quot;,
  mirror_type: &amp;quot;none&amp;quot;,
  signature_type: &amp;quot;fingerprints&amp;quot;,
  fingerprints: &amp;quot;/usr/share/keys/pkg&amp;quot;,
  enabled: yes
}
FreeBSD-base: {
  url: &amp;quot;http://bsdmirror.chofstede.dn42:8080/${ABI}/base_release_${VERSION_MINOR}&amp;quot;,
  mirror_type: &amp;quot;none&amp;quot;,
  signature_type: &amp;quot;fingerprints&amp;quot;,
  fingerprints: &amp;quot;/usr/share/keys/pkgbase-${VERSION_MAJOR}&amp;quot;,
  enabled: no
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;mirror_type: "none"&lt;/code&gt; matters: the stock configuration&amp;nbsp;uses &lt;code&gt;mirror_type: "srv"&lt;/code&gt;, which makes pkg look up &lt;span class="caps"&gt;SRV&lt;/span&gt; records to pick a mirror. Pointing at a single explicit host, that lookup is pointless at best and harmful at worst&amp;nbsp;- &lt;code&gt;"none"&lt;/code&gt; tells pkg to use the &lt;span class="caps"&gt;URL&lt;/span&gt; exactly as&amp;nbsp;written.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;${ABI}&lt;/code&gt; and &lt;code&gt;${VERSION_MINOR}&lt;/code&gt; expansions are what make the same config file portable across every FreeBSD machine in the mesh regardless of version and architecture - and what naturally grows the cache tree per-&lt;span class="caps"&gt;ABI&lt;/span&gt; on the mirror side, like&amp;nbsp;that &lt;code&gt;FreeBSD:14:powerpc64&lt;/code&gt; branch.&lt;/p&gt;
&lt;p&gt;And for jail hosts, one line&amp;nbsp;in &lt;code&gt;/usr/local/etc/bastille/bastille.conf&lt;/code&gt; points release bootstrapping at the&amp;nbsp;mirror:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille_url_freebsd=&amp;quot;http://bsdmirror.chofstede.dn42:8081/pub/FreeBSD/releases/&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The path&amp;nbsp;mirrors &lt;code&gt;ftp.freebsd.org&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s layout exactly, because the cache tree &lt;em&gt;is&lt;/em&gt; that layout&amp;nbsp;- &lt;code&gt;proxy_store&lt;/code&gt; again.&lt;/p&gt;
&lt;h2 id="the-proof-150-to-151-over-the-mesh"&gt;The Proof: 15.0 to 15.1 Over the&amp;nbsp;Mesh&lt;/h2&gt;
&lt;p&gt;The real test was &lt;a href="https://hofstede.it/upgrading-freebsd-150-release-to-151-release-the-official-paths/"&gt;the 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; upgrade&lt;/a&gt; on a &lt;span class="caps"&gt;DN42&lt;/span&gt;-only host, running entirely through the&amp;nbsp;mirror:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@nethost:~ # freebsd-update -r 15.1-RELEASE upgrade
src component not installed, skipped
Looking up bsdmirror.chofstede.dn42 mirrors... none found.
Fetching metadata signature for 15.0-RELEASE from bsdmirror.chofstede.dn42... done.
Fetching metadata index... done.
Inspecting system... done.

The following components of FreeBSD seem to be installed:
kernel/generic world/base

Does this look reasonable (y/n)? y

Fetching metadata signature for 15.1-RELEASE from bsdmirror.chofstede.dn42... done.
Fetching 1 metadata patches. done.
Applying metadata patches... done.
Preparing to download files... done.
Fetching 3194 patches.....10....20....30 [...] ....3190.. done.
Applying patches... done.
Fetching 661 files... ....10....20 [...] ....660 done.

To install the downloaded upgrades, run &amp;#39;freebsd-update [options] install&amp;#39;.
root@nethost:~ # freebsd-update install
Creating snapshot of existing boot environment... done.
Installing updates...
Kernel updates have been installed.  Please reboot and run
&amp;#39;freebsd-update [options] install&amp;#39; again to finish installing updates.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;3194 binary patches and 661 full files, fetched over WireGuard tunnels through the &lt;span class="caps"&gt;DN42&lt;/span&gt; mesh, cached on the way through - the second machine I upgraded pulled almost everything from local disk. Those patches are&amp;nbsp;the &lt;code&gt;to-15.1-RELEASE/amd64/bp/&lt;/code&gt; directory in the cache tree&amp;nbsp;above.&lt;/p&gt;
&lt;p&gt;One line worth&amp;nbsp;demystifying: &lt;code&gt;Looking up bsdmirror.chofstede.dn42 mirrors... none found.&lt;/code&gt; is not an&amp;nbsp;error. &lt;code&gt;freebsd-update&lt;/code&gt; queries &lt;span class="caps"&gt;SRV&lt;/span&gt; records&amp;nbsp;(&lt;code&gt;_http._tcp.&amp;lt;ServerName&amp;gt;&lt;/code&gt;) to discover mirror pools; a plain hostname with no &lt;span class="caps"&gt;SRV&lt;/span&gt; records yields &amp;#8220;none found&amp;#8221; and a clean fallback to the hostname itself. Everything after that line is the answer that&amp;nbsp;matters.&lt;/p&gt;
&lt;p&gt;And the same host, bootstrapping a Bastille jail from the release&amp;nbsp;mirror:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@nethost:~ # bastille bootstrap 15.1-RELEASE

Bootstrapping FreeBSD release: 15.1-RELEASE

Fetching MANIFEST...
/usr/local/bastille/cache/15.1-RELEASE/MANIFES        1044  B   11 MBps    00s

Fetching distfile: base.txz
/usr/local/bastille/cache/15.1-RELEASE/base.tx         156 MB 3530 kBps    45s

Validating checksum for archive: base.txz
MANIFEST: 3768988b151c20f965679062b065c63a977d6bbb9f47fd83695ec2c40790c18f
DOWNLOAD: 3768988b151c20f965679062b065c63a977d6bbb9f47fd83695ec2c40790c18f

Extracting archive: base.txz

Bootstrap successful!
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;156 &lt;span class="caps"&gt;MB&lt;/span&gt;&amp;nbsp;of &lt;code&gt;base.txz&lt;/code&gt; at 3.5 &lt;span class="caps"&gt;MB&lt;/span&gt;/s through the mesh, checksum-verified against the &lt;span class="caps"&gt;MANIFEST&lt;/span&gt; on arrival - and now sitting in the cache, so the next jail host that bootstraps 15.1 gets it at local-network&amp;nbsp;speed.&lt;/p&gt;
&lt;h3 id="the-full-jail-lifecycle"&gt;The Full Jail&amp;nbsp;Lifecycle&lt;/h3&gt;
&lt;p&gt;What makes this satisfying is that the &lt;em&gt;entire&lt;/em&gt; Bastille jail lifecycle now runs through the mirror, because every mechanism Bastille wraps happens to be one of the three&amp;nbsp;services. &lt;code&gt;bastille etcupdate bootstrap&lt;/code&gt; pulls &lt;code&gt;src.txz&lt;/code&gt; (another 241 &lt;span class="caps"&gt;MB&lt;/span&gt;) through the release mirror&amp;nbsp;on &lt;code&gt;:8081&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@nethost:~ # bastille etcupdate bootstrap 15.1-RELEASE

Attempting to bootstrap etcupdate release: 15.1-RELEASE...

Fetching distfile: src.txz
/usr/local/bastille/cache/15.1-RELEASE/src.txz         241 MB 3557 kBps 01m09s

Validating checksum for archive: src.txz
MANIFEST: cf5762da53fd52e1eaf0f9ceee9bf58cbe314c821031d0d9ffa76823185a89e1
DOWNLOAD: cf5762da53fd52e1eaf0f9ceee9bf58cbe314c821031d0d9ffa76823185a89e1

Etcupdate bootstrap complete: 15.1-RELEASE
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;bastille update&lt;/code&gt; patches the bootstrapped release&amp;nbsp;with &lt;code&gt;freebsd-update&lt;/code&gt; under the hood, so it lands on&amp;nbsp;the &lt;code&gt;:80&lt;/code&gt; mirror via the&amp;nbsp;host&amp;#8217;s &lt;code&gt;ServerName&lt;/code&gt; setting:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@nethost:~ # bastille update 15.1-RELEASE

Attempting to update release: 15.1-RELEASE
Looking up bsdmirror.chofstede.dn42 mirrors... none found.
Fetching public key from bsdmirror.chofstede.dn42... done.
Fetching metadata signature for 15.1-RELEASE from bsdmirror.chofstede.dn42... done.
Fetching 105 patches.....10....20 [...] ....100.. done.
Applying patches... done.
Installing updates... done.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And packages inside the jails come from&amp;nbsp;the &lt;code&gt;:8080&lt;/code&gt; pkg mirror, since the jails inherit the same repository&amp;nbsp;configuration:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@nethost:~ # bastille pkg ALL upgrade

[nethack]:
Installed packages to be UPGRADED:
        ldns: 1.9.0 -&amp;gt; 1.9.2 [FreeBSD-ports]
        nginx: 1.28.3,3 -&amp;gt; 1.30.3,3 [FreeBSD-ports]
        php85: 8.5.4 -&amp;gt; 8.5.6 [FreeBSD-ports]
        php85-sqlite3: 8.5.4 -&amp;gt; 8.5.6 [FreeBSD-ports]
        python311: 3.11.15_2 -&amp;gt; 3.11.15_4 [FreeBSD-ports]

[nethack] [3/5] Fetching php85-8.5.6: 100%    13 MiB   3.3 MB/s    00:04
[nethack] [5/5] Fetching python311-3.11.15_4: 100%    27 MiB   3.5 MB/s    00:08
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Bootstrap a release, patch it, populate its jails with packages - a jail host inside &lt;span class="caps"&gt;DN42&lt;/span&gt; goes from empty to fully patched and provisioned without ever holding a route to the&amp;nbsp;clearnet.&lt;/p&gt;
&lt;h2 id="operational-notes"&gt;Operational&amp;nbsp;Notes&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;proxy_store never expires anything, and pkg metadata is mutable.&lt;/strong&gt; This is the one sharp edge of the design. The hash-named freebsd-update content is immutable and safe to keep forever, but pkg repository metadata&amp;nbsp;(&lt;code&gt;meta.conf&lt;/code&gt;, &lt;code&gt;data.pkg&lt;/code&gt;, &lt;code&gt;packagesite.pkg&lt;/code&gt;) changes in place upstream whenever the quarterly branch gets updates. A stored copy pins clients to that snapshot of the repository. The pragmatic fix is a periodic cleanup job that drops the metadata files and lets the next request re-fetch&amp;nbsp;them:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/cron.d/bsdmirror - refresh pkg repo metadata daily&lt;/span&gt;
&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;root&lt;span class="w"&gt; &lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;/var/cache/nginx/pkg&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="se"&gt;\(&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;meta.conf&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;data.pkg&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;packagesite.*&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-mtime&lt;span class="w"&gt; &lt;/span&gt;+1&lt;span class="w"&gt; &lt;/span&gt;-delete
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Individual package files are versioned in their filenames, so they never need invalidation - old versions just stop being requested and can be pruned by age whenever disk pressure suggests it. Release tarballs are the same&amp;nbsp;story: &lt;code&gt;15.1-RELEASE/base.txz&lt;/code&gt; is immutable for its lifetime, and an entire release directory can be dropped wholesale once that release is &lt;span class="caps"&gt;EOL&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Disk usage stays civil.&lt;/strong&gt; A lazy mirror only holds what was actually requested. After the 15.1 upgrade wave plus normal package traffic across several ABIs, the cache sits comfortably in the low tens of gigabytes - compared to the multi-terabyte footprint of a full&amp;nbsp;mirror.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Watch the temp&amp;nbsp;path.&lt;/strong&gt; &lt;code&gt;proxy_temp_path /var/cache/nginx/pkg_tmp&lt;/code&gt; keeps in-flight downloads on the same filesystem as the store, so the final placement is an atomic rename rather than a cross-device copy. If interrupted transfers ever litter that directory, they are safe to&amp;nbsp;delete.&lt;/p&gt;
&lt;h2 id="public-dn42-service"&gt;Public &lt;span class="caps"&gt;DN42&lt;/span&gt;&amp;nbsp;Service&lt;/h2&gt;
&lt;div class="about-cta"&gt;
&lt;p&gt;&lt;strong&gt;Running FreeBSD on &lt;span class="caps"&gt;DN42&lt;/span&gt;?&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;code&gt;bsdmirror.chofstede.dn42&lt;/code&gt; is open to the whole mesh.&amp;nbsp;Point &lt;code&gt;ServerName&lt;/code&gt; in &lt;code&gt;/etc/freebsd-update.conf&lt;/code&gt; at it,&amp;nbsp;use &lt;code&gt;http://bsdmirror.chofstede.dn42:8080/${ABI}/quarterly&lt;/code&gt; as a pkg repository &lt;span class="caps"&gt;URL&lt;/span&gt;, or fetch release sets&amp;nbsp;from &lt;code&gt;http://bsdmirror.chofstede.dn42:8081/pub/FreeBSD/releases/&lt;/code&gt; for &lt;code&gt;bastille bootstrap&lt;/code&gt; and friends - signature and checksum verification against the official FreeBSD keys keeps working unchanged, so you are not extending any trust to me. If your architecture or release is not cached yet, the first fetch warms it for everyone after you. Feedback and peering requests: &lt;a href="mailto:peering@hofstede.it"&gt;peering@hofstede.it&lt;/a&gt;.&lt;/p&gt;
&lt;/div&gt;
&lt;h2 id="lessons-learned"&gt;Lessons&amp;nbsp;Learned&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;proxy_store is the forgotten middle ground between a cache and a mirror.&lt;/strong&gt; Everyone reaches&amp;nbsp;for &lt;code&gt;proxy_cache&lt;/code&gt; or a full rsync&amp;nbsp;mirror; &lt;code&gt;proxy_store&lt;/code&gt; gives you a demand-driven partial mirror with a human-readable on-disk layout for about fifteen lines of config. For content that is immutable and hash-named - which describes most package infrastructure - it is close to&amp;nbsp;ideal.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Know which files in a repository are mutable, and treat them differently.&lt;/strong&gt; The whole correctness story of this mirror rests on two observations: freebsd-update&amp;#8217;s content is hash-addressed and safe to store forever,&amp;nbsp;while &lt;code&gt;latest.ssl&lt;/code&gt;/&lt;code&gt;pub.ssl&lt;/code&gt; and pkg&amp;#8217;s repository metadata are pointers that must stay fresh. Get that split wrong and you build a mirror that silently freezes clients in the past, with valid signatures on stale&amp;nbsp;data.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Server-side redirect chasing keeps isolated clients isolated.&lt;/strong&gt;&amp;nbsp;The &lt;code&gt;error_page 301 302 307 = @named_location&lt;/code&gt; plus &lt;code&gt;$upstream_http_location&lt;/code&gt; pattern is obscure but powerful: the proxy absorbs the upstream&amp;#8217;s &lt;span class="caps"&gt;CDN&lt;/span&gt; topology so clients on a network that cannot reach that &lt;span class="caps"&gt;CDN&lt;/span&gt; never have to know it&amp;nbsp;exists.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Egress filtering is finally practical on single-purpose machines.&lt;/strong&gt; &amp;#8220;Default deny outbound&amp;#8221; is aspirational on a general-purpose server and trivial on a box whose entire job is fetching files from two well-known hostnames. The pf policy here is short enough to audit in one&amp;nbsp;screen.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;freebsd-update&amp;#8217;s constraints shape the design.&lt;/strong&gt; No port syntax&amp;nbsp;in &lt;code&gt;ServerName&lt;/code&gt;, &lt;span class="caps"&gt;SRV&lt;/span&gt;-based mirror discovery, signature files that rotate per patch level - none of this is documented as &amp;#8220;how to build a mirror&amp;#8221;, but all of it dictates one. Reading the client&amp;#8217;s behavior is the&amp;nbsp;specification.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;FreeBSD&amp;#8217;s entire update ecosystem funnels through three hostnames.&lt;/strong&gt; pkg, freebsd-update, and every Bastille subcommand - bootstrap, etcupdate, update, jail packages - ultimately fetch&amp;nbsp;from &lt;code&gt;pkg.freebsd.org&lt;/code&gt;, &lt;code&gt;update.freebsd.org&lt;/code&gt;,&amp;nbsp;or &lt;code&gt;ftp.freebsd.org&lt;/code&gt;. Mirror those three and there is no fourth thing to chase; a jail host can go from empty to fully provisioned without a clearnet route. That coherence is a quiet advantage of a base-system operating system, and it is what made this project three nginx server blocks instead of a zoo of special&amp;nbsp;cases.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A private itch makes a decent public service.&lt;/strong&gt; The marginal cost of opening this to all of &lt;span class="caps"&gt;DN42&lt;/span&gt; was one pf rule scoped to the mesh and a line in the wiki. In exchange, the cache warms itself with architectures I do not run, and other people&amp;#8217;s isolated FreeBSD boxes get a patch path. That trade is the &lt;span class="caps"&gt;DN42&lt;/span&gt; ethos in&amp;nbsp;miniature.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;There is a small FreeBSD &lt;span class="caps"&gt;VM&lt;/span&gt; in my network with a foot in two worlds: one interface that may only fetch from the FreeBSD project&amp;#8217;s servers, and one that may only serve the hobbyist mesh. Between them sit nginx, a directory tree that grows to mirror exactly the slice of FreeBSD&amp;#8217;s infrastructure my machines (and now other people&amp;#8217;s machines) actually use, and a pf policy short enough to&amp;nbsp;memorize.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;DN42&lt;/span&gt;-side hosts got what they needed - a patch path without a clearnet route - and the design stays honest about trust: the mirror moves bytes, the clients verify them, and the only real failure mode (staleness) is engineered around by refusing to cache the few files that change. If you run FreeBSD inside &lt;span class="caps"&gt;DN42&lt;/span&gt;, the mirror is yours to&amp;nbsp;use.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_store"&gt;nginx proxy_store documentation&lt;/a&gt; - the lazy-mirror directive at the heart of this&amp;nbsp;setup&lt;/li&gt;
&lt;li&gt;&lt;a href="https://nginx.org/en/docs/http/ngx_http_core_module.html#error_page"&gt;nginx error_page directive&lt;/a&gt; -&amp;nbsp;the &lt;code&gt;= @named_location&lt;/code&gt; form used for redirect&amp;nbsp;chasing&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=freebsd-update.conf&amp;amp;sektion=5"&gt;freebsd-update.conf(5) man page&lt;/a&gt;&amp;nbsp;- &lt;code&gt;ServerName&lt;/code&gt; and client&amp;nbsp;behavior&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pkg.conf&amp;amp;sektion=5"&gt;pkg.conf(5) man page&lt;/a&gt; - repository&amp;nbsp;configuration, &lt;code&gt;mirror_type&lt;/code&gt;, signature&amp;nbsp;verification&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pf.conf&amp;amp;sektion=5"&gt;pf.conf(5) man page&lt;/a&gt; - packet filter&amp;nbsp;configuration&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastillebsd.org/"&gt;Bastille&lt;/a&gt; - the jail manager&amp;nbsp;whose &lt;code&gt;bootstrap&lt;/code&gt; command the release mirror&amp;nbsp;serves&lt;/li&gt;
&lt;li&gt;&lt;a href="https://wiki.dn42.dev/services/Services"&gt;&lt;span class="caps"&gt;DN42&lt;/span&gt; Wiki: Services&lt;/a&gt; - the directory of public services on the&amp;nbsp;mesh&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/joining-dn42-a-mikrotik-border-three-wireguard-peerings-and-a-freebsd-jail-in-the-hobbyist-internet/"&gt;Joining &lt;span class="caps"&gt;DN42&lt;/span&gt;: A MikroTik Border, Three WireGuard Peerings, and a FreeBSD Jail&lt;/a&gt; - how this network came to&amp;nbsp;exist&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/upgrading-freebsd-150-release-to-151-release-the-official-paths/"&gt;Upgrading FreeBSD 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; to 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;&lt;/a&gt; - the upgrade that served as the end-to-end&amp;nbsp;test&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="dn42"/><category term="nginx"/><category term="pf"/><category term="caching"/><category term="pkg"/><category term="freebsd-update"/><category term="ipv6"/></entry><entry><title>Upgrading FreeBSD 15.0-RELEASE to 15.1-RELEASE: The Official Paths</title><link href="https://hofstede.it/upgrading-freebsd-150-release-to-151-release-the-official-paths/" rel="alternate"/><published>2026-07-01T00:00:00+02:00</published><updated>2026-07-01T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-07-01:/upgrading-freebsd-150-release-to-151-release-the-official-paths/</id><summary type="html">&lt;p&gt;FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; is here, and the official upgrade instructions depend on whether your system uses distribution sets or packaged base. This guide walks through both supported paths, covers the boot-loader update, and explains the configuration merging process - all based on the official 15.1 upgrading&amp;nbsp;documentation.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="FreeBSD 15.1 Upgrade Guide" src="https://hofstede.it/images/2026-07-01-freebsd-15.1-upgrade-hero.png" title="FreeBSD 15.0 to 15.1 upgrade guide: two paths forward"&gt;&lt;/p&gt;
&lt;p&gt;FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; was announced on June 16, 2026. If you&amp;#8217;re running 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;, you have a well-defined upgrade path - but the exact steps depend on how your base system is managed. The first question isn&amp;#8217;t &amp;#8220;which tool should I use?&amp;#8221; but &amp;#8220;how is my base system&amp;nbsp;installed?&amp;#8221;&lt;/p&gt;
&lt;p&gt;This article follows the official &lt;a href="https://www.freebsd.org/releases/15.1R/upgrading/"&gt;15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; upgrading instructions&lt;/a&gt;, explains the pre-upgrade detection, walks through both supported procedures (distribution-set and packaged-base), and covers the boot-loader update step included in the official&amp;nbsp;procedure.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Update (2026-07-02):&lt;/strong&gt; An earlier version put the boot-loader update after the first reboot in&amp;nbsp;the &lt;code&gt;freebsd-update&lt;/code&gt; path and did not&amp;nbsp;mention &lt;code&gt;FreeBSD-ports-kmods&lt;/code&gt; there. The sequence below now updates the boot loader before the first attempt to boot 15.1 and includes the ports-kmods repository step for systems using out-of-tree kernel modules. Thanks to &lt;a href="https://bsd.cafe/@grahamperrin"&gt;Graham Perrin (@grahamperrin@bsd.cafe)&lt;/a&gt; for the&amp;nbsp;correction.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Support timeline:&lt;/strong&gt; 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; is supported until March 31, 2027. 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; reaches &lt;span class="caps"&gt;EOL&lt;/span&gt; on September 30, 2026. The 15.x series as a whole is supported until December 31,&amp;nbsp;2029.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#before-you-begin-the-universal-checklist"&gt;Before You Begin: The Universal&amp;nbsp;Checklist&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#path-a-distribution-set-systems-freebsd-update"&gt;Path A: Distribution-Set Systems (freebsd-update)&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#step-1-apply-pending-patches-first"&gt;Step 1: Apply Pending Patches&amp;nbsp;First&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-2-upgrade-to-151-release"&gt;Step 2: Upgrade to 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-3-install-the-kernel-and-update-the-boot-loader"&gt;Step 3: Install the Kernel and Update the Boot&amp;nbsp;Loader&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-4-finish-the-userland-update"&gt;Step 4: Finish the Userland&amp;nbsp;Update&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-5-handle-third-party-packages"&gt;Step 5: Handle Third-Party&amp;nbsp;Packages&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-6-final-cleanup"&gt;Step 6: Final&amp;nbsp;Cleanup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-7-check-for-failed-configuration-updates"&gt;Step 7: Check for Failed Configuration&amp;nbsp;Updates&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-8-reboot"&gt;Step 8:&amp;nbsp;Reboot&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#path-b-packaged-base-systems-pkg"&gt;Path B: Packaged-Base Systems (pkg)&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#step-1-create-a-rollback-boot-environment"&gt;Step 1: Create a Rollback Boot&amp;nbsp;Environment&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-2-upgrade-the-base-system-package"&gt;Step 2: Upgrade the Base System&amp;nbsp;Package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-3-apply-the-abi-and-version-settings"&gt;Step 3: Apply the &lt;span class="caps"&gt;ABI&lt;/span&gt; and Version&amp;nbsp;Settings&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-4-update-kernel-modules"&gt;Step 4: Update Kernel&amp;nbsp;Modules&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-5-check-for-failed-configuration-updates"&gt;Step 5: Check for Failed Configuration&amp;nbsp;Updates&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-6-update-the-boot-loader"&gt;Step 6: Update the Boot&amp;nbsp;Loader&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-7-reboot"&gt;Step 7:&amp;nbsp;Reboot&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#advanced-staging-into-a-mounted-boot-environment"&gt;Advanced: Staging into a Mounted Boot&amp;nbsp;Environment&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#boot-loader-update"&gt;Boot-Loader Update&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#uefi-systems"&gt;&lt;span class="caps"&gt;UEFI&lt;/span&gt;&amp;nbsp;Systems&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#bios-uefi-csm-systems"&gt;&lt;span class="caps"&gt;BIOS&lt;/span&gt; / &lt;span class="caps"&gt;UEFI&lt;/span&gt;-&lt;span class="caps"&gt;CSM&lt;/span&gt;&amp;nbsp;Systems&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#configuration-merging-the-pkgnew-dance"&gt;Configuration Merging: The .pkgnew Dance&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#finding-pkgnew-files"&gt;Finding .pkgnew&amp;nbsp;Files&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#etcupdate-for-advanced-merges"&gt;etcupdate for Advanced&amp;nbsp;Merges&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#common-pkgnew-files-to-watch-for"&gt;Common .pkgnew Files to Watch&amp;nbsp;For&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#post-upgrade-verification"&gt;Post-Upgrade&amp;nbsp;Verification&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#common-pitfalls-and-how-to-avoid-them"&gt;Common Pitfalls and How to Avoid&amp;nbsp;Them&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="before-you-begin-the-universal-checklist"&gt;Before You Begin: The Universal&amp;nbsp;Checklist&lt;/h2&gt;
&lt;p&gt;Before you start any upgrade, confirm a few basics and protect your ability to roll&amp;nbsp;back.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# 1. Confirm what you&amp;#39;re running&lt;/span&gt;
freebsd-version&lt;span class="w"&gt; &lt;/span&gt;-u
uname&lt;span class="w"&gt; &lt;/span&gt;-a

&lt;span class="c1"&gt;# 2. Read the 15.1 release notes and errata&lt;/span&gt;
&lt;span class="c1"&gt;# https://www.freebsd.org/releases/15.1R/relnotes/&lt;/span&gt;
&lt;span class="c1"&gt;# https://www.freebsd.org/releases/15.1R/errata/&lt;/span&gt;
&lt;span class="c1"&gt;# As of June 30, 2026, the errata includes security advisories for VM, jail,&lt;/span&gt;
&lt;span class="c1"&gt;# execve, ZFS, and libalias - review these before upgrading.&lt;/span&gt;

&lt;span class="c1"&gt;# 3. Determine how your base system is managed&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;which&lt;span class="w"&gt; &lt;/span&gt;/usr/bin/uname
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That last command is critical. It tells you which upgrade path applies to your&amp;nbsp;system:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;If it&amp;nbsp;returns &lt;code&gt;/usr/bin/uname was not installed by a package&lt;/code&gt;, you&amp;#8217;re using &lt;strong&gt;distribution sets&lt;/strong&gt; and should follow&amp;nbsp;the &lt;code&gt;freebsd-update&lt;/code&gt; path.&lt;/li&gt;
&lt;li&gt;If it returns a package name such&amp;nbsp;as &lt;code&gt;FreeBSD-runtime-15.0&lt;/code&gt;, you&amp;#8217;re using &lt;strong&gt;packaged base&lt;/strong&gt; and should follow the pkg-based&amp;nbsp;path.&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# 4. If booting from ZFS, create a snapshot or boot environment&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;zroot@before-15.1-upgrade

&lt;span class="c1"&gt;# Or, if you use bectl:&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;before-15.1-upgrade
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The snapshot or boot environment is your safety net. If 15.1 breaks something critical, you can roll back in&amp;nbsp;minutes.&lt;/p&gt;
&lt;h2 id="path-a-distribution-set-systems-freebsd-update"&gt;Path A: Distribution-Set Systems&amp;nbsp;(freebsd-update)&lt;/h2&gt;
&lt;p&gt;If &lt;code&gt;pkg which /usr/bin/uname&lt;/code&gt; reports that the file was not installed by a package, you&amp;#8217;re on the traditional distribution-set upgrade path. This is the&amp;nbsp;classic &lt;code&gt;freebsd-update&lt;/code&gt; workflow that FreeBSD has used for&amp;nbsp;years.&lt;/p&gt;
&lt;h3 id="step-1-apply-pending-patches-first"&gt;Step 1: Apply Pending Patches&amp;nbsp;First&lt;/h3&gt;
&lt;p&gt;Before starting the release upgrade, bring your current system fully up to date with all pending&amp;nbsp;patches:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-update&lt;span class="w"&gt; &lt;/span&gt;fetch
freebsd-update&lt;span class="w"&gt; &lt;/span&gt;install
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This ensures you&amp;#8217;re not compounding patch-level changes with the release upgrade&amp;nbsp;itself.&lt;/p&gt;
&lt;h3 id="step-2-upgrade-to-151-release"&gt;Step 2: Upgrade to 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Fetch the metadata for 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; and review what will&amp;nbsp;change:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-update&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.1-RELEASE
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The tool will show you a list of files it intends to update. Review this for anything unexpected, particularly in your configuration&amp;nbsp;files.&lt;/p&gt;
&lt;h3 id="step-3-install-the-kernel-and-update-the-boot-loader"&gt;Step 3: Install the Kernel and Update the Boot&amp;nbsp;Loader&lt;/h3&gt;
&lt;p&gt;Install the first stage of the upgrade. This installs the new kernel and boot&amp;nbsp;files:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-update&lt;span class="w"&gt; &lt;/span&gt;install
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Before rebooting, update your boot loader using the &lt;a href="#boot-loader-update"&gt;Boot-Loader Update&lt;/a&gt; section below. Do this before the first attempt to boot 15.1, especially on &lt;span class="caps"&gt;ZFS&lt;/span&gt;-root&amp;nbsp;systems.&lt;/p&gt;
&lt;p&gt;When the boot loader is current, reboot into the new&amp;nbsp;kernel:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;shutdown&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;+10min&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Rebooting for 15.1-RELEASE kernel upgrade&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;+10min&lt;/code&gt; gives connected users a graceful warning. Adjust to your&amp;nbsp;needs.&lt;/p&gt;
&lt;h3 id="step-4-finish-the-userland-update"&gt;Step 4: Finish the Userland&amp;nbsp;Update&lt;/h3&gt;
&lt;p&gt;After the reboot, complete the userland&amp;nbsp;update:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-update&lt;span class="w"&gt; &lt;/span&gt;install
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="step-5-handle-third-party-packages"&gt;Step 5: Handle Third-Party&amp;nbsp;Packages&lt;/h3&gt;
&lt;p&gt;At this&amp;nbsp;point, &lt;code&gt;freebsd-update&lt;/code&gt; may prompt you about rebuilding or reinstalling third-party software due to library changes. Follow those prompts as&amp;nbsp;needed.&lt;/p&gt;
&lt;p&gt;For a point release such as 15.0 → 15.1, a&amp;nbsp;normal &lt;code&gt;pkg upgrade&lt;/code&gt; is often&amp;nbsp;sufficient:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If the system uses out-of-tree kernel modules from the FreeBSD ports-kmods repository, update those as part of the same&amp;nbsp;pass:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-ports-kmods
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This matters for modules such as graphics drivers, VirtualBox, WireGuard variants, or other kernel modules installed outside the base system. They need to match the kernel you just&amp;nbsp;booted.&lt;/p&gt;
&lt;p&gt;Reserve &lt;code&gt;pkg upgrade -f&lt;/code&gt; or &lt;code&gt;pkg-static upgrade -f&lt;/code&gt; for cases where the upgrade instructions explicitly call for a forced reinstall due to &lt;span class="caps"&gt;ABI&lt;/span&gt; changes. The forced reinstall is primarily for major-version upgrades where the &lt;span class="caps"&gt;ABI&lt;/span&gt; has&amp;nbsp;shifted.&lt;/p&gt;
&lt;h3 id="step-6-final-cleanup"&gt;Step 6: Final&amp;nbsp;Cleanup&lt;/h3&gt;
&lt;p&gt;Run the&amp;nbsp;final &lt;code&gt;freebsd-update install&lt;/code&gt; to remove stale base system&amp;nbsp;files:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-update&lt;span class="w"&gt; &lt;/span&gt;install
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="step-7-check-for-failed-configuration-updates"&gt;Step 7: Check for Failed Configuration&amp;nbsp;Updates&lt;/h3&gt;
&lt;p&gt;After the base upgrade and package upgrades, check for configuration files that couldn&amp;#8217;t be merged&amp;nbsp;automatically:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;find&lt;span class="w"&gt; &lt;/span&gt;/etc&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;*.pkgnew&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-ls
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Merge these manually as needed - see the &lt;a href="#configuration-merging-the-pkgnew-dance"&gt;Configuration Merging&lt;/a&gt; section&amp;nbsp;below.&lt;/p&gt;
&lt;h3 id="step-8-reboot"&gt;Step 8:&amp;nbsp;Reboot&lt;/h3&gt;
&lt;p&gt;After the final cleanup and configuration checks, reboot once&amp;nbsp;more:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;shutdown&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;now
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="path-b-packaged-base-systems-pkg"&gt;Path B: Packaged-Base Systems&amp;nbsp;(pkg)&lt;/h2&gt;
&lt;p&gt;If &lt;code&gt;pkg which /usr/bin/uname&lt;/code&gt; returns a package name such&amp;nbsp;as &lt;code&gt;FreeBSD-runtime-15.0&lt;/code&gt;, your base system is managed via packages. The 15.1 upgrade uses pkg commands rather&amp;nbsp;than &lt;code&gt;freebsd-update&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Packaged base is supported as a tech preview starting with 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; and will eventually replace distribution sets, but in 15.x it should be treated as a newer workflow rather than the universally preferred&amp;nbsp;default.&lt;/p&gt;
&lt;h3 id="step-1-create-a-rollback-boot-environment"&gt;Step 1: Create a Rollback Boot&amp;nbsp;Environment&lt;/h3&gt;
&lt;p&gt;If booting from &lt;span class="caps"&gt;ZFS&lt;/span&gt;, create a boot environment you can roll back to if&amp;nbsp;needed:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bectl&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;pre-15.1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-r&lt;/code&gt; flag creates the boot environment from the currently running&amp;nbsp;system.&lt;/p&gt;
&lt;h3 id="step-2-upgrade-the-base-system-package"&gt;Step 2: Upgrade the Base System&amp;nbsp;Package&lt;/h3&gt;
&lt;p&gt;Update the base system to&amp;nbsp;15.1:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-base
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This makes sure the currently installed packaged-base system is up to date before the release&amp;nbsp;upgrade.&lt;/p&gt;
&lt;h3 id="step-3-apply-the-abi-and-version-settings"&gt;Step 3: Apply the &lt;span class="caps"&gt;ABI&lt;/span&gt; and Version&amp;nbsp;Settings&lt;/h3&gt;
&lt;p&gt;Set the &lt;span class="caps"&gt;ABI&lt;/span&gt; target for FreeBSD 15 and the specific &lt;span class="caps"&gt;OSVERSION&lt;/span&gt; for 15.1, then&amp;nbsp;upgrade:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;-oABI&lt;span class="o"&gt;=&lt;/span&gt;FreeBSD:15:&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-oOSVERSION&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1501000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-base
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-o&lt;/code&gt; flags override the pkg configuration for this command only:&amp;nbsp;- &lt;code&gt;ABI=FreeBSD:15:$(uname -p)&lt;/code&gt; targets the FreeBSD 15 &lt;span class="caps"&gt;ABI&lt;/span&gt; for your architecture&amp;nbsp;- &lt;code&gt;OSVERSION=1501000&lt;/code&gt; specifies the 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;&amp;nbsp;version&lt;/p&gt;
&lt;p&gt;After the upgrade, review any messages printed&amp;nbsp;by &lt;code&gt;pkg&lt;/code&gt;. Some packages may require additional setup steps, such as&amp;nbsp;running &lt;code&gt;service &amp;lt;name&amp;gt; setup&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Note the use&amp;nbsp;of &lt;code&gt;$(uname -p)&lt;/code&gt; rather than&amp;nbsp;hard-coding &lt;code&gt;amd64&lt;/code&gt; - this keeps the command portable across&amp;nbsp;architectures.&lt;/p&gt;
&lt;h3 id="step-4-update-kernel-modules"&gt;Step 4: Update Kernel&amp;nbsp;Modules&lt;/h3&gt;
&lt;p&gt;Update any out-of-tree kernel modules from&amp;nbsp;ports:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-ports-kmods
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This ensures that third-party kernel modules match the new&amp;nbsp;kernel.&lt;/p&gt;
&lt;h3 id="step-5-check-for-failed-configuration-updates"&gt;Step 5: Check for Failed Configuration&amp;nbsp;Updates&lt;/h3&gt;
&lt;p&gt;After the base upgrade and package upgrades, check for configuration files that couldn&amp;#8217;t be merged&amp;nbsp;automatically:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;find&lt;span class="w"&gt; &lt;/span&gt;/etc&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;*.pkgnew&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-ls
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Merge these manually as needed - see the &lt;a href="#configuration-merging-the-pkgnew-dance"&gt;Configuration Merging&lt;/a&gt; section&amp;nbsp;below.&lt;/p&gt;
&lt;h3 id="step-6-update-the-boot-loader"&gt;Step 6: Update the Boot&amp;nbsp;Loader&lt;/h3&gt;
&lt;p&gt;After the software upgrades, update your boot loader. See the &lt;a href="#boot-loader-update"&gt;Boot-Loader Update&lt;/a&gt; section below for the specific commands for your boot&amp;nbsp;method.&lt;/p&gt;
&lt;h3 id="step-7-reboot"&gt;Step 7:&amp;nbsp;Reboot&lt;/h3&gt;
&lt;p&gt;Reboot into the updated&amp;nbsp;system:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;shutdown&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;now
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After reboot, verify you&amp;#8217;re on&amp;nbsp;15.1:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-version&lt;span class="w"&gt; &lt;/span&gt;-u
&lt;span class="c1"&gt;# Should show: 15.1-RELEASE&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="advanced-staging-into-a-mounted-boot-environment"&gt;Advanced: Staging into a Mounted Boot&amp;nbsp;Environment&lt;/h2&gt;
&lt;p&gt;The packaged-base procedure above upgrades in-place. The &lt;a href="https://docs.freebsd.org/en/books/handbook/cutting-edge/#pkgbase"&gt;Handbook&amp;#8217;s pkgbase section&lt;/a&gt; describes an advanced variant: build the new system into a mounted boot environment, activate it, and boot into it. This is closer to a major-upgrade or experimental&amp;nbsp;pattern.&lt;/p&gt;
&lt;p&gt;The Handbook&amp;#8217;s example&amp;nbsp;uses &lt;code&gt;pkg-static&lt;/code&gt; for this mounted-environment&amp;nbsp;workflow:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a new boot environment&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.1-RELEASE

&lt;span class="c1"&gt;# Mount it&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/mnt/upgrade
bectl&lt;span class="w"&gt; &lt;/span&gt;mount&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.1-RELEASE&lt;span class="w"&gt; &lt;/span&gt;/mnt/upgrade

&lt;span class="c1"&gt;# Install the base system into the mounted environment (Handbook syntax)&lt;/span&gt;
env&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;ABI&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;FreeBSD:15:&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pkg-static&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;/mnt/upgrade&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;FreeBSD-base

&lt;span class="c1"&gt;# Activate temporarily (one boot only)&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;activate&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.1-RELEASE

&lt;span class="c1"&gt;# Reboot and verify&lt;/span&gt;
shutdown&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;now
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If the new environment boots successfully, make it&amp;nbsp;permanent:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bectl&lt;span class="w"&gt; &lt;/span&gt;activate&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.1-RELEASE
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If it doesn&amp;#8217;t, reboot again and select your previous environment from the loader&amp;nbsp;menu.&lt;/p&gt;
&lt;p&gt;This mounted-&lt;span class="caps"&gt;BE&lt;/span&gt; method is useful for testing and parallel-system management, but for a release-specific 15.1 workflow, prefer the official in-place packaged-base procedure above unless you have tested this variant for your&amp;nbsp;environment.&lt;/p&gt;
&lt;h2 id="boot-loader-update"&gt;Boot-Loader&amp;nbsp;Update&lt;/h2&gt;
&lt;p&gt;After either upgrade path, you may need to update your boot loader. The official 15.1 instructions include this step, and it&amp;#8217;s particularly important for &lt;span class="caps"&gt;ZFS&lt;/span&gt;-root&amp;nbsp;systems.&lt;/p&gt;
&lt;p&gt;For&amp;nbsp;the &lt;code&gt;freebsd-update&lt;/code&gt; path, perform this immediately after the&amp;nbsp;first &lt;code&gt;freebsd-update install&lt;/code&gt; and before the first reboot into 15.1. For the packaged-base path, perform it after the base and kernel-module package upgrades and before&amp;nbsp;rebooting.&lt;/p&gt;
&lt;p&gt;First, check your boot&amp;nbsp;method:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;machdep.bootmethod
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="uefi-systems"&gt;&lt;span class="caps"&gt;UEFI&lt;/span&gt;&amp;nbsp;Systems&lt;/h3&gt;
&lt;p&gt;If &lt;code&gt;machdep.bootmethod&lt;/code&gt; reports &lt;code&gt;UEFI&lt;/code&gt;, identify the active &lt;span class="caps"&gt;EFI&lt;/span&gt; System&amp;nbsp;Partition:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;efibootmgr&lt;span class="w"&gt; &lt;/span&gt;-v
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The active boot loader entry is preceded with&amp;nbsp;a &lt;code&gt;+&lt;/code&gt; character. In the official example, that line begins&amp;nbsp;with &lt;code&gt;+Boot0000*&lt;/code&gt;. Note the partition and loader path. If the &lt;span class="caps"&gt;ESP&lt;/span&gt; is not already mounted&amp;nbsp;at &lt;code&gt;/boot/efi&lt;/code&gt;, mount it, adjusting the device name to match your&amp;nbsp;system:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mount_msdosfs&lt;span class="w"&gt; &lt;/span&gt;/dev/nda0p1&lt;span class="w"&gt; &lt;/span&gt;/boot/efi
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On amd64 systems, update the configured and fallback&amp;nbsp;loaders:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cp&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.efi&lt;span class="w"&gt; &lt;/span&gt;/boot/efi/efi/freebsd/loader.efi
cp&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.efi&lt;span class="w"&gt; &lt;/span&gt;/boot/efi/efi/boot/bootx64.efi
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Some systems use different case, such&amp;nbsp;as &lt;code&gt;EFI/BOOT/BOOTX64.EFI&lt;/code&gt;, or may not&amp;nbsp;have &lt;code&gt;efi/freebsd/loader.efi&lt;/code&gt;. Match the paths reported&amp;nbsp;by &lt;code&gt;efibootmgr -v&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;On AArch64 systems,&amp;nbsp;use:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cp&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.efi&lt;span class="w"&gt; &lt;/span&gt;/boot/efi/efi/freebsd/loader.efi
cp&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.efi&lt;span class="w"&gt; &lt;/span&gt;/boot/efi/efi/boot/bootaa64.efi
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="bios-uefi-csm-systems"&gt;&lt;span class="caps"&gt;BIOS&lt;/span&gt; / &lt;span class="caps"&gt;UEFI&lt;/span&gt;-&lt;span class="caps"&gt;CSM&lt;/span&gt;&amp;nbsp;Systems&lt;/h3&gt;
&lt;p&gt;For &lt;span class="caps"&gt;BIOS&lt;/span&gt; or &lt;span class="caps"&gt;UEFI&lt;/span&gt;-&lt;span class="caps"&gt;CSM&lt;/span&gt; systems using &lt;span class="caps"&gt;GPT&lt;/span&gt;, first identify&amp;nbsp;the &lt;code&gt;freebsd-boot&lt;/code&gt; partition:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gpart&lt;span class="w"&gt; &lt;/span&gt;show
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note the disk name and partition index. In the example from the official documentation, the disk&amp;nbsp;is &lt;code&gt;ada0&lt;/code&gt; and&amp;nbsp;the &lt;code&gt;freebsd-boot&lt;/code&gt; partition index&amp;nbsp;is &lt;code&gt;1&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;For &lt;span class="caps"&gt;ZFS&lt;/span&gt;-on-&lt;span class="caps"&gt;BIOS&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gpart&lt;span class="w"&gt; &lt;/span&gt;bootcode&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;/boot/pmbr&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/boot/gptzfsboot&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ada0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For &lt;span class="caps"&gt;UFS&lt;/span&gt;-on-&lt;span class="caps"&gt;BIOS&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gpart&lt;span class="w"&gt; &lt;/span&gt;bootcode&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;/boot/pmbr&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/boot/gptboot&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ada0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Adjust &lt;code&gt;ada0&lt;/code&gt; and &lt;code&gt;-i 1&lt;/code&gt; for your system. If booting from a storage array, repeat the command for all disks in the&amp;nbsp;pool.&lt;/p&gt;
&lt;p&gt;Bootcode upgrades are optional for &lt;span class="caps"&gt;BIOS&lt;/span&gt;/&lt;span class="caps"&gt;UEFI&lt;/span&gt;-&lt;span class="caps"&gt;CSM&lt;/span&gt; unless a &lt;span class="caps"&gt;ZFS&lt;/span&gt; root pool is upgraded, which the official instructions discourage. Bootcode upgrades are only possible if&amp;nbsp;the &lt;code&gt;freebsd-boot&lt;/code&gt; partition is at least 180K; 512K is now&amp;nbsp;standard.&lt;/p&gt;
&lt;h2 id="configuration-merging-the-pkgnew-dance"&gt;Configuration Merging: The .pkgnew&amp;nbsp;Dance&lt;/h2&gt;
&lt;p&gt;After the base upgrade and package upgrades, you&amp;#8217;ll need to merge configuration files that couldn&amp;#8217;t be updated&amp;nbsp;automatically.&lt;/p&gt;
&lt;h3 id="finding-pkgnew-files"&gt;Finding .pkgnew&amp;nbsp;Files&lt;/h3&gt;
&lt;p&gt;After any upgrade, find what needs&amp;nbsp;attention:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;find&lt;span class="w"&gt; &lt;/span&gt;/etc&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;*.pkgnew&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-ls
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;/etc/*.pkgnew&lt;/code&gt; files come from the FreeBSD base&amp;nbsp;upgrade; &lt;code&gt;/usr/local/etc/*.pkgnew&lt;/code&gt; files come from third-party package&amp;nbsp;upgrades.&lt;/p&gt;
&lt;p&gt;For each file, you have three&amp;nbsp;options:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Option 1: Your changes are important; merge manually&lt;/span&gt;
diff&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config.pkgnew
&lt;span class="c1"&gt;# Edit /etc/ssh/sshd_config to incorporate new upstream changes&lt;/span&gt;
rm&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config.pkgnew

&lt;span class="c1"&gt;# Option 2: Upstream changes are irrelevant; keep yours&lt;/span&gt;
rm&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config.pkgnew

&lt;span class="c1"&gt;# Option 3: Upstream changes matter; replace yours&lt;/span&gt;
cp&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config.pkgnew&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config
rm&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config.pkgnew
&lt;span class="c1"&gt;# Re-apply local customizations&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A three-way merge tool helps see what&amp;nbsp;changed:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vimdiff&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config&lt;span class="w"&gt; &lt;/span&gt;/etc/ssh/sshd_config.pkgnew
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="etcupdate-for-advanced-merges"&gt;etcupdate for Advanced&amp;nbsp;Merges&lt;/h3&gt;
&lt;p&gt;For source-based upgrades or environments where you&amp;nbsp;manage &lt;code&gt;/etc&lt;/code&gt; separately, &lt;code&gt;etcupdate(8)&lt;/code&gt; can help with three-way merges. On modern&amp;nbsp;FreeBSD, &lt;code&gt;etcupdate&lt;/code&gt; is part of the base system and is designed for merging configuration files from source&amp;nbsp;upgrades.&lt;/p&gt;
&lt;p&gt;Note&amp;nbsp;that &lt;code&gt;etcupdate&lt;/code&gt; does not manage every file&amp;nbsp;in &lt;code&gt;/etc&lt;/code&gt;. Its man page states it does not manage files not maintained in the source tree, such&amp;nbsp;as &lt;code&gt;/etc/fstab&lt;/code&gt; and &lt;code&gt;/etc/rc.conf&lt;/code&gt;. Those files are yours to manage&amp;nbsp;entirely.&lt;/p&gt;
&lt;h3 id="common-pkgnew-files-to-watch-for"&gt;Common .pkgnew Files to Watch&amp;nbsp;For&lt;/h3&gt;
&lt;p&gt;After a release upgrade, pay particular attention&amp;nbsp;to:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;/etc/ssh/sshd_config.pkgnew&lt;/code&gt; - &lt;span class="caps"&gt;SSH&lt;/span&gt; deprecations and new security&amp;nbsp;defaults&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/etc/sysctl.conf.pkgnew&lt;/code&gt; - New sysctls worth&amp;nbsp;considering&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/etc/make.conf.pkgnew&lt;/code&gt; - Build system changes if you build&amp;nbsp;ports&lt;/li&gt;
&lt;li&gt;Files&amp;nbsp;under &lt;code&gt;/usr/local/etc/&lt;/code&gt; - Third-party daemon configs from package&amp;nbsp;upgrades&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="post-upgrade-verification"&gt;Post-Upgrade&amp;nbsp;Verification&lt;/h2&gt;
&lt;p&gt;After either method, run through this checklist before calling the upgrade&amp;nbsp;complete:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# 1. Confirm the release version&lt;/span&gt;
freebsd-version&lt;span class="w"&gt; &lt;/span&gt;-k
freebsd-version&lt;span class="w"&gt; &lt;/span&gt;-u

&lt;span class="c1"&gt;# 2. Check that all kernel modules match the kernel&lt;/span&gt;
kldstat

&lt;span class="c1"&gt;# 3. Verify services are running&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;-e

&lt;span class="c1"&gt;# 4. Check for remaining .pkgnew files&lt;/span&gt;
find&lt;span class="w"&gt; &lt;/span&gt;/etc&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc&lt;span class="w"&gt; &lt;/span&gt;-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;*.pkgnew&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-ls

&lt;span class="c1"&gt;# 5. Review dmesg for anything unexpected&lt;/span&gt;
dmesg&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tail&lt;span class="w"&gt; &lt;/span&gt;-50

&lt;span class="c1"&gt;# 6. Confirm your boot method and loader are current&lt;/span&gt;
sysctl&lt;span class="w"&gt; &lt;/span&gt;machdep.bootmethod
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you use monitoring, verify that metrics and logs are flowing correctly. Upgrades sometimes change log paths or metric&amp;nbsp;names.&lt;/p&gt;
&lt;h2 id="common-pitfalls-and-how-to-avoid-them"&gt;Common Pitfalls and How to Avoid&amp;nbsp;Them&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Not creating a boot environment before upgrading.&lt;/strong&gt; Both paths can be protected with a &lt;span class="caps"&gt;ZFS&lt;/span&gt; boot environment when the system boots from &lt;span class="caps"&gt;ZFS&lt;/span&gt;. With packaged-base&amp;nbsp;upgrades, &lt;code&gt;bectl create -r&lt;/code&gt; gives you a rollback point.&amp;nbsp;With &lt;code&gt;freebsd-update&lt;/code&gt;, remember&amp;nbsp;to &lt;code&gt;bectl create&lt;/code&gt; or &lt;code&gt;zfs snapshot&lt;/code&gt; &lt;em&gt;before&lt;/em&gt; you start. Once you&amp;#8217;ve replaced the kernel, there&amp;#8217;s no clean rollback without having planned&amp;nbsp;ahead.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Forgetting to apply pending patches first.&lt;/strong&gt; The&amp;nbsp;pre-upgrade &lt;code&gt;freebsd-update fetch &amp;amp;&amp;amp; freebsd-update install&lt;/code&gt; step is easy to skip, but it ensures you&amp;#8217;re starting from a clean, fully-patched 15.0 before jumping to&amp;nbsp;15.1.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Over-using &lt;code&gt;pkg upgrade -f&lt;/code&gt;.&lt;/strong&gt; Forced package reinstalls are for &lt;span class="caps"&gt;ABI&lt;/span&gt; changes, not routine point releases. Follow the prompts&amp;nbsp;from &lt;code&gt;freebsd-update&lt;/code&gt; and&amp;nbsp;reserve &lt;code&gt;-f&lt;/code&gt; for when the upgrade instructions explicitly require&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Ignoring &lt;code&gt;.pkgnew&lt;/code&gt; files.&lt;/strong&gt; It&amp;#8217;s tempting to assume &amp;#8220;if it&amp;#8217;s still working, I&amp;#8217;ll deal with it later.&amp;#8221; The errata for 15.1 includes multiple security advisories; new config files may contain relevant security fixes. Deal&amp;nbsp;with &lt;code&gt;.pkgnew&lt;/code&gt; files while the upgrade is&amp;nbsp;fresh.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Skipping or delaying the boot-loader update.&lt;/strong&gt; For &lt;span class="caps"&gt;UEFI&lt;/span&gt; systems in particular, the boot loader update is a separate step that&amp;#8217;s easy to miss. Update it before the first 15.1 boot attempt; a stale loader can cause boot failures even with a correct&amp;nbsp;kernel.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Not testing on a non-production system first.&lt;/strong&gt; If you have a test &lt;span class="caps"&gt;VM&lt;/span&gt; or a less critical server, upgrade that one first. You&amp;#8217;ll discover any surprises in a controlled&amp;nbsp;environment.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Upgrading on a Friday evening.&lt;/strong&gt; It&amp;#8217;s a cliché because it&amp;#8217;s true. Give yourself at least one full business day to deal with unexpected&amp;nbsp;issues.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Upgrading FreeBSD from 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; to 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; is straightforward when you follow the official path for your base system type. The key is knowing which path you&amp;#8217;re on&amp;nbsp;- &lt;code&gt;pkg which /usr/bin/uname&lt;/code&gt; tells you that - and then following the documented procedure for either distribution sets&amp;nbsp;(&lt;code&gt;freebsd-update&lt;/code&gt;) or packaged base&amp;nbsp;(&lt;code&gt;pkg&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;Both paths handle configuration merges&amp;nbsp;via &lt;code&gt;.pkgnew&lt;/code&gt; files, and both include a boot-loader update that shouldn&amp;#8217;t be skipped or delayed past the first 15.1 boot attempt. The habits that matter: create a rollback point before starting (if booting from &lt;span class="caps"&gt;ZFS&lt;/span&gt;), apply pending patches first, keep out-of-tree kernel modules in sync, deal&amp;nbsp;with &lt;code&gt;.pkgnew&lt;/code&gt; files promptly, and verify thoroughly before declaring&amp;nbsp;victory.&lt;/p&gt;
&lt;p&gt;With 15.0 reaching &lt;span class="caps"&gt;EOL&lt;/span&gt; on September 30, 2026, and 15.1 supported until March 31, 2027, now is the time to plan your upgrade. The next release in the 15.x series will be easier once you&amp;#8217;ve established your upgrade&amp;nbsp;pattern.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.freebsd.org/releases/15.1R/upgrading/"&gt;FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; Upgrading Instructions&lt;/a&gt; - Primary reference for this&amp;nbsp;article&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freebsd.org/releases/15.1R/announce/"&gt;FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; Announcement&lt;/a&gt; - Support dates and&amp;nbsp;overview&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freebsd.org/releases/15.1R/errata/"&gt;FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; Errata&lt;/a&gt; - Security advisories and late-breaking&amp;nbsp;notes&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freebsd.org/releases/15.1R/relnotes/"&gt;FreeBSD 15.1-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; Release Notes&lt;/a&gt; - Technical changes in&amp;nbsp;15.1&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/cutting-edge/#pkgbase"&gt;FreeBSD Handbook - Updating and Upgrading (PKGBase)&lt;/a&gt; - PKGBase context and advanced&amp;nbsp;workflows&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=freebsd-update&amp;amp;sektion=8"&gt;freebsd-update(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=bectl&amp;amp;sektion=8"&gt;bectl(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pkg&amp;amp;sektion=8"&gt;pkg(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=etcupdate&amp;amp;sektion=8"&gt;etcupdate(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="upgrade"/><category term="freebsd-update"/><category term="pkgbase"/><category term="bectl"/><category term="boot-environments"/><category term="zfs"/></entry><entry><title>FreeBSD Foundationals: The Boot Process - From the Loader to Boot Environments</title><link href="https://hofstede.it/freebsd-foundationals-the-boot-process-from-the-loader-to-boot-environments/" rel="alternate"/><published>2026-06-25T00:00:00+02:00</published><updated>2026-06-25T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-06-25:/freebsd-foundationals-the-boot-process-from-the-loader-to-boot-environments/</id><summary type="html">&lt;p&gt;The third in the FreeBSD Foundationals series. This one covers how FreeBSD actually boots: the stages from power-on to login, &lt;span class="caps"&gt;BIOS&lt;/span&gt; vs &lt;span class="caps"&gt;UEFI&lt;/span&gt;, the loader and loader.conf, the difference between read-only tunables and runtime sysctls, the modern way to load kernel modules with kld_list, managing it all safely with sysrc, a practical security-hardening baseline, and - the headline act - boot environments with bectl, the undo button that turns a broken upgrade into a thirty-second&amp;nbsp;reboot.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;The two previous articles in this series covered the things that store your data and the things that run your workloads. This one covers the part that nobody thinks about until it&amp;#8217;s three in the morning and the machine won&amp;#8217;t come back up: how FreeBSD actually boots, and - more importantly - how to make a bad boot&amp;nbsp;recoverable.&lt;/p&gt;
&lt;p&gt;This is the third article in the &lt;strong&gt;FreeBSD Foundationals&lt;/strong&gt; series. The &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;first covered Jails&lt;/a&gt;, the &lt;a href="https://hofstede.it/freebsd-foundationals-zfs-the-last-filesystem-youll-ever-need/"&gt;second covered &lt;span class="caps"&gt;ZFS&lt;/span&gt;&lt;/a&gt;. We&amp;#8217;re covering the boot process now because everything else depends on it and because, compared to other &lt;span class="caps"&gt;UNIX&lt;/span&gt;-like systems, FreeBSD keeps the early boot chain unusually small and inspectable. There&amp;#8217;s a small, documented chain of stages, a text file you edit by hand, and - if you&amp;#8217;re on root-on-&lt;span class="caps"&gt;ZFS&lt;/span&gt; - a recovery mechanism that is genuinely one of the best reasons to run FreeBSD on a&amp;nbsp;server.&lt;/p&gt;
&lt;p&gt;By the end of this article you&amp;#8217;ll understand what each boot stage does, where to put a setting and why, the difference between a tunable and a sysctl (people get this wrong constantly), the modern way to load kernel modules, and how boot environments turn &amp;#8220;I bricked it during an upgrade&amp;#8221; into a&amp;nbsp;non-event.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-boot-chain-power-on-to-login"&gt;The Boot Chain: Power-On to Login&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#where-does-zfs-fit-in"&gt;Where Does &lt;span class="caps"&gt;ZFS&lt;/span&gt; Fit&amp;nbsp;In?&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-loader-stage-three-and-the-last-friendly-face"&gt;The Loader: Stage Three and the Last Friendly Face&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#loaderconf-what-goes-here-and-why"&gt;loader.conf: What Goes Here and&amp;nbsp;Why&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#tunables-vs-sysctls-the-distinction-everyone-gets-wrong"&gt;Tunables vs Sysctls: The Distinction Everyone Gets Wrong&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#sysctlconf-where-runtime-knobs-live"&gt;sysctl.conf: Where Runtime Knobs&amp;nbsp;Live&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#loading-kernel-modules-the-modern-way-is-kld_list"&gt;Loading Kernel Modules: The Modern Way Is&amp;nbsp;kld_list&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#sysrc-stop-hand-editing-rcconf"&gt;sysrc: Stop Hand-Editing&amp;nbsp;rc.conf&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-practical-security-hardening-baseline"&gt;A Practical Security-Hardening Baseline&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#loaderconf-hardening"&gt;loader.conf&amp;nbsp;hardening&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#sysctlconf-hardening"&gt;sysctl.conf&amp;nbsp;hardening&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#dont-filter-bridge-traffic-twice"&gt;Don&amp;#8217;t filter bridge traffic&amp;nbsp;twice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#cpu-mitigations-and-entropy"&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt; mitigations and&amp;nbsp;entropy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-note-on-routers"&gt;A note on&amp;nbsp;routers&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#boot-environments-the-undo-button"&gt;Boot Environments: The Undo Button&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-workflow-that-saves-you"&gt;The Workflow That Saves&amp;nbsp;You&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#bectl-in-practice"&gt;bectl in&amp;nbsp;Practice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#boot-environments-and-upgrades"&gt;Boot Environments and&amp;nbsp;Upgrades&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-temporary-activation-trick"&gt;The Temporary-Activation&amp;nbsp;Trick&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#when-the-boot-goes-wrong-single-user-and-the-loader-prompt"&gt;When the Boot Goes Wrong: Single-User and the Loader&amp;nbsp;Prompt&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#common-pitfalls"&gt;Common&amp;nbsp;Pitfalls&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-boot-chain-power-on-to-login"&gt;The Boot Chain: Power-On to&amp;nbsp;Login&lt;/h2&gt;
&lt;p&gt;FreeBSD boots in stages. Each stage has exactly one job: find and start the next, slightly smarter, stage. The chain differs at the very bottom depending on whether your firmware is legacy &lt;span class="caps"&gt;BIOS&lt;/span&gt; or &lt;span class="caps"&gt;UEFI&lt;/span&gt;, but it converges quickly on the same component - the &lt;strong&gt;loader&lt;/strong&gt; - and from there the path is&amp;nbsp;identical.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Legacy BIOS (MBR/GPT)              UEFI

┌──────────────────┐              ┌──────────────────┐
│   BIOS firmware  │              │  UEFI firmware   │
└────────┬─────────┘              └────────┬─────────┘
         │                                 │ reads the ESP
   boot0 / boot1                           │ (FAT partition)
   boot2 / gptzfsboot                      │
         │                                 │
         └──────────────┬──────────────────┘
                        │ loads
                ┌───────┴────────┐
                │  /boot/loader  │   ← stage 3: the loader
                │ (loader.efi on │     reads loader.conf,
                │     UEFI)      │     loads kernel + modules
                └───────┬────────┘
                        │
                ┌───────┴────────┐
                │  kernel        │   ← mounts root, starts init
                └───────┬────────┘
                        │
                ┌───────┴────────┐
                │  init(8)       │   ← runs /etc/rc
                └───────┬────────┘
                        │
                ┌───────┴────────┐
                │  rc(8) scripts │   ← rc.conf, services, jails
                └───────┬────────┘
                        │
                     login
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On &lt;strong&gt;legacy &lt;span class="caps"&gt;BIOS&lt;/span&gt;&lt;/strong&gt;, the firmware reads the first sector of the boot disk (the &lt;span class="caps"&gt;MBR&lt;/span&gt;,&amp;nbsp;containing &lt;code&gt;boot0&lt;/code&gt;), which chains to a&amp;nbsp;partition&amp;#8217;s &lt;code&gt;boot1&lt;/code&gt;/&lt;code&gt;boot2&lt;/code&gt; (or &lt;code&gt;gptboot&lt;/code&gt; on &lt;span class="caps"&gt;GPT&lt;/span&gt;,&amp;nbsp;or &lt;code&gt;gptzfsboot&lt;/code&gt; when root is on &lt;span class="caps"&gt;ZFS&lt;/span&gt;). These tiny programs exist only to understand just enough of the filesystem to find and load the loader. They are measured in hundreds of bytes for a reason: there isn&amp;#8217;t much room in a boot&amp;nbsp;sector.&lt;/p&gt;
&lt;p&gt;On &lt;strong&gt;&lt;span class="caps"&gt;UEFI&lt;/span&gt;&lt;/strong&gt;, all of that disappears. The firmware itself understands the &lt;span class="caps"&gt;FAT&lt;/span&gt;-formatted &lt;strong&gt;&lt;span class="caps"&gt;EFI&lt;/span&gt; System Partition (&lt;span class="caps"&gt;ESP&lt;/span&gt;)&lt;/strong&gt; and loads an &lt;span class="caps"&gt;EFI&lt;/span&gt; executable directly - for FreeBSD&amp;nbsp;that&amp;#8217;s &lt;code&gt;loader.efi&lt;/code&gt;, installed&amp;nbsp;at &lt;code&gt;/EFI/freebsd/loader.efi&lt;/code&gt; and usually also as the&amp;nbsp;fallback &lt;code&gt;/EFI/BOOT/BOOTX64.EFI&lt;/code&gt;. No boot0, no boot2, no chain of sectors. &lt;span class="caps"&gt;UEFI&lt;/span&gt; is generally the default for modern installs and simplifies the boot chain on most&amp;nbsp;hardware.&lt;/p&gt;
&lt;p&gt;Either way, you land at the &lt;strong&gt;loader&lt;/strong&gt;, and that&amp;#8217;s where things get&amp;nbsp;interesting.&lt;/p&gt;
&lt;h3 id="where-does-zfs-fit-in"&gt;Where Does &lt;span class="caps"&gt;ZFS&lt;/span&gt; Fit&amp;nbsp;In?&lt;/h3&gt;
&lt;p&gt;If your root filesystem is on &lt;span class="caps"&gt;ZFS&lt;/span&gt; - which, after the &lt;a href="https://hofstede.it/freebsd-foundationals-zfs-the-last-filesystem-youll-ever-need/"&gt;previous article&lt;/a&gt;, it should be - the early boot code needs to understand &lt;span class="caps"&gt;ZFS&lt;/span&gt; well enough to find the &lt;strong&gt;bootfs&lt;/strong&gt;: the dataset marked as the one to boot from. On &lt;span class="caps"&gt;BIOS&lt;/span&gt; this is the job&amp;nbsp;of &lt;code&gt;gptzfsboot&lt;/code&gt;; on &lt;span class="caps"&gt;UEFI&lt;/span&gt;, &lt;code&gt;loader.efi&lt;/code&gt; has &lt;span class="caps"&gt;ZFS&lt;/span&gt; support built in. The loader reads the pool, finds the active boot environment, loads the kernel from it, and hands off. Hold onto the phrase &amp;#8220;active boot environment&amp;#8221; - it&amp;#8217;s the whole reason this article&amp;nbsp;exists.&lt;/p&gt;
&lt;h2 id="the-loader-stage-three-and-the-last-friendly-face"&gt;The Loader: Stage Three and the Last Friendly&amp;nbsp;Face&lt;/h2&gt;
&lt;p&gt;The loader&amp;nbsp;(&lt;code&gt;/boot/loader&lt;/code&gt;,&amp;nbsp;or &lt;code&gt;loader.efi&lt;/code&gt; under &lt;span class="caps"&gt;UEFI&lt;/span&gt;) is the last component before the kernel takes over, and it&amp;#8217;s the most capable. It&amp;#8217;s a small interactive environment - scripted primarily in &lt;strong&gt;Forth&lt;/strong&gt;, with optional &lt;strong&gt;Lua&lt;/strong&gt; support available in modern FreeBSD builds -&amp;nbsp;that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Shows you the &lt;strong&gt;boot menu&lt;/strong&gt; (the orb menu with the&amp;nbsp;options)&lt;/li&gt;
&lt;li&gt;Reads &lt;code&gt;/boot/loader.conf&lt;/code&gt; for&amp;nbsp;configuration&lt;/li&gt;
&lt;li&gt;Loads the kernel and any kernel modules you&amp;#8217;ve requested (early KLDs only - once the kernel starts, module management shifts to the rc system&amp;nbsp;and &lt;code&gt;kld(4)&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Sets &lt;strong&gt;boot-time tunables&lt;/strong&gt; (more on these&amp;nbsp;shortly)&lt;/li&gt;
&lt;li&gt;Lets you drop to an interactive prompt to fix a system that won&amp;#8217;t&amp;nbsp;boot&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That last point is worth dwelling on. When the boot menu appears, pressing the right key (the menu lists them) lets you boot single-user, select a different kernel, choose a &lt;strong&gt;boot environment&lt;/strong&gt;, or escape to the loader prompt entirely. At the prompt you can inspect and change anything before the kernel&amp;nbsp;starts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;OK show                  # show all loader variables
OK lsdev                 # list devices the loader can see
OK set vfs.root.mountfrom=zfs:zroot/ROOT/default
OK unload                # unload the default kernel
OK load /boot/kernel.old/kernel   # load a known-good kernel instead
OK boot -s               # boot into single-user mode
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is the FreeBSD equivalent of a &lt;span class="caps"&gt;GRUB&lt;/span&gt; rescue prompt, except documented, consistent, and pleasant to&amp;nbsp;use.&lt;/p&gt;
&lt;h3 id="loaderconf-what-goes-here-and-why"&gt;loader.conf: What Goes Here and&amp;nbsp;Why&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;/boot/loader.conf&lt;/code&gt; is the loader&amp;#8217;s configuration file. The defaults live&amp;nbsp;in &lt;code&gt;/boot/defaults/loader.conf&lt;/code&gt; (read it once to see what&amp;#8217;s available - don&amp;#8217;t edit it); your overrides go&amp;nbsp;in &lt;code&gt;/boot/loader.conf&lt;/code&gt;. There&amp;#8217;s&amp;nbsp;also &lt;code&gt;/boot/loader.conf.local&lt;/code&gt; for layered configs (handy if a config-management tool&amp;nbsp;owns &lt;code&gt;loader.conf&lt;/code&gt; and you want machine-specific&amp;nbsp;additions).&lt;/p&gt;
&lt;p&gt;Two distinct kinds of things go&amp;nbsp;in &lt;code&gt;loader.conf&lt;/code&gt;:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Module load directives&lt;/strong&gt;&amp;nbsp;- &lt;code&gt;something_load="YES"&lt;/code&gt;, which tells the loader to&amp;nbsp;load &lt;code&gt;something.ko&lt;/code&gt; before the kernel&amp;nbsp;starts.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Boot-time tunables&lt;/strong&gt; - kernel variables that can &lt;em&gt;only&lt;/em&gt; be set before the kernel boots and are read-only&amp;nbsp;afterward.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Here&amp;#8217;s a&amp;nbsp;real &lt;code&gt;loader.conf&lt;/code&gt; from one of my jail hosts,&amp;nbsp;annotated:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Boot-time tunables: read-only once the kernel is running.&lt;/span&gt;
&lt;span class="c1"&gt;# These MUST live in loader.conf - you cannot set them later.&lt;/span&gt;
&lt;span class="na"&gt;kern.geom.label.disk_ident.enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# don&amp;#39;t clutter /dev with diskid/ labels&lt;/span&gt;
&lt;span class="na"&gt;kern.geom.label.gptid.enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# prefer stable gpt/ labels over gptids&lt;/span&gt;
&lt;span class="na"&gt;net.fibs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;                            &lt;/span&gt;&lt;span class="c1"&gt;# enable a second routing table (FIB)&lt;/span&gt;
&lt;span class="na"&gt;kern.racct.enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;1&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;                   &lt;/span&gt;&lt;span class="c1"&gt;# turn on resource accounting (see RCTL article)&lt;/span&gt;

&lt;span class="c1"&gt;# Modules loaded early by the loader&lt;/span&gt;
&lt;span class="na"&gt;cryptodev_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;zfs_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;                          &lt;/span&gt;&lt;span class="c1"&gt;# required: root is on ZFS&lt;/span&gt;
&lt;span class="na"&gt;pf_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;virtio_random_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="c1"&gt;# early entropy on a VM&lt;/span&gt;

&lt;span class="c1"&gt;# Filesystem modules&lt;/span&gt;
&lt;span class="na"&gt;nullfs_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;fdescfs_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Network modules for VNET jails&lt;/span&gt;
&lt;span class="na"&gt;if_epair_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;if_bridge_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;if_tuntap_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every line here works. But - and this is one of the main points of the article - &lt;strong&gt;a good portion of it can often be moved elsewhere, depending on intent.&lt;/strong&gt; We&amp;#8217;ll come back to that in the modules&amp;nbsp;section.&lt;/p&gt;
&lt;p&gt;A few&amp;nbsp;more &lt;code&gt;loader.conf&lt;/code&gt; knobs worth knowing, taken from a bhyve host where boot speed and virtualization&amp;nbsp;matter:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Boot menu / speed&lt;/span&gt;
&lt;span class="na"&gt;autoboot_delay&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="c1"&gt;# seconds to wait at the menu (default 10; &amp;quot;-1&amp;quot; waits forever, &amp;quot;NO&amp;quot; skips)&lt;/span&gt;
&lt;span class="na"&gt;hw.usb.no_boot_wait&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;1&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="c1"&gt;# don&amp;#39;t stall boot probing USB devices&lt;/span&gt;

&lt;span class="c1"&gt;# bhyve virtualization&lt;/span&gt;
&lt;span class="na"&gt;vmm_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="c1"&gt;# the bhyve hypervisor module&lt;/span&gt;
&lt;span class="na"&gt;nmdm_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="c1"&gt;# null-modem devices for VM serial consoles&lt;/span&gt;

&lt;span class="c1"&gt;# Asynchronous I/O - recommended for ZFS and database workloads&lt;/span&gt;
&lt;span class="na"&gt;aio_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Load a TCP congestion-control algorithm so it can be selected at runtime&lt;/span&gt;
&lt;span class="na"&gt;cc_bbr_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That last line is a perfect illustration of the loader/sysctl split, so let&amp;#8217;s make it explicit.&amp;nbsp;Loading &lt;code&gt;cc_bbr&lt;/code&gt; makes the &lt;span class="caps"&gt;BBR&lt;/span&gt; congestion-control algorithm &lt;em&gt;available&lt;/em&gt;. It does not make anything &lt;em&gt;use&lt;/em&gt; it. Selecting it is a runtime&amp;nbsp;sysctl:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# In /etc/sysctl.conf - selects the algorithm the module made available&lt;/span&gt;
&lt;span class="na"&gt;net.inet.tcp.cc.algorithm&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;bbr&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The module has to be loaded (loader.conf&amp;nbsp;or &lt;code&gt;kld_list&lt;/code&gt;), and then the choice is a sysctl. Get this relationship clear in your head and a lot of FreeBSD tuning stops being&amp;nbsp;mysterious.&lt;/p&gt;
&lt;p&gt;Before we go deeper, here&amp;#8217;s the whole configuration model on one screen - keep this map in mind as the rest of the article fills in each&amp;nbsp;piece:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;loader.conf  →  early boot constraints (boot-time tunables + root-critical modules)
sysctl.conf  →  runtime kernel behavior
rc.conf      →  system services (and kld_list)
kld_list     →  late module availability (after root is mounted)
bectl        →  rollback boundary (undo a bad change)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="tunables-vs-sysctls-the-distinction-everyone-gets-wrong"&gt;Tunables vs Sysctls: The Distinction Everyone Gets&amp;nbsp;Wrong&lt;/h2&gt;
&lt;p&gt;This trips up nearly everyone coming from Linux,&amp;nbsp;where &lt;code&gt;sysctl.conf&lt;/code&gt; is the one place you put kernel knobs. FreeBSD splits kernel parameters into two categories, and putting one in the wrong file means it either silently does nothing or refuses to take&amp;nbsp;effect.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Tunable&lt;/strong&gt;&lt;/th&gt;
&lt;th&gt;&lt;strong&gt;Sysctl (runtime)&lt;/strong&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Set where&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/boot/loader.conf&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/etc/sysctl.conf&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set when&lt;/td&gt;
&lt;td&gt;Before the kernel boots&lt;/td&gt;
&lt;td&gt;Any time, including at boot&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Changeable at runtime&lt;/td&gt;
&lt;td&gt;Typically not (often read-only once set)&lt;/td&gt;
&lt;td&gt;Yes&amp;nbsp;- &lt;code&gt;sysctl name=value&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Examples&lt;/td&gt;
&lt;td&gt;&lt;code&gt;net.fibs&lt;/code&gt;, &lt;code&gt;kern.racct.enable&lt;/code&gt;, &lt;code&gt;kern.geom.label.*&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;net.inet.tcp.blackhole&lt;/code&gt;, &lt;code&gt;security.bsd.see_other_uids&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The rule of thumb: some kernel values are baked in at boot because changing them later would be unsafe or meaningless (the number of routing tables, whether resource accounting is compiled into the running kernel&amp;#8217;s behavior, geometry labelling). Those are &lt;strong&gt;tunables&lt;/strong&gt; and&amp;nbsp;only &lt;code&gt;loader.conf&lt;/code&gt; can set them. Everything that&amp;#8217;s safe to change on a running system is a &lt;strong&gt;sysctl&lt;/strong&gt; and belongs&amp;nbsp;in &lt;code&gt;sysctl.conf&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;There&amp;#8217;s a dead-simple diagnostic to tell which is which. Try to set it at&amp;nbsp;runtime:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# If this works, it&amp;#39;s a runtime sysctl -&amp;gt; put it in /etc/sysctl.conf&lt;/span&gt;
sysctl&lt;span class="w"&gt; &lt;/span&gt;net.inet.tcp.blackhole&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;

&lt;span class="c1"&gt;# If you get &amp;quot;oid is read only&amp;quot;, it&amp;#39;s a boot-time tunable -&amp;gt; put it in /boot/loader.conf&lt;/span&gt;
sysctl&lt;span class="w"&gt; &lt;/span&gt;net.fibs&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="c1"&gt;# sysctl: oid &amp;#39;net.fibs&amp;#39; is read only&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You can also ask sysctl to describe any value, which is great when you&amp;#8217;re not sure what a knob&amp;nbsp;does:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;security.bsd.see_other_uids
&lt;span class="c1"&gt;# security.bsd.see_other_uids: Unprivileged processes may see subjects/objects with different real uid&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="sysctlconf-where-runtime-knobs-live"&gt;sysctl.conf: Where Runtime Knobs&amp;nbsp;Live&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;/etc/sysctl.conf&lt;/code&gt; is read late in boot and piped&amp;nbsp;through &lt;code&gt;sysctl&lt;/code&gt; to apply runtime values. The format is&amp;nbsp;bare &lt;code&gt;name=value&lt;/code&gt;, one per line (no quoting, unlike loader.conf). To apply changes without&amp;nbsp;rebooting:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Re-apply the whole file (idempotent for writable values)&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;restart
&lt;span class="c1"&gt;# ...which is just a wrapper around:&lt;/span&gt;
sysctl&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/sysctl.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If a value is read-only and you accidentally put it here, you&amp;#8217;ll see the &amp;#8220;oid is read only&amp;#8221; error at boot and in&amp;nbsp;the &lt;code&gt;service sysctl restart&lt;/code&gt; output - a useful signal that it belongs&amp;nbsp;in &lt;code&gt;loader.conf&lt;/code&gt; instead.&lt;/p&gt;
&lt;h2 id="loading-kernel-modules-the-modern-way-is-kld_list"&gt;Loading Kernel Modules: The Modern Way Is&amp;nbsp;kld_list&lt;/h2&gt;
&lt;p&gt;Now back to that claim about moving modules out&amp;nbsp;of &lt;code&gt;loader.conf&lt;/code&gt;. There are two ways to load a kernel module at boot, and the right choice depends on &lt;em&gt;when the module is actually needed&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;module_load="YES"&lt;/code&gt; in &lt;code&gt;loader.conf&lt;/code&gt;&lt;/strong&gt; loads the module in stage three, before the kernel starts. Use this only for modules that must be present that&amp;nbsp;early:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The filesystem driver for your root&amp;nbsp;(&lt;code&gt;zfs&lt;/code&gt; on root-on-&lt;span class="caps"&gt;ZFS&lt;/span&gt;).&lt;/li&gt;
&lt;li&gt;Disk encryption that wraps the root&amp;nbsp;(&lt;code&gt;geom_eli&lt;/code&gt; / &lt;code&gt;aesni&lt;/code&gt; for a &lt;span class="caps"&gt;GELI&lt;/span&gt;-encrypted&amp;nbsp;root).&lt;/li&gt;
&lt;li&gt;Anything the loader or early kernel genuinely needs before root is&amp;nbsp;mounted.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;kld_list&lt;/code&gt; in &lt;code&gt;/etc/rc.conf&lt;/code&gt;&lt;/strong&gt; loads modules during the rc startup sequence, right after the disks are mounted but before services start. This is the &lt;strong&gt;modern, preferred&lt;/strong&gt; way for everything that doesn&amp;#8217;t need to exist at loader time. It&amp;nbsp;keeps &lt;code&gt;loader.conf&lt;/code&gt; minimal, keeps the loader fast, and loads modules with the fully running kernel rather than the constrained loader&amp;nbsp;environment.&lt;/p&gt;
&lt;p&gt;The test is simple: &lt;strong&gt;does the module need to be loaded before root is mounted?&lt;/strong&gt; If no, it belongs&amp;nbsp;in &lt;code&gt;kld_list&lt;/code&gt;. Almost all network-interface&amp;nbsp;modules, &lt;code&gt;nullfs&lt;/code&gt;, &lt;code&gt;fdescfs&lt;/code&gt;, &lt;code&gt;pf&lt;/code&gt;, &lt;code&gt;vmm&lt;/code&gt;, and similar fall into&amp;nbsp;&amp;#8220;no.&amp;#8221;&lt;/p&gt;
&lt;p&gt;Look again at the jail&amp;nbsp;host&amp;#8217;s &lt;code&gt;loader.conf&lt;/code&gt; from earlier.&amp;nbsp;The &lt;code&gt;if_epair&lt;/code&gt;, &lt;code&gt;if_bridge&lt;/code&gt;, &lt;code&gt;if_tuntap&lt;/code&gt;, &lt;code&gt;nullfs&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;fdescfs&lt;/code&gt; modules are only needed by the time jails start - long after root is mounted. They have no business slowing down the loader. Here&amp;#8217;s the same machine,&amp;nbsp;modernized:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /boot/loader.conf - only what the loader genuinely needs early&lt;/span&gt;
&lt;span class="c1"&gt;# Boot-time tunables (read-only after boot)&lt;/span&gt;
&lt;span class="na"&gt;kern.geom.label.disk_ident.enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;kern.geom.label.gptid.enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;net.fibs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;kern.racct.enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Modules required before/at root mount&lt;/span&gt;
&lt;span class="na"&gt;zfs_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;cryptodev_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;virtio_random_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# early entropy on a VM&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/rc.conf - modules that only need to exist by the time services start&lt;/span&gt;
&lt;span class="nv"&gt;kld_list&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;nullfs fdescfs if_epair if_bridge if_tuntap&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note&amp;nbsp;that &lt;code&gt;pf&lt;/code&gt; dropped off the list entirely. You don&amp;#8217;t&amp;nbsp;need &lt;code&gt;pf_load&lt;/code&gt; &lt;em&gt;or&lt;/em&gt; &lt;code&gt;pf&lt;/code&gt; in &lt;code&gt;kld_list&lt;/code&gt;:&amp;nbsp;the &lt;code&gt;pf&lt;/code&gt; rc script (enabled&amp;nbsp;with &lt;code&gt;pf_enable="YES"&lt;/code&gt;) initializes the firewall and, in most standard configurations, loads the module if it isn&amp;#8217;t already built in. (See the &lt;a href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; firewall guide&lt;/a&gt; for the full firewall story.) Fewer redundant knobs, same&amp;nbsp;result.&lt;/p&gt;
&lt;p&gt;The payoff: a&amp;nbsp;lean &lt;code&gt;loader.conf&lt;/code&gt; that boots fast and only contains things that truly must be early, and&amp;nbsp;a &lt;code&gt;kld_list&lt;/code&gt; that&amp;#8217;s easy to read and lives alongside the rest of your service configuration&amp;nbsp;in &lt;code&gt;rc.conf&lt;/code&gt;. When you add a new &lt;span class="caps"&gt;VNET&lt;/span&gt; jail and realize you need another interface module, you add it&amp;nbsp;to &lt;code&gt;kld_list&lt;/code&gt; - not to the&amp;nbsp;loader.&lt;/p&gt;
&lt;p&gt;One naming&amp;nbsp;note: &lt;code&gt;kld_list&lt;/code&gt; uses module base names only&amp;nbsp;(e.g. &lt;code&gt;if_epair&lt;/code&gt;,&amp;nbsp;not &lt;code&gt;if_epair.ko&lt;/code&gt; or the&amp;nbsp;loader.conf-style &lt;code&gt;if_epair_load&lt;/code&gt; variables).&lt;/p&gt;
&lt;h2 id="sysrc-stop-hand-editing-rcconf"&gt;sysrc: Stop Hand-Editing&amp;nbsp;rc.conf&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;/etc/rc.conf&lt;/code&gt; controls what the system &lt;em&gt;does&lt;/em&gt; after it&amp;#8217;s booted - which services run, the hostname, networking, jails, and (as we just&amp;nbsp;saw) &lt;code&gt;kld_list&lt;/code&gt;. You &lt;em&gt;can&lt;/em&gt; edit it&amp;nbsp;with &lt;code&gt;vi&lt;/code&gt;. You shouldn&amp;#8217;t, most of the time,&amp;nbsp;because &lt;code&gt;sysrc(8)&lt;/code&gt; does it safely, idempotently, and&amp;nbsp;scriptably.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Read a value&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;sshd_enable
&lt;span class="c1"&gt;# sshd_enable: YES&lt;/span&gt;

&lt;span class="c1"&gt;# Set a value (adds it if missing, replaces it if present)&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sshd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES

&lt;span class="c1"&gt;# Append to a list-valued variable without clobbering the rest&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;kld_list&lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;if_lagg&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Remove a single element from a list&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;kld_list-&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;if_tuntap&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Delete a variable entirely&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;-x&lt;span class="w"&gt; &lt;/span&gt;moused_enable

&lt;span class="c1"&gt;# Show everything that&amp;#39;s been set (excluding defaults)&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;-a
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;+=&lt;/code&gt; and &lt;code&gt;-=&lt;/code&gt; operators are the reason to&amp;nbsp;use &lt;code&gt;sysrc&lt;/code&gt; even if you&amp;#8217;re comfortable in an editor. Appending one module&amp;nbsp;to &lt;code&gt;kld_list&lt;/code&gt; without having to retype the whole quoted string - and without risking a typo that breaks every module on the line - is exactly the kind of small safety that adds up across dozens of&amp;nbsp;machines.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;sysrc&lt;/code&gt; isn&amp;#8217;t limited&amp;nbsp;to &lt;code&gt;rc.conf&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;-f&lt;/code&gt; flag points it at any rc-style file,&amp;nbsp;including &lt;code&gt;loader.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Edit loader.conf safely from a script&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.conf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;zfs_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES

&lt;span class="c1"&gt;# Read a loader tunable&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.conf&lt;span class="w"&gt; &lt;/span&gt;net.fibs
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This&amp;nbsp;makes &lt;code&gt;sysrc&lt;/code&gt; a natural fit for automation. In the &lt;a href="https://hofstede.it/automating-freebsd-jails-with-cdist-zero-dependencies-inside-the-jail/"&gt;cdist&lt;/a&gt; and &lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;Ansible&lt;/a&gt; world,&amp;nbsp;idempotent &lt;code&gt;sysrc&lt;/code&gt; calls are far cleaner than templating the whole file and far safer than line-based edits. It either ends in the desired state or tells you why it&amp;nbsp;couldn&amp;#8217;t.&lt;/p&gt;
&lt;h2 id="a-practical-security-hardening-baseline"&gt;A Practical Security-Hardening&amp;nbsp;Baseline&lt;/h2&gt;
&lt;p&gt;Boot configuration is where a lot of host hardening actually lives, because the strongest knobs are kernel-level and set exactly once at boot. Here&amp;#8217;s a baseline I apply to internet-facing FreeBSD hosts, split correctly&amp;nbsp;between &lt;code&gt;loader.conf&lt;/code&gt; (tunables)&amp;nbsp;and &lt;code&gt;sysctl.conf&lt;/code&gt; (runtime). Treat it as a starting point, not gospel - understand each line before you ship&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="loaderconf-hardening"&gt;loader.conf&amp;nbsp;hardening&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Refuse the dangerous, system-modifying DTrace actions outright&lt;/span&gt;
&lt;span class="na"&gt;security.bsd.allow_destructive_dtrace&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;allow_destructive_dtrace&lt;/code&gt; is a boot-time tunable: DTrace&amp;#8217;s destructive actions (which can stop processes, modify memory, raise signals) are something you decide about before the kernel runs, not after. On a production box that isn&amp;#8217;t actively being debugged with destructive probes, set it&amp;nbsp;to &lt;code&gt;0&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="sysctlconf-hardening"&gt;sysctl.conf&amp;nbsp;hardening&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# --- Process and information isolation ---&lt;/span&gt;
security.bsd.see_other_uids&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="c1"&gt;# users can&amp;#39;t see other users&amp;#39; processes&lt;/span&gt;
security.bsd.see_other_gids&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
security.bsd.see_jail_proc&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# host root can&amp;#39;t see into jail process tables&lt;/span&gt;
security.bsd.unprivileged_read_msgbuf&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# hide the kernel message buffer (dmesg)&lt;/span&gt;
security.bsd.unprivileged_proc_debug&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# forbid ptrace/debugging for unprivileged users&lt;/span&gt;
kern.randompid&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;                       &lt;/span&gt;&lt;span class="c1"&gt;# randomize PIDs&lt;/span&gt;

&lt;span class="c1"&gt;# --- Hardlink restrictions (prevent some symlink/hardlink attacks) ---&lt;/span&gt;
security.bsd.hardlink_check_uid&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
security.bsd.hardlink_check_gid&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
security.bsd.unprivileged_mlock&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;

&lt;span class="c1"&gt;# --- Address-space layout randomization ---&lt;/span&gt;
kern.elf64.aslr.enable&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
kern.elf32.aslr.enable&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;

&lt;span class="c1"&gt;# --- Network stealth: drop, don&amp;#39;t reject, on closed ports ---&lt;/span&gt;
net.inet.tcp.blackhole&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="c1"&gt;# closed TCP ports send no RST&lt;/span&gt;
net.inet.udp.blackhole&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="c1"&gt;# closed UDP ports send no ICMP unreachable&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few of these deserve a&amp;nbsp;sentence:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;see_jail_proc=0&lt;/code&gt;&lt;/strong&gt; is part of the jail isolation story from the &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;first article&lt;/a&gt;: it stops even host root from casually listing processes inside jails, narrowing the blast radius of a compromised host&amp;nbsp;account.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;blackhole&lt;/code&gt;&lt;/strong&gt; reduces the active feedback a host gives during a port scan - a probe against a closed port gets silence instead of a helpful &lt;span class="caps"&gt;RST&lt;/span&gt; or &lt;span class="caps"&gt;ICMP&lt;/span&gt; message. It slows down reconnaissance, though packet timing and other behavior can still reveal that the host is alive. It does not replace a firewall; it complements&amp;nbsp;one.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;ASLR&lt;/span&gt;&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;kern.elf*.aslr.enable=1&lt;/code&gt;) randomizes the memory layout of executables, raising the bar for memory-corruption exploits. It&amp;#8217;s generally enabled by default on modern FreeBSD releases, but worth verifying&amp;nbsp;explicitly.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="dont-filter-bridge-traffic-twice"&gt;Don&amp;#8217;t filter bridge traffic&amp;nbsp;twice&lt;/h3&gt;
&lt;p&gt;If you run &lt;span class="caps"&gt;VNET&lt;/span&gt; jails on a bridge, there&amp;#8217;s a subtle pair of sysctls worth setting deliberately rather than inheriting by&amp;nbsp;accident:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Keep packets crossing the bridge from being filtered on the host&amp;#39;s pf,&lt;/span&gt;
&lt;span class="c1"&gt;# unless you specifically intend to run pf on the bridge itself.&lt;/span&gt;
net.link.bridge.pfil_bridge&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
net.link.bridge.pfil_member&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;These&amp;nbsp;are &lt;code&gt;0&lt;/code&gt; by default, but stating them&amp;nbsp;in &lt;code&gt;sysctl.conf&lt;/code&gt; documents the intent: filtering happens inside each jail&amp;#8217;s own pf, not redundantly on the host bridge. This connects directly to the bridge-and-epair model from the Jails&amp;nbsp;article.&lt;/p&gt;
&lt;h3 id="cpu-mitigations-and-entropy"&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt; mitigations and&amp;nbsp;entropy&lt;/h3&gt;
&lt;p&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt; side-channel mitigations (&lt;span class="caps"&gt;MDS&lt;/span&gt;, etc.) are largely automatic on current FreeBSD, but you can inspect and pin them. On older Intel hardware where you want to be&amp;nbsp;explicit:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Automatic MDS mitigation selection (0=off, 3=auto)&lt;/span&gt;
hw.mds_disable&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Check what the running kernel actually decided with&amp;nbsp;the &lt;code&gt;machdep.mitigations&lt;/code&gt; sysctl&amp;nbsp;tree:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;machdep.mitigations
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="a-note-on-routers"&gt;A note on&amp;nbsp;routers&lt;/h3&gt;
&lt;p&gt;The hardening above is for general hosts. A box doing a specific job loads specific modules. My &lt;span class="caps"&gt;BGP&lt;/span&gt; edge routers, for instance, carry&amp;nbsp;a &lt;code&gt;loader.conf&lt;/code&gt; that&amp;#8217;s all about crypto and &lt;span class="caps"&gt;TCP&lt;/span&gt;-&lt;span class="caps"&gt;MD5&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;aesni_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="c1"&gt;# AES-NI hardware acceleration&lt;/span&gt;
&lt;span class="na"&gt;crypto_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;cryptodev_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;cc_htcp_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="c1"&gt;# H-TCP congestion control&lt;/span&gt;
&lt;span class="na"&gt;tcpmd5_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# TCP-MD5 signatures for BGP sessions (RFC 2385)&lt;/span&gt;
&lt;span class="na"&gt;ipsec_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;virtio_random_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;tcpmd5&lt;/code&gt; and &lt;code&gt;ipsec&lt;/code&gt; are exactly the kind of modules you might keep&amp;nbsp;in &lt;code&gt;loader.conf&lt;/code&gt;: they relate to the network stack&amp;#8217;s security posture and there&amp;#8217;s an argument for having them present as early as possible. The point isn&amp;#8217;t that one file is always right - it&amp;#8217;s that you should be able to justify, for each module, &lt;em&gt;why&lt;/em&gt; it&amp;#8217;s where it is. If you can&amp;#8217;t, it probably belongs&amp;nbsp;in &lt;code&gt;kld_list&lt;/code&gt;. (The whole &lt;span class="caps"&gt;BGP&lt;/span&gt; build-out is covered in the &lt;a href="https://hofstede.it/running-your-own-as-bgp-on-freebsd-with-frr-gre-tunnels-and-policy-routing/"&gt;running your own &lt;span class="caps"&gt;AS&lt;/span&gt;&lt;/a&gt; series if you want the routing&amp;nbsp;context.)&lt;/p&gt;
&lt;h2 id="boot-environments-the-undo-button"&gt;Boot Environments: The Undo&amp;nbsp;Button&lt;/h2&gt;
&lt;p&gt;This is the part that makes the whole article worth reading. If you run root-on-&lt;span class="caps"&gt;ZFS&lt;/span&gt;, you have access to &lt;strong&gt;boot environments&lt;/strong&gt;, and they change the risk calculus of every system upgrade you will ever&amp;nbsp;do.&lt;/p&gt;
&lt;p&gt;A &lt;strong&gt;boot environment&lt;/strong&gt; (&lt;span class="caps"&gt;BE&lt;/span&gt;) is a bootable clone of your root filesystem. Recall from the &lt;a href="https://hofstede.it/freebsd-foundationals-zfs-the-last-filesystem-youll-ever-need/"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; article&lt;/a&gt; that a clone is a writable snapshot that costs essentially nothing until it diverges. FreeBSD&amp;#8217;s root-on-&lt;span class="caps"&gt;ZFS&lt;/span&gt; layout is designed to support this&amp;nbsp;pattern:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;zroot
├── ROOT
│   └── default        ← this is the active boot environment (your &amp;quot;/&amp;quot;)
├── usr
│   ├── home
│   └── ...
├── var
│   └── ...
└── tmp
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Everything&amp;nbsp;under &lt;code&gt;zroot/ROOT/&amp;lt;name&amp;gt;&lt;/code&gt; is a candidate root filesystem. The loader reads which one is marked active (the &lt;strong&gt;bootfs&lt;/strong&gt;) and boots&amp;nbsp;it. &lt;code&gt;bectl(8)&lt;/code&gt; is the tool that manages them, and the loader&amp;#8217;s boot menu can switch between them&amp;nbsp;interactively.&lt;/p&gt;
&lt;p&gt;The mental model: &lt;strong&gt;a boot environment is a complete, independent copy of your operating system root that you can switch between at the loader menu.&lt;/strong&gt; Your data&amp;nbsp;(&lt;code&gt;/usr/home&lt;/code&gt;, &lt;code&gt;/var&lt;/code&gt;, etc.) lives in separate datasets and is shared across BEs - only the &lt;span class="caps"&gt;OS&lt;/span&gt; root differs. That&amp;#8217;s the magic: rolling back the &lt;span class="caps"&gt;OS&lt;/span&gt; doesn&amp;#8217;t roll back your&amp;nbsp;data.&lt;/p&gt;
&lt;h3 id="the-workflow-that-saves-you"&gt;The Workflow That Saves&amp;nbsp;You&lt;/h3&gt;
&lt;p&gt;Here&amp;#8217;s the entire point, in five&amp;nbsp;lines:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Before any risky change: take a named boot environment&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;before-14.2-upgrade

&lt;span class="c1"&gt;# Do the scary thing&lt;/span&gt;
freebsd-update&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;14&lt;/span&gt;.2-RELEASE
freebsd-update&lt;span class="w"&gt; &lt;/span&gt;install
reboot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If the upgrade went fine, you&amp;#8217;re done - the new system is running&amp;nbsp;and &lt;code&gt;before-14.2-upgrade&lt;/code&gt; sits there as a free safety net you can destroy later. If the upgrade went &lt;em&gt;badly&lt;/em&gt; - kernel panic, broken userland, a service that won&amp;#8217;t start, a box that won&amp;#8217;t even reach multi-user - you don&amp;#8217;t restore from backup and you don&amp;#8217;t reinstall. You reboot, and at the loader&amp;#8217;s &lt;strong&gt;Boot Environments&lt;/strong&gt; menu, you&amp;nbsp;select &lt;code&gt;before-14.2-upgrade&lt;/code&gt;. Thirty seconds later you&amp;#8217;re running the exact system you had before you started, down to the last&amp;nbsp;byte.&lt;/p&gt;
&lt;p&gt;That&amp;#8217;s the undo button. A failed upgrade stops being an outage and becomes an&amp;nbsp;inconvenience.&lt;/p&gt;
&lt;h3 id="bectl-in-practice"&gt;bectl in&amp;nbsp;Practice&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# List boot environments (Active &amp;quot;N&amp;quot; = active now, &amp;quot;R&amp;quot; = active on Reboot)&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;list
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;BE                  Active Mountpoint Space Created
default             NR     /          12.4G 2026-01-04 09:12
before-14.2-upgrade -      -          1.20G 2026-06-25 21:40
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a BE from the current root&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;before-14.2-upgrade

&lt;span class="c1"&gt;# Create a BE from a specific point - e.g. an existing snapshot&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;default@clean&lt;span class="w"&gt; &lt;/span&gt;before-experiment

&lt;span class="c1"&gt;# Activate a BE for the NEXT boot (persistent until changed)&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;activate&lt;span class="w"&gt; &lt;/span&gt;before-14.2-upgrade

&lt;span class="c1"&gt;# Activate for ONE boot only - falls back to the previous BE afterward&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;activate&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;before-14.2-upgrade

&lt;span class="c1"&gt;# Mount a BE to inspect or copy files out of it without booting it&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;mount&lt;span class="w"&gt; &lt;/span&gt;before-14.2-upgrade&lt;span class="w"&gt; &lt;/span&gt;/mnt
bectl&lt;span class="w"&gt; &lt;/span&gt;umount&lt;span class="w"&gt; &lt;/span&gt;before-14.2-upgrade

&lt;span class="c1"&gt;# Rename and destroy&lt;/span&gt;
bectl&lt;span class="w"&gt; &lt;/span&gt;rename&lt;span class="w"&gt; &lt;/span&gt;before-experiment&lt;span class="w"&gt; &lt;/span&gt;kept-for-reference
bectl&lt;span class="w"&gt; &lt;/span&gt;destroy&lt;span class="w"&gt; &lt;/span&gt;before-14.2-upgrade
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;Active&lt;/code&gt; column is the one to read&amp;nbsp;carefully. &lt;code&gt;N&lt;/code&gt; means &lt;em&gt;now&lt;/em&gt; (the &lt;span class="caps"&gt;BE&lt;/span&gt; you&amp;#8217;re currently&amp;nbsp;running). &lt;code&gt;R&lt;/code&gt; means &lt;em&gt;reboot&lt;/em&gt; (the &lt;span class="caps"&gt;BE&lt;/span&gt; that will be active next&amp;nbsp;boot). &lt;code&gt;NR&lt;/code&gt; together means the running &lt;span class="caps"&gt;BE&lt;/span&gt; is also the one set for next boot - the normal steady state.&amp;nbsp;After &lt;code&gt;bectl activate other&lt;/code&gt;, you&amp;#8217;ll&amp;nbsp;see &lt;code&gt;N&lt;/code&gt; on your current &lt;span class="caps"&gt;BE&lt;/span&gt;&amp;nbsp;and &lt;code&gt;R&lt;/code&gt; on &lt;code&gt;other&lt;/code&gt;, which is your cue that a reboot will switch&amp;nbsp;you.&lt;/p&gt;
&lt;h3 id="boot-environments-and-upgrades"&gt;Boot Environments and&amp;nbsp;Upgrades&lt;/h3&gt;
&lt;p&gt;The single best habit you can build on a FreeBSD server: &lt;strong&gt;&lt;code&gt;bectl create&lt;/code&gt; before every upgrade.&lt;/strong&gt; Not just &lt;span class="caps"&gt;OS&lt;/span&gt; upgrades - package upgrades&amp;nbsp;too. &lt;code&gt;pkg upgrade&lt;/code&gt; can pull in a broken dependency or a config change that takes down a service,&amp;nbsp;and &lt;code&gt;pkg&lt;/code&gt; has no transactional rollback mechanism comparable to boot environments. A boot environment is your&amp;nbsp;rollback:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bectl&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;before-pkg-&lt;span class="k"&gt;$(&lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;+%Y%m%d&lt;span class="k"&gt;)&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade
&lt;span class="c1"&gt;# ...if something breaks and you can&amp;#39;t fix it forward, reboot into the BE&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;freebsd-update&lt;/code&gt; on recent FreeBSD can integrate with boot environments directly, and some shops&amp;nbsp;wire &lt;code&gt;bectl create&lt;/code&gt; into their package automation so every change is automatically reversible. Even without automation, a single&amp;nbsp;manual &lt;code&gt;bectl create&lt;/code&gt; costs you nothing and a few seconds, and one day it will save you an&amp;nbsp;evening.&lt;/p&gt;
&lt;h3 id="the-temporary-activation-trick"&gt;The Temporary-Activation&amp;nbsp;Trick&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;bectl activate -t&lt;/code&gt; is underused and worth knowing. It activates a &lt;span class="caps"&gt;BE&lt;/span&gt; for exactly one boot. If that boot succeeds and you&amp;#8217;re happy, you can make it permanent; if the machine doesn&amp;#8217;t come back (and you have console/&lt;span class="caps"&gt;IPMI&lt;/span&gt; to reboot it), the &lt;em&gt;next&lt;/em&gt; boot automatically reverts to the previously active &lt;span class="caps"&gt;BE&lt;/span&gt;. This is the safest possible way to test a new &lt;span class="caps"&gt;BE&lt;/span&gt; on a remote machine: worst case, a power cycle puts you back on&amp;nbsp;known-good.&lt;/p&gt;
&lt;h2 id="when-the-boot-goes-wrong-single-user-and-the-loader-prompt"&gt;When the Boot Goes Wrong: Single-User and the Loader&amp;nbsp;Prompt&lt;/h2&gt;
&lt;p&gt;Boot environments cover the &amp;#8220;a change broke the system&amp;#8221; case. For the &amp;#8220;I need to fix something with the filesystem barely mounted&amp;#8221; case, there&amp;#8217;s &lt;strong&gt;single-user mode&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;From the boot menu, choose single-user, or from the loader&amp;nbsp;prompt:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;OK boot -s
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Single-user drops you to a root shell with only the root filesystem mounted (often read-only, depending on configuration) and no services running. It&amp;#8217;s where you go&amp;nbsp;to:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Fix a&amp;nbsp;broken &lt;code&gt;/etc/rc.conf&lt;/code&gt; that&amp;#8217;s killing the&amp;nbsp;boot&lt;/li&gt;
&lt;li&gt;Run &lt;code&gt;fsck&lt;/code&gt; on a &lt;span class="caps"&gt;UFS&lt;/span&gt; filesystem (not needed for &lt;span class="caps"&gt;ZFS&lt;/span&gt;)&lt;/li&gt;
&lt;li&gt;Reset a forgotten root&amp;nbsp;password&lt;/li&gt;
&lt;li&gt;Repair&amp;nbsp;a &lt;code&gt;loader.conf&lt;/code&gt; that loads a panicking&amp;nbsp;module&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Typical single-user repair&amp;nbsp;session:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Remount root read-write&lt;/span&gt;
mount&lt;span class="w"&gt; &lt;/span&gt;-uw&lt;span class="w"&gt; &lt;/span&gt;/

&lt;span class="c1"&gt;# Mount the rest&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;mount&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# on root-on-ZFS&lt;/span&gt;
&lt;span class="c1"&gt;# or: mount -a       # on UFS&lt;/span&gt;

&lt;span class="c1"&gt;# Fix the offending file, then continue booting to multi-user&lt;/span&gt;
&lt;span class="nb"&gt;exit&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For one-off kernel selection without editing&amp;nbsp;anything, &lt;code&gt;nextboot(8)&lt;/code&gt; sets the kernel (or other loader variables) for &lt;em&gt;only the next&lt;/em&gt;&amp;nbsp;boot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Boot kernel.old exactly once on the next reboot&lt;/span&gt;
nextboot&lt;span class="w"&gt; &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;kernel.old
reboot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Between &lt;code&gt;bectl&lt;/code&gt;, single-user&amp;nbsp;mode, &lt;code&gt;nextboot&lt;/code&gt;, and the loader&amp;nbsp;prompt&amp;#8217;s &lt;code&gt;load&lt;/code&gt;/&lt;code&gt;unload&lt;/code&gt;/&lt;code&gt;boot&lt;/code&gt;, there is almost no boot failure you can&amp;#8217;t recover from in-place - provided you can reach a console. Which leads neatly to the&amp;nbsp;pitfalls.&lt;/p&gt;
&lt;h2 id="common-pitfalls"&gt;Common&amp;nbsp;Pitfalls&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Putting a tunable&amp;nbsp;in &lt;code&gt;sysctl.conf&lt;/code&gt;.&lt;/strong&gt;&amp;nbsp;Setting &lt;code&gt;net.fibs=2&lt;/code&gt; in &lt;code&gt;/etc/sysctl.conf&lt;/code&gt; does nothing useful - it&amp;#8217;s read-only at runtime, so you get an &amp;#8220;oid is read only&amp;#8221; error at boot and the second &lt;span class="caps"&gt;FIB&lt;/span&gt; never appears. Boot-time tunables go&amp;nbsp;in &lt;code&gt;loader.conf&lt;/code&gt;, full stop. When in doubt,&amp;nbsp;try &lt;code&gt;sysctl name=value&lt;/code&gt; at runtime: if it errors read-only, it&amp;#8217;s a&amp;nbsp;tunable.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Loading everything&amp;nbsp;in &lt;code&gt;loader.conf&lt;/code&gt;.&lt;/strong&gt; It works, but it bloats the loader and obscures what&amp;#8217;s actually essential to early boot.&amp;nbsp;Reserve &lt;code&gt;loader.conf&lt;/code&gt; for tunables and root-critical modules; put the rest&amp;nbsp;in &lt;code&gt;kld_list&lt;/code&gt;. Your future self, scanning the file at 3 &lt;span class="caps"&gt;AM&lt;/span&gt;, will thank&amp;nbsp;you.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;No console access on a remote box.&lt;/strong&gt; Boot environments and single-user mode are only useful if you can reboot and reach the loader menu or a serial console. On a rented server, this means &lt;span class="caps"&gt;IPMI&lt;/span&gt;/&lt;span class="caps"&gt;KVM&lt;/span&gt; or a provider rescue system. &lt;em&gt;Set this up before you need it&lt;/em&gt;, not during the outage. Test that you can actually reach the loader menu&amp;nbsp;remotely.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Forgetting &lt;code&gt;bectl create&lt;/code&gt; before upgrades.&lt;/strong&gt; The entire safety net is opt-in. A &lt;span class="caps"&gt;BE&lt;/span&gt; you didn&amp;#8217;t create can&amp;#8217;t save you. Build the habit until it&amp;#8217;s reflexive:&amp;nbsp;no &lt;code&gt;freebsd-update install&lt;/code&gt; and&amp;nbsp;no &lt;code&gt;pkg upgrade&lt;/code&gt; on a server without a fresh &lt;span class="caps"&gt;BE&lt;/span&gt;&amp;nbsp;first.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Assuming boot environments roll back your data.&lt;/strong&gt; They don&amp;#8217;t, and that&amp;#8217;s by design. A &lt;span class="caps"&gt;BE&lt;/span&gt; clones the &lt;em&gt;&lt;span class="caps"&gt;OS&lt;/span&gt; root&lt;/em&gt;,&amp;nbsp;not &lt;code&gt;/var&lt;/code&gt; or &lt;code&gt;/usr/home&lt;/code&gt;. If an upgrade corrupted a database&amp;nbsp;in &lt;code&gt;/var/db&lt;/code&gt;, booting an old &lt;span class="caps"&gt;BE&lt;/span&gt; gives you the old binaries but the &lt;em&gt;same&lt;/em&gt; damaged data. For data, you still want &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots and off-site replication (the syncoid setup from the &lt;span class="caps"&gt;ZFS&lt;/span&gt; article). BEs and snapshots are complementary, not&amp;nbsp;redundant.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Editing &lt;code&gt;/boot/defaults/loader.conf&lt;/code&gt;.&lt;/strong&gt; That file is overwritten on upgrade. Put overrides&amp;nbsp;in &lt;code&gt;/boot/loader.conf&lt;/code&gt; (or &lt;code&gt;loader.conf.local&lt;/code&gt;). Same logic as never&amp;nbsp;editing &lt;code&gt;/etc/defaults/rc.conf&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;FreeBSD&amp;#8217;s boot path is short, documented, and - crucially - recoverable. A handful of staged loaders hands off to the loader, the loader reads a plain text file and loads your kernel, the kernel mounts root and&amp;nbsp;runs &lt;code&gt;rc&lt;/code&gt;, and you&amp;#8217;re at a login prompt. Every layer is inspectable and every layer is fixable from a&amp;nbsp;console.&lt;/p&gt;
&lt;p&gt;The mental model to carry&amp;nbsp;away: &lt;code&gt;loader.conf&lt;/code&gt; is for things the loader must do &lt;em&gt;before&lt;/em&gt; the kernel exists - boot-time tunables and root-critical&amp;nbsp;modules. &lt;code&gt;sysctl.conf&lt;/code&gt; is for runtime kernel&amp;nbsp;knobs. &lt;code&gt;kld_list&lt;/code&gt; in &lt;code&gt;rc.conf&lt;/code&gt; is the modern home for every module that can wait until after root is mounted,&amp;nbsp;and &lt;code&gt;sysrc&lt;/code&gt; is how you edit all of it safely. And boot environments - the single best reason to put your root on &lt;span class="caps"&gt;ZFS&lt;/span&gt; - turn the scariest operation a sysadmin performs, the upgrade, into something you can undo at the boot menu in under a&amp;nbsp;minute.&lt;/p&gt;
&lt;p&gt;If you take four habits from this article:&amp;nbsp;keep &lt;code&gt;loader.conf&lt;/code&gt; minimal and move non-essential modules&amp;nbsp;to &lt;code&gt;kld_list&lt;/code&gt;; know the tunable-vs-sysctl distinction cold; harden at the kernel level with a&amp;nbsp;reviewed &lt;code&gt;sysctl.conf&lt;/code&gt; baseline; and&amp;nbsp;run &lt;code&gt;bectl create&lt;/code&gt; before every single upgrade. That last one will, eventually, be the best thirty seconds you ever&amp;nbsp;spent.&lt;/p&gt;
&lt;p&gt;The next article in this series tackles the other thing Linux users consistently get wrong about FreeBSD: the &lt;strong&gt;base system versus ports and packages&lt;/strong&gt; - where all the software actually comes from, why base and third-party software are deliberately separate worlds, and&amp;nbsp;how &lt;code&gt;freebsd-update&lt;/code&gt;, &lt;code&gt;pkg&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;poudriere&lt;/code&gt; fit&amp;nbsp;together.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/boot/"&gt;FreeBSD Handbook - The Boot&amp;nbsp;Process&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=loader.conf&amp;amp;sektion=5"&gt;loader.conf(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=loader&amp;amp;sektion=8"&gt;loader(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=rc.conf&amp;amp;sektion=5"&gt;rc.conf(5) man page&lt;/a&gt; -&amp;nbsp;see &lt;code&gt;kld_list&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=sysrc&amp;amp;sektion=8"&gt;sysrc(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=sysctl.conf&amp;amp;sektion=5"&gt;sysctl.conf(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=bectl&amp;amp;sektion=8"&gt;bectl(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=nextboot&amp;amp;sektion=8"&gt;nextboot(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/zfs/"&gt;FreeBSD Handbook - &lt;span class="caps"&gt;ZFS&lt;/span&gt; Boot&amp;nbsp;Environments&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=security&amp;amp;sektion=7"&gt;security(7) man page&lt;/a&gt; - the canonical list of hardening&amp;nbsp;knobs&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="boot"/><category term="loader"/><category term="loader.conf"/><category term="kld_list"/><category term="sysctl"/><category term="sysrc"/><category term="bectl"/><category term="boot-environments"/><category term="security"/><category term="hardening"/></entry><entry><title>Monitoring a FreeBSD Mastodon Instance with Prometheus, Grafana, and Loki</title><link href="https://hofstede.it/monitoring-a-freebsd-mastodon-instance-with-prometheus-grafana-and-loki/" rel="alternate"/><published>2026-06-21T00:00:00+02:00</published><updated>2026-06-21T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-06-21:/monitoring-a-freebsd-mastodon-instance-with-prometheus-grafana-and-loki/</id><summary type="html">&lt;p&gt;How I watch burningboard.net, my multi-jail FreeBSD Mastodon instance, from a separate observer host. A pull-based Prometheus stack reaching exporters across my own &lt;span class="caps"&gt;AS201379&lt;/span&gt; backbone (locked down at the perimeter firewall, not by binding to unroutable addresses), Loki and Promtail for nginx logs, a textfile collector that fills the gaps FreeBSD exporters leave (&lt;span class="caps"&gt;ZFS&lt;/span&gt;, S3, pkg audit, Mastodon &lt;span class="caps"&gt;API&lt;/span&gt; stats), one Grafana dashboard that tells me at a glance whether the instance is healthy, and an Alertmanager ruleset that emails me before users&amp;nbsp;notice.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A Grafana dashboard for a Mastodon instance: four green UP panels across the top, gauges for CPU, memory and ZFS capacity, PostgreSQL connection counts, Sidekiq queue depth, and Mastodon user and federation statistics" src="https://hofstede.it/images/2026-06-21-monitoring-mastodon-freebsd-grafana-prometheus-loki.png" title="One screen, the whole instance. If the top row is green and the red Sidekiq failure counter is not climbing, burningboard.net is healthy. Everything else is detail."&gt;&lt;/p&gt;
&lt;p&gt;A while ago I &lt;a href="https://hofstede.it/migrating-burningboardnet-mastodon-instance-to-a-multi-jail-freebsd-setup/"&gt;migrated burningboard.net to a multi-jail FreeBSD setup&lt;/a&gt;: nginx, Puma, Sidekiq, and the database each in their own jail, with the host doing all the &lt;span class="caps"&gt;PF&lt;/span&gt; and routing. That post ended on the architecture. What it did not cover is the question that matters the morning after you put real users on a thing: &lt;em&gt;is it actually healthy right now, and if it is not, will I find out before they&amp;nbsp;do?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;This is the observability half of that story. I run a completely separate machine whose only job is to watch the Mastodon host, plus a handful of other boxes in my network. It runs Prometheus for metrics, Loki for logs, Grafana to draw both, and Alertmanager to wake me up. None of it lives on the Mastodon host itself, because a monitoring system that shares a fate with the thing it watches is not a monitoring system, it is a single point of failure with extra&amp;nbsp;steps.&lt;/p&gt;
&lt;p&gt;I covered the &lt;a href="https://hofstede.it/freebsd-resource-monitoring-accounting-and-troubleshooting/"&gt;in-the-moment FreeBSD troubleshooting toolkit&lt;/a&gt;&amp;nbsp;(&lt;code&gt;top&lt;/code&gt;, &lt;code&gt;gstat&lt;/code&gt;, &lt;code&gt;netstat&lt;/code&gt;, &lt;code&gt;rctl&lt;/code&gt;) in an earlier article. This one is the opposite end: history, dashboards, and alerting. When something is on fire &lt;em&gt;right now&lt;/em&gt;, you reach&amp;nbsp;for &lt;code&gt;top -SHPj&lt;/code&gt;. When you want to know what was on fire at 3am while you were asleep, you reach for&amp;nbsp;this.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-shape-of-the-stack"&gt;The shape of the&amp;nbsp;stack&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#locking-the-exporters-down-at-the-as-perimeter"&gt;Locking the exporters down at the &lt;span class="caps"&gt;AS&lt;/span&gt;&amp;nbsp;perimeter&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#installing-the-stack-on-the-monitor-jail"&gt;Installing the stack on the monitor&amp;nbsp;jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-exporters-on-burningboard"&gt;The exporters on&amp;nbsp;burningboard&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#filling-the-gaps-the-textfile-collector"&gt;Filling the gaps: the textfile&amp;nbsp;collector&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#logs-loki-and-promtail"&gt;Logs: Loki and&amp;nbsp;Promtail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#one-dashboard-to-rule-them-all"&gt;One dashboard to rule them&amp;nbsp;all&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#alerting-the-part-that-actually-wakes-me-up"&gt;Alerting: the part that actually wakes me&amp;nbsp;up&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#freebsd-gotchas-worth-knowing"&gt;FreeBSD gotchas worth&amp;nbsp;knowing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#wrap-up"&gt;Wrap-up&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-shape-of-the-stack"&gt;The shape of the&amp;nbsp;stack&lt;/h2&gt;
&lt;p&gt;Two machines matter&amp;nbsp;here.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;burningboard&lt;/code&gt;&lt;/strong&gt; is the Mastodon host. It runs the jails and, alongside them, a set of Prometheus &lt;em&gt;exporters&lt;/em&gt;: small daemons that expose metrics over &lt;span class="caps"&gt;HTTP&lt;/span&gt;. Exporters do not push anywhere; they sit and wait to be&amp;nbsp;scraped.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;observer&lt;/code&gt;&lt;/strong&gt; is a small dedicated &lt;span class="caps"&gt;VM&lt;/span&gt; running a single Bastille jail&amp;nbsp;called &lt;code&gt;monitor&lt;/code&gt;. That jail holds the entire monitoring stack. It reaches out and pulls metrics from every exporter on a fixed&amp;nbsp;interval.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                         ┌───────────────────────────────────┐
   AS201379 backbone     │  observer  -&amp;gt;  monitor jail       │
   2a06:9801:1c::/48     │   - Prometheus   (scrape + rules) │
        ▲                │   - Grafana      (dashboards)     │
        │ pull / scrape  │   - Loki         (log store)      │
        │                │   - Alertmanager (routing/notify) │
        │                └───────────────────────────────────┘
        │
 ┌──────┴────────────────────────────────────────────────────┐
 │  burningboard (Mastodon host)                             │
 │   node_exporter      :9100   host + textfile metrics      │
 │   jail_exporter      :9452   per-jail racct               │
 │   postgres_exporter  :9187   (database jail, via pf rdr)  │
 │   sidekiq_exporter   :9394   (sidekiq jail, via pf rdr)   │
 │   promtail           :9080   ships nginx logs to Loki     │
 └───────────────────────────────────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The whole thing is &lt;strong&gt;pull-based and pull-based only&lt;/strong&gt;. Prometheus on the monitor jail decides when to scrape; the exporters have no idea Prometheus exists until it knocks. This is the single most important property for a comfortable life: I can restart, upgrade, or move the entire monitoring stack and the Mastodon host neither notices nor&amp;nbsp;cares.&lt;/p&gt;
&lt;h2 id="locking-the-exporters-down-at-the-as-perimeter"&gt;Locking the exporters down at the &lt;span class="caps"&gt;AS&lt;/span&gt;&amp;nbsp;perimeter&lt;/h2&gt;
&lt;p&gt;This is the part people get wrong, so it goes&amp;nbsp;first.&lt;/p&gt;
&lt;p&gt;The exporters&amp;nbsp;on &lt;code&gt;burningboard&lt;/code&gt; expose raw, unauthenticated&amp;nbsp;metrics. &lt;code&gt;node_exporter&lt;/code&gt; will happily tell anyone who asks how much memory is free, which kernel you boot, and how many packages are pending a security update. None of that can be reachable from the open&amp;nbsp;internet.&lt;/p&gt;
&lt;p&gt;The tempting shortcut is to bind everything to a loopback or some unroutable private range and call it secure-by-accident. I do not do that. The exporters bind to a stable address from my own &lt;span class="caps"&gt;AS201379&lt;/span&gt; &lt;span class="caps"&gt;GUA&lt;/span&gt; space&amp;nbsp;(&lt;code&gt;2a06:9801:1c::/48&lt;/code&gt;), carried on a loopback so it does not move when a physical interface does. That address is a real, globally routable IPv6 address, not a &lt;span class="caps"&gt;ULA&lt;/span&gt;. The host reaches the rest of the network over&amp;nbsp;a &lt;code&gt;gif(4)&lt;/code&gt; tunnel to my internal &lt;span class="caps"&gt;BGP&lt;/span&gt;&amp;nbsp;router:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/rc.conf on burningboard (excerpt)&lt;/span&gt;

&lt;span class="c1"&gt;# Point-to-point tunnel to the internal router&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_gif0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;tunnel 172.16.0.3 172.16.0.4 mtu 1480&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_gif0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2a06:9801:1c:fff3::2 2a06:9801:1c:fff3::1 prefixlen 128&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Stable GUA (AS201379 space) the monitoring exporters listen on&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_lo1&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2a06:9801:1c:4000::1 prefixlen 64&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Reach the rest of the AS via the internal router&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_static_routes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;internal&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_route_internal&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2a06:9801:1c::/48 2a06:9801:1c:fff3::1&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;node_exporter_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;node_exporter_listen_address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[2a06:9801:1c:4000::1]:9100&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;jail_exporter_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;jail_exporter_listen_address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[2a06:9801:1c:4000::1]:9452&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So binding is convenient, not protective. What actually keeps the rest of the internet out is the firewall at the edge of the autonomous system. I run a single perimeter firewall in front of the whole &lt;span class="caps"&gt;AS&lt;/span&gt;, and it has the final say on what is allowed&amp;nbsp;in:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;my_network_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2a06:9801:1c::/48&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Allow only the monitoring host and its jail to reach the exporter ports&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="n"&gt;a06&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;9801&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;21&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="n"&gt;a06&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;9801&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;2000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;my_network_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9342&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9187&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9394&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9452&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Everyone else is dropped at the edge, before it reaches any host&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9100&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9342&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9187&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9394&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9452&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;2a06:9801:1c:2000::21&lt;/code&gt; is&amp;nbsp;the &lt;code&gt;observer&lt;/code&gt; host&amp;nbsp;and &lt;code&gt;::25&lt;/code&gt; is&amp;nbsp;the &lt;code&gt;monitor&lt;/code&gt; jail. Those two source addresses, and only those two, may open a connection to the exporter ports anywhere in the &lt;span class="caps"&gt;AS&lt;/span&gt;: &lt;code&gt;node_exporter&lt;/code&gt; (9100), &lt;code&gt;frr_exporter&lt;/code&gt; on the routers&amp;nbsp;(9342), &lt;code&gt;postgres_exporter&lt;/code&gt; (9187), &lt;code&gt;sidekiq_exporter&lt;/code&gt; (9394),&amp;nbsp;and &lt;code&gt;jail_exporter&lt;/code&gt; (9452).&amp;nbsp;The &lt;code&gt;block in quick&lt;/code&gt; for the same ports&amp;nbsp;from &lt;code&gt;any&lt;/code&gt; to &lt;code&gt;any&lt;/code&gt; catches everything else: a port scan from the outside is dropped at the perimeter and never forwarded to the host that would otherwise answer&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;I prefer this to per-host binding tricks for one reason: it is centralised and auditable. The policy for &amp;#8220;who may scrape metrics in this network&amp;#8221; lives in exactly one ruleset at one chokepoint, not smeared across&amp;nbsp;the &lt;code&gt;rc.conf&lt;/code&gt; of every host that happens to run an exporter. Stand up a new host, give its exporter the same &lt;span class="caps"&gt;GUA&lt;/span&gt; pattern, and it is already covered: reachable from the monitoring box and from nowhere else, without touching any firewall on the new host&amp;nbsp;itself.&lt;/p&gt;
&lt;p&gt;The two exporters that live &lt;em&gt;inside&lt;/em&gt; jails (PostgreSQL in the database jail, Sidekiq in the sidekiq jail) are redirected onto the host&amp;#8217;s address with a host-local &lt;span class="caps"&gt;PF&lt;/span&gt; &lt;code&gt;rdr&lt;/code&gt;, so from Prometheus&amp;#8217;s point of view every target is just a different port&amp;nbsp;on &lt;code&gt;2a06:9801:1c:4000::1&lt;/code&gt;. That redirect is internal plumbing and entirely separate from the perimeter policy&amp;nbsp;above.&lt;/p&gt;
&lt;h2 id="installing-the-stack-on-the-monitor-jail"&gt;Installing the stack on the monitor&amp;nbsp;jail&lt;/h2&gt;
&lt;p&gt;Everything is in pkg. There is no need to compile anything or chase a Helm&amp;nbsp;chart.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# inside the monitor jail&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;prometheus&lt;span class="w"&gt; &lt;/span&gt;grafana&lt;span class="w"&gt; &lt;/span&gt;loki&lt;span class="w"&gt; &lt;/span&gt;alertmanager

sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;prometheus_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;grafana_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;loki_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;alertmanager_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

service&lt;span class="w"&gt; &lt;/span&gt;prometheus&lt;span class="w"&gt; &lt;/span&gt;start
service&lt;span class="w"&gt; &lt;/span&gt;grafana&lt;span class="w"&gt; &lt;/span&gt;start
service&lt;span class="w"&gt; &lt;/span&gt;loki&lt;span class="w"&gt; &lt;/span&gt;start
service&lt;span class="w"&gt; &lt;/span&gt;alertmanager&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The jail itself is a thin Bastille jail and publishes a couple of ports to the host so I can reach&amp;nbsp;Grafana:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@observer:~ # bastille list
 JID  Name     State  IP Address              Published Ports        Release
 1    monitor  Up     10.254.254.25           tcp/3000:3000,...      15.0-RELEASE-p10
                      2a06:9801:1c:2000::25
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Grafana is&amp;nbsp;on &lt;code&gt;:3000&lt;/code&gt;, Prometheus&amp;nbsp;on &lt;code&gt;:9090&lt;/code&gt;, Alertmanager&amp;nbsp;on &lt;code&gt;:9093&lt;/code&gt;, Loki&amp;nbsp;on &lt;code&gt;:3100&lt;/code&gt;. Only Grafana is meant to be looked at by a human; the rest are&amp;nbsp;plumbing.&lt;/p&gt;
&lt;h2 id="the-exporters-on-burningboard"&gt;The exporters on&amp;nbsp;burningboard&lt;/h2&gt;
&lt;p&gt;Four exporters cover the bulk of the instance. All come from pkg, and Prometheus reaches every one of them on the loopback &lt;span class="caps"&gt;GUA&lt;/span&gt; &lt;code&gt;2a06:9801:1c:4000::1&lt;/code&gt;: the host exporters bind it directly, the jailed ones arrive through&amp;nbsp;the &lt;code&gt;rdr&lt;/code&gt; described&amp;nbsp;above.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;node_exporter&lt;/code&gt; (&lt;code&gt;:9100&lt;/code&gt;)&lt;/strong&gt; is the foundation: &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, load, filesystems, network counters. On FreeBSD it also serves as the host for my &lt;em&gt;textfile collector&lt;/em&gt;, which I lean on heavily&amp;nbsp;below.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;jail_exporter&lt;/code&gt; (&lt;code&gt;:9452&lt;/code&gt;)&lt;/strong&gt;&amp;nbsp;reads &lt;code&gt;kern.racct&lt;/code&gt; and exposes per-jail &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, and process counts. It is the Prometheus version of&amp;nbsp;the &lt;code&gt;rctl -hu&lt;/code&gt; view I &lt;a href="https://hofstede.it/freebsd-resource-monitoring-accounting-and-troubleshooting/"&gt;wrote about before&lt;/a&gt;. Worth its own line in the dashboard so I can see which jail is&amp;nbsp;growing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;postgres_exporter&lt;/code&gt; (&lt;code&gt;:9187&lt;/code&gt;)&lt;/strong&gt; runs against the Mastodon database. The two series I actually watch&amp;nbsp;are &lt;code&gt;pg_stat_database_numbackends&lt;/code&gt; (connection pressure)&amp;nbsp;and &lt;code&gt;pg_stat_database_deadlocks&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;sidekiq_exporter&lt;/code&gt; (&lt;code&gt;:9394&lt;/code&gt;)&lt;/strong&gt; is the one that matters most for a Mastodon instance. Sidekiq is where federation happens. If its queues back up, posts stop delivering and the instance &lt;em&gt;feels&lt;/em&gt; broken long before anything actually&amp;nbsp;errors.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A&amp;nbsp;trimmed &lt;code&gt;prometheus.yml&lt;/code&gt; showing the scrape&amp;nbsp;jobs:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;global&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;scrape_interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10s&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;evaluation_interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10s&lt;/span&gt;

&lt;span class="nt"&gt;rule_files&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;alerts.yml&amp;quot;&lt;/span&gt;

&lt;span class="nt"&gt;alerting&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;alertmanagers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;localhost:9093&amp;quot;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;

&lt;span class="nt"&gt;scrape_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;job_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;node&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;[2a06:9801:1c:4000::1]:9100&amp;#39;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;instance&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;burningboard.net&amp;#39;&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;job_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;jail_exporter&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;[2a06:9801:1c:4000::1]:9452&amp;#39;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;instance&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;burningboard.net&amp;#39;&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;job_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;postgres_exporter&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;[2a06:9801:1c:4000::1]:9187&amp;#39;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;instance&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;burningboard.net&amp;#39;&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;job_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;sidekiq_exporter&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;[2a06:9801:1c:4000::1]:9394&amp;#39;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;instance&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;burningboard.net&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The same Prometheus also scrapes my mail server, my &lt;span class="caps"&gt;BGP&lt;/span&gt; routers&amp;nbsp;(&lt;code&gt;frr_exporter&lt;/code&gt;), and a couple of other hosts on the backbone, but those are off-topic&amp;nbsp;here.&lt;/p&gt;
&lt;h2 id="filling-the-gaps-the-textfile-collector"&gt;Filling the gaps: the textfile&amp;nbsp;collector&lt;/h2&gt;
&lt;p&gt;Out of the box, the exporters leave real holes.&amp;nbsp;FreeBSD&amp;#8217;s &lt;code&gt;node_exporter&lt;/code&gt; does &lt;strong&gt;not&lt;/strong&gt; expose &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool metrics natively the way the Linux build does. Nothing knows my Scaleway S3 media bucket exists. Nothing knows&amp;nbsp;whether &lt;code&gt;pkg audit&lt;/code&gt; is unhappy. And the most interesting numbers of all, the actual &lt;em&gt;Mastodon&lt;/em&gt; statistics (users, toots, federated domains, monthly active users), live behind the instance &lt;span class="caps"&gt;API&lt;/span&gt;, not in any&amp;nbsp;exporter.&lt;/p&gt;
&lt;p&gt;The clean FreeBSD answer is the &lt;strong&gt;textfile collector&lt;/strong&gt;. &lt;code&gt;node_exporter&lt;/code&gt; can be told to read&amp;nbsp;every &lt;code&gt;*.prom&lt;/code&gt; file in a directory and serve its contents as if they were native metrics. You enable it with one&amp;nbsp;argument:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/rc.conf on burningboard&lt;/span&gt;
&lt;span class="nv"&gt;node_exporter_args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--collector.textfile.directory=/var/tmp/node_exporter&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Create that directory yourself first&amp;nbsp;(&lt;code&gt;mkdir -p /var/tmp/node_exporter&lt;/code&gt;); node_exporter will not create it, and if it is missing the&amp;nbsp;final &lt;code&gt;mv&lt;/code&gt; in the script below fails and the custom metrics silently never&amp;nbsp;appear.&lt;/p&gt;
&lt;p&gt;Then you write a script that produces well-formed Prometheus text into that directory and run it on a schedule. Mine pulls four sources together and writes them as a&amp;nbsp;single &lt;code&gt;.prom&lt;/code&gt; file (Scaleway credentials&amp;nbsp;redacted):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# /usr/local/bin/mastodon_custom_metrics.sh&lt;/span&gt;

&lt;span class="nv"&gt;PROM_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/tmp/node_exporter/mastodon_custom.prom&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;TMP_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;PROM_FILE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.&lt;/span&gt;&lt;span class="nv"&gt;$$&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Scaleway credentials. Keep this file chmod 600 and out of version control.&lt;/span&gt;
&lt;span class="nv"&gt;SCW_SECRET_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;...&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# redacted&lt;/span&gt;
&lt;span class="nv"&gt;PROJECT_ID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;...&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="c1"&gt;# redacted&lt;/span&gt;
&lt;span class="nv"&gt;BUCKET_NAME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;...&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="c1"&gt;# redacted&lt;/span&gt;

&lt;span class="c1"&gt;# --- 1. Mastodon instance API ---&lt;/span&gt;
&lt;span class="nv"&gt;INSTANCE_JSON&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://burningboard.net/api/v1/instance&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;INSTANCE_V2_JSON&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://burningboard.net/api/v2/instance&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;USERS_TOTAL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$INSTANCE_JSON&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.stats.user_count   // 0&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;TOOTS_TOTAL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$INSTANCE_JSON&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.stats.status_count  // 0&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;DOMAINS_TOTAL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$INSTANCE_JSON&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.stats.domain_count  // 0&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;ACTIVE_MONTH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$INSTANCE_V2_JSON&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.usage.users.active_month // 0&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Weekly activity (index [0] is the current week)&lt;/span&gt;
&lt;span class="nv"&gt;ACTIVITY_JSON&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://burningboard.net/api/v1/instance/activity&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;WEEKLY_REG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ACTIVITY_JSON&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.[0].registrations // 0&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;WEEKLY_LOGINS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ACTIVITY_JSON&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;.[0].logins // 0&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- 2. Scaleway S3 media bucket (size + object count) ---&lt;/span&gt;
&lt;span class="nv"&gt;S3_JSON&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;-X&lt;span class="w"&gt; &lt;/span&gt;POST&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://api.scaleway.com/object-private/v1/regions/fr-par/buckets-info&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;X-Auth-Token: &lt;/span&gt;&lt;span class="nv"&gt;$SCW_SECRET_KEY&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Content-Type: application/json&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--data-raw&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;{\&amp;quot;buckets_name\&amp;quot;:[\&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$BUCKET_NAME&lt;/span&gt;&lt;span class="s2"&gt;\&amp;quot;],\&amp;quot;project_id\&amp;quot;:\&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PROJECT_ID&lt;/span&gt;&lt;span class="s2"&gt;\&amp;quot;}&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;S3_SIZE_BYTES&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$S3_JSON&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.buckets.\&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$BUCKET_NAME&lt;/span&gt;&lt;span class="s2"&gt;\&amp;quot;.current_size    // 0&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;S3_OBJECT_COUNT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$S3_JSON&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;.buckets.\&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$BUCKET_NAME&lt;/span&gt;&lt;span class="s2"&gt;\&amp;quot;.current_objects // 0&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- 3. FreeBSD package / security posture ---&lt;/span&gt;
&lt;span class="nv"&gt;PKG_UPDATES&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;version&lt;span class="w"&gt; &lt;/span&gt;-vRL&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;lt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;PKG_AUDIT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;audit&lt;span class="w"&gt; &lt;/span&gt;-Fq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;^&lt;/span&gt;$&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;wc&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39; &amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;freebsd-version&lt;span class="w"&gt; &lt;/span&gt;-k&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;REBOOT_REQ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;REBOOT_REQ&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;

&lt;span class="c1"&gt;# --- 4. ZFS pool metrics (not native on FreeBSD node_exporter) ---&lt;/span&gt;
&lt;span class="nv"&gt;ZPOOL_INFO&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;zpool&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;-H&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;zroot&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;ZPOOL_FREE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ZPOOL_INFO&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{print $4}&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;ZPOOL_FRAG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ZPOOL_INFO&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{print $7}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;ZPOOL_CAP&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ZPOOL_INFO&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{print $8}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- Emit Prometheus text, then move atomically into place ---&lt;/span&gt;
cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF &amp;gt; &amp;quot;$TMP_FILE&amp;quot;&lt;/span&gt;
&lt;span class="s"&gt;# --- ZFS ---&lt;/span&gt;
&lt;span class="s"&gt;# HELP zfs_pool_free_bytes Free bytes in zpool&lt;/span&gt;
&lt;span class="s"&gt;# TYPE zfs_pool_free_bytes gauge&lt;/span&gt;
&lt;span class="s"&gt;zfs_pool_free_bytes{pool=&amp;quot;zroot&amp;quot;} ${ZPOOL_FREE:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP zfs_pool_capacity_percent Capacity percentage&lt;/span&gt;
&lt;span class="s"&gt;# TYPE zfs_pool_capacity_percent gauge&lt;/span&gt;
&lt;span class="s"&gt;zfs_pool_capacity_percent{pool=&amp;quot;zroot&amp;quot;} ${ZPOOL_CAP:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP zfs_pool_fragmentation_percent Fragmentation percentage&lt;/span&gt;
&lt;span class="s"&gt;# TYPE zfs_pool_fragmentation_percent gauge&lt;/span&gt;
&lt;span class="s"&gt;zfs_pool_fragmentation_percent{pool=&amp;quot;zroot&amp;quot;} ${ZPOOL_FRAG:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP zfs_pool_health ZFS pool health (1=ONLINE, 0=otherwise)&lt;/span&gt;
&lt;span class="s"&gt;# TYPE zfs_pool_health gauge&lt;/span&gt;
&lt;span class="s"&gt;zfs_pool_health{pool=&amp;quot;zroot&amp;quot;} $(zpool list -H -o health zroot | grep -q ONLINE &amp;amp;&amp;amp; echo 1 || echo 0)&lt;/span&gt;

&lt;span class="s"&gt;# HELP zfs_postgres_data_used_bytes Bytes used by the postgres_data dataset&lt;/span&gt;
&lt;span class="s"&gt;# TYPE zfs_postgres_data_used_bytes gauge&lt;/span&gt;
&lt;span class="s"&gt;zfs_postgres_data_used_bytes $(zfs get -Hp used zroot/postgres_data 2&amp;gt;/dev/null | awk &amp;#39;{print $3}&amp;#39;)&lt;/span&gt;

&lt;span class="s"&gt;# --- Mastodon (instance API) ---&lt;/span&gt;
&lt;span class="s"&gt;# HELP mastodon_local_users_total Total local accounts&lt;/span&gt;
&lt;span class="s"&gt;# TYPE mastodon_local_users_total gauge&lt;/span&gt;
&lt;span class="s"&gt;mastodon_local_users_total ${USERS_TOTAL:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP mastodon_local_toots_total Total local statuses&lt;/span&gt;
&lt;span class="s"&gt;# TYPE mastodon_local_toots_total gauge&lt;/span&gt;
&lt;span class="s"&gt;mastodon_local_toots_total ${TOOTS_TOTAL:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP mastodon_known_domains Total federated domains&lt;/span&gt;
&lt;span class="s"&gt;# TYPE mastodon_known_domains gauge&lt;/span&gt;
&lt;span class="s"&gt;mastodon_known_domains ${DOMAINS_TOTAL:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP mastodon_active_month_users Monthly active users (MAU)&lt;/span&gt;
&lt;span class="s"&gt;# TYPE mastodon_active_month_users gauge&lt;/span&gt;
&lt;span class="s"&gt;mastodon_active_month_users ${ACTIVE_MONTH:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP mastodon_weekly_registrations Registrations in the current week&lt;/span&gt;
&lt;span class="s"&gt;# TYPE mastodon_weekly_registrations gauge&lt;/span&gt;
&lt;span class="s"&gt;mastodon_weekly_registrations ${WEEKLY_REG:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP mastodon_weekly_logins Logins in the current week&lt;/span&gt;
&lt;span class="s"&gt;# TYPE mastodon_weekly_logins gauge&lt;/span&gt;
&lt;span class="s"&gt;mastodon_weekly_logins ${WEEKLY_LOGINS:-0}&lt;/span&gt;

&lt;span class="s"&gt;# --- Scaleway S3 media bucket ---&lt;/span&gt;
&lt;span class="s"&gt;# HELP scaleway_s3_bucket_size_bytes Current size of the S3 media bucket&lt;/span&gt;
&lt;span class="s"&gt;# TYPE scaleway_s3_bucket_size_bytes gauge&lt;/span&gt;
&lt;span class="s"&gt;scaleway_s3_bucket_size_bytes ${S3_SIZE_BYTES:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP scaleway_s3_objects_total Number of objects in the S3 media bucket&lt;/span&gt;
&lt;span class="s"&gt;# TYPE scaleway_s3_objects_total gauge&lt;/span&gt;
&lt;span class="s"&gt;scaleway_s3_objects_total ${S3_OBJECT_COUNT:-0}&lt;/span&gt;

&lt;span class="s"&gt;# --- FreeBSD package / security posture ---&lt;/span&gt;
&lt;span class="s"&gt;# HELP freebsd_pending_pkg_updates Number of pending package upgrades&lt;/span&gt;
&lt;span class="s"&gt;# TYPE freebsd_pending_pkg_updates gauge&lt;/span&gt;
&lt;span class="s"&gt;freebsd_pending_pkg_updates ${PKG_UPDATES:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP freebsd_security_audits Number of vulnerable packages&lt;/span&gt;
&lt;span class="s"&gt;# TYPE freebsd_security_audits gauge&lt;/span&gt;
&lt;span class="s"&gt;freebsd_security_audits ${PKG_AUDIT:-0}&lt;/span&gt;

&lt;span class="s"&gt;# HELP freebsd_reboot_required 1 if a kernel update is pending&lt;/span&gt;
&lt;span class="s"&gt;# TYPE freebsd_reboot_required gauge&lt;/span&gt;
&lt;span class="s"&gt;freebsd_reboot_required ${REBOOT_REQ:-0}&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;

mv&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMP_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PROM_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two details that are easy to get wrong and that&amp;nbsp;matter:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Write to a temp file&amp;nbsp;and &lt;code&gt;mv&lt;/code&gt; it into&amp;nbsp;place.&lt;/strong&gt; &lt;code&gt;node_exporter&lt;/code&gt; may read the directory at any instant. A&amp;nbsp;half-written &lt;code&gt;.prom&lt;/code&gt; file is a parse error and a gap in your&amp;nbsp;graphs. &lt;code&gt;mv&lt;/code&gt; on the same filesystem is atomic, so the reader always sees either the old file or the new one, never a torn&amp;nbsp;one.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Always default&amp;nbsp;to &lt;code&gt;0&lt;/code&gt;.&lt;/strong&gt; Every value&amp;nbsp;uses &lt;code&gt;${VAR:-0}&lt;/code&gt;.&amp;nbsp;If &lt;code&gt;curl&lt;/code&gt; to the Scaleway &lt;span class="caps"&gt;API&lt;/span&gt; times out, I want the metric to&amp;nbsp;read &lt;code&gt;0&lt;/code&gt; (and trip an alert), not vanish and silently leave a&amp;nbsp;hole.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;A cron entry runs it every five&amp;nbsp;minutes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;*/5&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;root&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bin/mastodon_custom_metrics.sh
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That five-minute cadence is deliberate. These are slow-moving numbers (a bucket&amp;#8217;s object count and your &lt;span class="caps"&gt;MAU&lt;/span&gt; do not change second to second), and hammering the public instance &lt;span class="caps"&gt;API&lt;/span&gt; or the Scaleway control plane every ten seconds would be rude and&amp;nbsp;pointless.&lt;/p&gt;
&lt;h2 id="logs-loki-and-promtail"&gt;Logs: Loki and&amp;nbsp;Promtail&lt;/h2&gt;
&lt;p&gt;Metrics tell you &lt;em&gt;that&lt;/em&gt; something is wrong. Logs tell you &lt;em&gt;what&lt;/em&gt;. For the log half I run Loki on the monitor jail and Promtail on the nginx&amp;nbsp;jail.&lt;/p&gt;
&lt;p&gt;Promtail tails the nginx access logs, does a little filtering, and pushes the survivors to Loki over the same &lt;span class="caps"&gt;AS201379&lt;/span&gt;&amp;nbsp;backbone:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /usr/local/etc/promtail.yaml on the nginx jail&lt;/span&gt;
&lt;span class="nt"&gt;clients&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;url&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;http://[2a06:9801:1c:2000::25]:3100/loki/api/v1/push&lt;/span&gt;

&lt;span class="nt"&gt;scrape_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;job_name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;nginx&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;static_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;targets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;localhost&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;job&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;nginx&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;host&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;burningboard.net&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;__path__&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/var/log/nginx/analytics*log&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;pipeline_stages&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;json&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;expressions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nt"&gt;http_user_agent&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nt"&gt;request_uri&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;http_user_agent&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;expression&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;(bot|Bot|RSS|spider|crawler|Crawler|Inspect)&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;request_uri&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;expression&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/(assets|img)/&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;source&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;request_uri&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;expression&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;(robots.txt|favicon.ico|index.php)&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;drop&lt;/code&gt; stages are the point. A Mastodon instance is crawled relentlessly by bots, link-preview fetchers, and &lt;span class="caps"&gt;RSS&lt;/span&gt; readers, and serves a flood of static assets. None of that is interesting when you are trying to read real human traffic, and all of it costs storage in Loki. Dropping it at the agent, before it ever crosses the network, keeps the log store small and the queries fast. nginx is configured to write &lt;span class="caps"&gt;JSON&lt;/span&gt; access logs, which is what lets Promtail&amp;nbsp;parse &lt;code&gt;http_user_agent&lt;/code&gt; and &lt;code&gt;request_uri&lt;/code&gt; as structured fields instead of regex-wrestling a log&amp;nbsp;line.&lt;/p&gt;
&lt;p&gt;Loki itself runs in the simplest possible single-binary, filesystem-backed mode. For one Mastodon instance and a handful of hosts there is no reason to reach for object storage or a microservices&amp;nbsp;deployment:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /usr/local/etc/loki.yaml (excerpt)&lt;/span&gt;
&lt;span class="nt"&gt;auth_enabled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="nt"&gt;common&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;path_prefix&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/var/db/loki&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;storage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;filesystem&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;chunks_directory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/var/db/loki/chunks&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;rules_directory&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/var/db/loki/rules&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;replication_factor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;1&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ring&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;kvstore&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;store&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;inmemory&lt;/span&gt;
&lt;span class="nt"&gt;schema_config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;from&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2020-10-24&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;store&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;tsdb&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;object_store&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;filesystem&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;schema&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;v13&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;index&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;index_&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;period&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;24h&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In Grafana, Loki becomes a second data source next to Prometheus, and a LogQL query&amp;nbsp;like &lt;code&gt;{job="nginx"} |= "499"&lt;/code&gt; next to the request-rate graph turns &amp;#8220;the dashboard went red&amp;#8221; into &amp;#8220;here are the exact requests that did it&amp;#8221; without leaving the&amp;nbsp;browser.&lt;/p&gt;
&lt;h2 id="one-dashboard-to-rule-them-all"&gt;One dashboard to rule them&amp;nbsp;all&lt;/h2&gt;
&lt;p&gt;The screenshot at the top of this post is the dashboard I actually look at. It is deliberately one screen, organised top to bottom by how quickly I need the&amp;nbsp;information:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Top row, the health check.&lt;/strong&gt; Four&amp;nbsp;big &lt;code&gt;UP&lt;/code&gt;/&lt;code&gt;DOWN&lt;/code&gt; panels, one per exporter&amp;nbsp;(&lt;code&gt;up{job="..."} == 1&lt;/code&gt;). Green means Prometheus is successfully scraping. If any of these go red, nothing below them can be trusted, so they go&amp;nbsp;first.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;System resources.&lt;/strong&gt; &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory utilisation, and &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool capacity as gauges, with time-series underneath. The gauges are for the glance; the graphs are for &amp;#8220;when did this&amp;nbsp;start.&amp;#8221;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Storage and I/O.&lt;/strong&gt; &lt;span class="caps"&gt;ZFS&lt;/span&gt; free space and health, PostgreSQL data set size, and the S3 media bucket (size and object count) straight from the textfile&amp;nbsp;metrics.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;PostgreSQL.&lt;/strong&gt; Active connections against&amp;nbsp;the &lt;code&gt;max_connections&lt;/code&gt; ceiling, plus deadlocks. On a busy Mastodon instance the connection pool is the thing most likely to bite&amp;nbsp;you.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Sidekiq and Mastodon.&lt;/strong&gt; Queue backlog and latency, processed/failed job counters, and the human-facing numbers: local users, statuses, known instances, weekly logins and registrations. The red failed-jobs counter is the one panel I check first every morning. A &lt;em&gt;flat&lt;/em&gt; failure count is fine; a &lt;em&gt;climbing&lt;/em&gt; one means federation is&amp;nbsp;unhappy.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The whole design philosophy is &lt;em&gt;answer the binary question first&lt;/em&gt;. If the top row is green and the Sidekiq failure rate is flat, the instance is healthy and I can close the tab. Only when something is off do I start reading the detail panels&amp;nbsp;underneath.&lt;/p&gt;
&lt;h2 id="alerting-the-part-that-actually-wakes-me-up"&gt;Alerting: the part that actually wakes me&amp;nbsp;up&lt;/h2&gt;
&lt;p&gt;A dashboard you have to look at is not monitoring, it is a hobby. The point of the whole exercise is that I should not &lt;em&gt;have&lt;/em&gt; to look. Prometheus evaluates a ruleset and hands firing alerts to Alertmanager, which routes them to email (and optionally a Telegram&amp;nbsp;bridge).&lt;/p&gt;
&lt;p&gt;A representative slice&amp;nbsp;of &lt;code&gt;alerts.yml&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;node_exporter&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;NodeDown&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;up{job=&amp;quot;node&amp;quot;} == 0&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2m&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;{&lt;/span&gt;&lt;span class="nt"&gt; severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;critical&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Node&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;exporter&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;unreachable:&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$labels.instance&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;disk_zfs&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ZfsPoolLowSpace&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;zfs_pool_free_bytes &amp;lt; 10737418240&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# &amp;lt; 10 GiB&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;5m&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;{&lt;/span&gt;&lt;span class="nt"&gt; severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;critical&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;ZFS&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;pool&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;10&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;GiB&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;free&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;({{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$labels.pool&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}})&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ZfsPoolDegraded&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;zfs_pool_health != 1&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;0m&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;{&lt;/span&gt;&lt;span class="nt"&gt; severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;critical&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;postgresql&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;PostgresConnectionsHigh&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;sum(pg_stat_database_numbackends{datname=&amp;quot;mastodon&amp;quot;}) &amp;gt; 80&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;3m&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;{&lt;/span&gt;&lt;span class="nt"&gt; severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;warning&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;PostgreSQL&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;connections&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;approaching&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;max&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;(100)&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;sidekiq&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;SidekiqQueueBacklog&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;sidekiq_queue_backlog &amp;gt; 100&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10m&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;{&lt;/span&gt;&lt;span class="nt"&gt; severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;warning&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Sidekiq&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;backlog&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$value&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;on&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$labels.queue&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;-&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;federation&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;stuck&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;SidekiqLatencyHigh&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;sidekiq_queue_latency_seconds &amp;gt; 300&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;5m&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;{&lt;/span&gt;&lt;span class="nt"&gt; severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;warning&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastodon&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;rules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;FreeBSDSecurityAudit&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;freebsd_security_audits &amp;gt; 0&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;6h&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;{&lt;/span&gt;&lt;span class="nt"&gt; severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;warning&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Vulnerable&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;packages&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;on&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$labels.instance&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;({{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;$value&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;CVEs)&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;alert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;S3MediaBucketEmpty&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;expr&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;scaleway_s3_objects_total == 0&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;for&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;1h&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;labels&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;{&lt;/span&gt;&lt;span class="nt"&gt; severity&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;critical&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;annotations&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;summary&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;S3&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;media&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;bucket&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;reporting&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;objects&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;-&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;users&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;will&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;see&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;broken&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;media&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every rule has&amp;nbsp;a &lt;code&gt;for:&lt;/code&gt; clause so a single bad scrape does not page&amp;nbsp;me. &lt;code&gt;NodeDown&lt;/code&gt; has to be true for two minutes; a backlog has to persist for ten. The thresholds encode hard-won opinions: 80 PostgreSQL connections is &amp;#8220;pay attention,&amp;#8221; 100 is &amp;#8220;you are out.&amp;#8221; A Sidekiq backlog over 100 sustained for ten minutes means workers cannot keep up with federation and someone is about to notice their posts are not&amp;nbsp;landing.&lt;/p&gt;
&lt;p&gt;Alertmanager handles the delivery and, crucially, the grouping and de-duplication so a flapping host does not bury me in&amp;nbsp;mail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;route&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;receiver&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;email-admins&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;group_by&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="s"&gt;&amp;#39;alertname&amp;#39;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;instance&amp;#39;&lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;group_wait&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;30s&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;group_interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;5m&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;repeat_interval&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;4h&lt;/span&gt;

&lt;span class="nt"&gt;receivers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;email-admins&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;email_configs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;backups@linuxserver.pro&amp;#39;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;send_resolved&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;send_resolved: true&lt;/code&gt; is the underrated setting: I get a second mail when the condition clears, so an alert at 02:00 that resolves itself by 02:06 is visibly a transient, not a four-hour mystery I wake up&amp;nbsp;to.&lt;/p&gt;
&lt;h2 id="freebsd-gotchas-worth-knowing"&gt;FreeBSD gotchas worth&amp;nbsp;knowing&lt;/h2&gt;
&lt;p&gt;A few things cost me time, so they might save you&amp;nbsp;some.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Verify metric names against your live Prometheus, not the docs.&lt;/strong&gt; This is the big one. The Sidekiq backlog series&amp;nbsp;is &lt;code&gt;sidekiq_queue_backlog&lt;/code&gt;,&amp;nbsp;not &lt;code&gt;sidekiq_queue_enqueued&lt;/code&gt;. &lt;code&gt;frr_bgp_peer_state == 1&lt;/code&gt; means &lt;em&gt;Established&lt;/em&gt;, not the &lt;span class="caps"&gt;IETF&lt;/span&gt; value of 6. I wrote a whole ruleset against names from blog posts and documentation, then watched half of it silently never fire because the series did not exist. Open the Prometheus expression browser, type the metric prefix, and confirm the series is real before you write an alert that depends on it. An alert on a nonexistent metric is worse than no alert: it gives you false&amp;nbsp;confidence.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;FreeBSD memory metrics are not the Linux ones.&lt;/strong&gt; There is&amp;nbsp;no &lt;code&gt;node_memory_MemAvailable_bytes&lt;/code&gt; on FreeBSD. You&amp;nbsp;get &lt;code&gt;node_memory_size_bytes&lt;/code&gt; and the active/inactive/wired/cache breakdown, and you compute &amp;#8220;available&amp;#8221; yourself.&amp;nbsp;My &lt;code&gt;HostOutOfMemory&lt;/code&gt; rule carries both forms so the same ruleset works across my Linux and FreeBSD&amp;nbsp;nodes.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; is not exported natively.&lt;/strong&gt; As covered above, the&amp;nbsp;FreeBSD &lt;code&gt;node_exporter&lt;/code&gt; build does not ship the &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool collector the Linux one has. The textfile collector is the pragmatic fix, and it has the side benefit that I emit exactly the handful&amp;nbsp;of &lt;code&gt;zfs_pool_*&lt;/code&gt; series I care about rather than the firehose the native collector&amp;nbsp;produces.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;kern_securelevel="2"&lt;/code&gt; does not get in the way.&lt;/strong&gt; The Mastodon host runs at securelevel 2, which is great for integrity and mildly alarming the first time you wonder whether an exporter will trip over it. It does not. node_exporter, jail_exporter, and a shell script writing&amp;nbsp;to &lt;code&gt;/var/tmp&lt;/code&gt; all operate well within what securelevel&amp;nbsp;allows.&lt;/p&gt;
&lt;h2 id="wrap-up"&gt;Wrap-up&lt;/h2&gt;
&lt;p&gt;The stack is, by design, unremarkable. Prometheus pulls, Grafana draws, Loki holds the logs, Alertmanager sends the mail. Every piece is in pkg, every config is a flat file under version control, and the whole thing runs in one Bastille jail on a machine that does nothing else. There is no agent mesh, no push gateway, no time-series database to babysit, and nothing on the Mastodon host that can take it&amp;nbsp;down.&lt;/p&gt;
&lt;p&gt;What makes it &lt;em&gt;useful&lt;/em&gt; rather than just present is the unglamorous middle layer: a textfile collector that turns the Mastodon &lt;span class="caps"&gt;API&lt;/span&gt;, a Scaleway&amp;nbsp;bucket, &lt;code&gt;pkg audit&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;zpool list&lt;/code&gt; into first-class metrics; Promtail dropping bot noise before it costs anything; and an alert ruleset whose thresholds are opinions I actually hold about when to care. That is the part you cannot install from a&amp;nbsp;package.&lt;/p&gt;
&lt;p&gt;The result is the same thing I want from all my infrastructure: I look at one green screen in the morning, and on the rare day it is not green, I already got the mail. Boringly reliable, which is exactly how I like&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/migrating-burningboardnet-mastodon-instance-to-a-multi-jail-freebsd-setup/"&gt;Migrating burningboard.net to a Multi-Jail FreeBSD&amp;nbsp;Setup&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/freebsd-resource-monitoring-accounting-and-troubleshooting/"&gt;FreeBSD Resource Monitoring, Accounting, and&amp;nbsp;Troubleshooting&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://prometheus.io/docs/"&gt;Prometheus&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/prometheus/node_exporter#textfile-collector"&gt;node_exporter textfile&amp;nbsp;collector&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/phyber/jail_exporter"&gt;jail_exporter&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://grafana.com/docs/loki/latest/"&gt;Grafana&amp;nbsp;Loki&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://grafana.com/docs/loki/latest/send-data/promtail/"&gt;Promtail&amp;nbsp;configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://prometheus.io/docs/alerting/latest/alertmanager/"&gt;Alertmanager&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.joinmastodon.org/methods/instance/"&gt;Mastodon instance &lt;span class="caps"&gt;API&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="mastodon"/><category term="monitoring"/><category term="observability"/><category term="prometheus"/><category term="grafana"/><category term="loki"/><category term="jails"/><category term="ipv6"/></entry><entry><title>mdo on FreeBSD 15: Base-System Privilege Delegation with mac_do</title><link href="https://hofstede.it/mdo-on-freebsd-15-base-system-privilege-delegation-with-mac_do/" rel="alternate"/><published>2026-05-18T00:00:00+02:00</published><updated>2026-05-18T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-05-18:/mdo-on-freebsd-15-base-system-privilege-delegation-with-mac_do/</id><summary type="html">&lt;p&gt;FreeBSD 15&amp;nbsp;ships &lt;code&gt;mdo(1)&lt;/code&gt; and&amp;nbsp;the &lt;code&gt;mac_do(4)&lt;/code&gt; policy module in the base system. It&amp;nbsp;replaces &lt;code&gt;sudo&lt;/code&gt; and &lt;code&gt;doas&lt;/code&gt; for most of my hosts, needs no ports, and configures with a single sysctl. This article walks through enabling it, the rule syntax, a few real examples, and a short detour into the security sysctls I run on every&amp;nbsp;box.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;div class="alert alert-info" role="alert"&gt;
&lt;p&gt;&lt;strong&gt;Update (2026-05-18):&lt;/strong&gt; An earlier version of this article&amp;nbsp;used &lt;code&gt;gid=wheel&lt;/code&gt; in&amp;nbsp;a &lt;code&gt;mac_do&lt;/code&gt; rule example. That does not parse: the rule grammar takes numeric IDs only, and the kernel rejects the name&amp;nbsp;with &lt;code&gt;Invalid argument&lt;/code&gt;. The correct form&amp;nbsp;is &lt;code&gt;gid=0&lt;/code&gt; (the numeric &lt;span class="caps"&gt;GID&lt;/span&gt;&amp;nbsp;of &lt;code&gt;wheel&lt;/code&gt;). The &lt;a href="#the-rule-language"&gt;Rule Language&lt;/a&gt; section has been corrected and a short note added about this sharp edge. Thanks to &lt;a href="https://friedcheese.us/@feld"&gt;@feld@friedcheese.us&lt;/a&gt; for the&amp;nbsp;catch.&lt;/p&gt;
&lt;/div&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-mdo-actually-is"&gt;What mdo Actually&amp;nbsp;Is&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#enabling-mac_do"&gt;Enabling mac_do&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#loader-configuration"&gt;Loader&amp;nbsp;configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rule-configuration"&gt;Rule&amp;nbsp;configuration&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-rule-language"&gt;The Rule Language&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#persisting-versus-live"&gt;Persisting versus&amp;nbsp;live&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#day-to-day-use"&gt;Day-to-Day&amp;nbsp;Use&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#where-mdo-is-not-the-right-tool"&gt;Where mdo Is Not the Right&amp;nbsp;Tool&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#optional-the-hardening-block-around-it"&gt;Optional: The Hardening Block Around&amp;nbsp;It&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#wrap"&gt;Wrap&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;Almost every FreeBSD install I touch grows&amp;nbsp;a &lt;code&gt;security/sudo&lt;/code&gt; or &lt;code&gt;security/doas&lt;/code&gt; package within the first ten minutes. It is reflex at this point. The pattern is: install base, create an admin&amp;nbsp;user, &lt;code&gt;pkg install sudo&lt;/code&gt;, drop a file&amp;nbsp;in &lt;code&gt;sudoers.d&lt;/code&gt;, move&amp;nbsp;on.&lt;/p&gt;
&lt;p&gt;Modern FreeBSD quietly makes that habit&amp;nbsp;unnecessary. &lt;code&gt;mdo(1)&lt;/code&gt; and its backing policy&amp;nbsp;module &lt;code&gt;mac_do(4)&lt;/code&gt; are in the base system. There is no package to install, no extra repository to trust, no second parser to keep current. The whole thing is one kernel module, one sysctl with a rule string, and a tiny userland command. On the hosts I have migrated, it has&amp;nbsp;replaced &lt;code&gt;sudo&lt;/code&gt; entirely.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;mdo(1)&lt;/code&gt; first appeared in FreeBSD 14.2; the group-related and fine-grained credential controls used below are FreeBSD 15.0-era functionality. If you are on 14.2 or 14.3, check the local man pages first:&amp;nbsp;basic &lt;code&gt;mdo&lt;/code&gt; use exists there, but several of the 15.0 rule constructs shown below may not&amp;nbsp;parse.&lt;/p&gt;
&lt;p&gt;It is also surprisingly&amp;nbsp;little-known. &lt;code&gt;mac_do&lt;/code&gt; shipped with the &lt;span class="caps"&gt;MAC&lt;/span&gt; framework rework, but most write-ups about FreeBSD privilege escalation still assume you reach&amp;nbsp;for &lt;code&gt;sudo&lt;/code&gt;. This article is the walkthrough I wish I had read before flipping over the first box: how to enable it, how the rule language actually works, the patterns I use day to day, and a short closing detour into the hardening sysctls that sit next to&amp;nbsp;the &lt;code&gt;mac_do&lt;/code&gt; rule on my&amp;nbsp;systems.&lt;/p&gt;
&lt;p&gt;Everything below is on FreeBSD 15.0. If you are on an older release,&amp;nbsp;check &lt;code&gt;MAC_DO(4)&lt;/code&gt; first, because the rule grammar tightened up between&amp;nbsp;versions.&lt;/p&gt;
&lt;h2 id="what-mdo-actually-is"&gt;What mdo Actually&amp;nbsp;Is&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;mdo&lt;/code&gt; is, on the surface, very boring. You run it, the kernel checks the&amp;nbsp;active &lt;code&gt;mac_do&lt;/code&gt; ruleset for whether your &lt;span class="caps"&gt;UID&lt;/span&gt; is allowed to become the requested target, and if it is, your process gets its credentials swapped in place. There is no setuid binary in the middle, no separate daemon, no &lt;span class="caps"&gt;PAM&lt;/span&gt; stack, no per-command policy file. The decision lives in the&amp;nbsp;kernel.&lt;/p&gt;
&lt;p&gt;Compared&amp;nbsp;to &lt;code&gt;sudo&lt;/code&gt;, the practical differences&amp;nbsp;are:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;No &lt;code&gt;sudoers&lt;/code&gt; parser. Rules are a single sysctl&amp;nbsp;string.&lt;/li&gt;
&lt;li&gt;No setuid root binary on disk to audit, patch, or worry&amp;nbsp;about. &lt;code&gt;mdo&lt;/code&gt; is not setuid; the credential swap is a kernel operation gated by the &lt;span class="caps"&gt;MAC&lt;/span&gt;&amp;nbsp;framework.&lt;/li&gt;
&lt;li&gt;No per-command&amp;nbsp;allowlist. &lt;code&gt;mac_do&lt;/code&gt; is about &lt;em&gt;which identities can be assumed by which users&lt;/em&gt;, not &lt;em&gt;which programs they may run&lt;/em&gt;. If you want command-level control, you still&amp;nbsp;want &lt;code&gt;sudo&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;No password prompt by default. Authentication is identity-based: your &lt;span class="caps"&gt;UID&lt;/span&gt; is on the source side of a rule or it is not. On a workstation that is jarring; on a server with no interactive logins except via &lt;span class="caps"&gt;SSH&lt;/span&gt; keys, it is&amp;nbsp;correct.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That last bullet is the most important one to&amp;nbsp;internalise. &lt;code&gt;mdo&lt;/code&gt; is&amp;nbsp;not &lt;code&gt;sudo&lt;/code&gt; with a different parser. It is a different model. It assumes the gate is at the &lt;span class="caps"&gt;SSH&lt;/span&gt; layer, not at the&amp;nbsp;shell.&lt;/p&gt;
&lt;p&gt;A first run looks exactly like the user described&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chofstede@nethost:~ $ mdo
root@nethost:~ #
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;No password, no banner, no log line&amp;nbsp;in &lt;code&gt;/var/log/messages&lt;/code&gt; or &lt;code&gt;/var/log/security&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;A consequence worth stating plainly: treat every account allowed&amp;nbsp;by &lt;code&gt;mac_do&lt;/code&gt; as effectively privileged. If an attacker gets that user&amp;#8217;s &lt;span class="caps"&gt;SSH&lt;/span&gt; key or&amp;nbsp;shell, &lt;code&gt;mdo&lt;/code&gt; will not provide a second authentication barrier. The trade is intentional, but it has to be reflected in how you manage the source-side accounts (key hygiene, no shared logins, no interactive password fallback on &lt;span class="caps"&gt;SSH&lt;/span&gt;).&lt;/p&gt;
&lt;h2 id="enabling-mac_do"&gt;Enabling&amp;nbsp;mac_do&lt;/h2&gt;
&lt;p&gt;Two pieces have to be in place: the kernel module loaded at boot, and a rule string set via sysctl. Neither survives a reboot unless you put it in the right config file, so do both at&amp;nbsp;once.&lt;/p&gt;
&lt;h3 id="loader-configuration"&gt;Loader&amp;nbsp;configuration&lt;/h3&gt;
&lt;p&gt;On the host&amp;nbsp;above, &lt;code&gt;/boot/loader.conf&lt;/code&gt; looks like this. The relevant line&amp;nbsp;is &lt;code&gt;mac_do_load="YES"&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;aesni_load=&amp;quot;YES&amp;quot;
cc_htcp_load=&amp;quot;YES&amp;quot;
crypto_load=&amp;quot;YES&amp;quot;
cryptodev_load=&amp;quot;YES&amp;quot;
virtio_random_load=&amp;quot;YES&amp;quot;
mac_do_load=&amp;quot;YES&amp;quot;         # &amp;lt;---- HERE
kern.geom.label.disk_ident.enable=&amp;quot;0&amp;quot;
kern.geom.label.gptid.enable=&amp;quot;0&amp;quot;
zfs_load=&amp;quot;YES&amp;quot;

# Jail accounting
kern.racct.enable=1

# Disable uart
hint.uart.0.disabled=&amp;quot;1&amp;quot;
hint.uart.1.disabled=&amp;quot;1&amp;quot;

security.bsd.allow_destructive_dtrace=0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few things worth flagging in passing, because they are habits I keep across every&amp;nbsp;box:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;aesni_load&lt;/code&gt;, &lt;code&gt;cryptodev_load&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;crypto_load&lt;/code&gt; give you accelerated &lt;span class="caps"&gt;AES&lt;/span&gt; and the in-kernel crypto framework. Useful&amp;nbsp;for &lt;code&gt;geli&lt;/code&gt;, IPsec, and anything that&amp;nbsp;pulls &lt;code&gt;/dev/crypto&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;cc_htcp_load&lt;/code&gt; plus the&amp;nbsp;matching &lt;code&gt;net.inet.tcp.cc.algorithm=htcp&lt;/code&gt; further down switches the default &lt;span class="caps"&gt;TCP&lt;/span&gt; congestion control to H-&lt;span class="caps"&gt;TCP&lt;/span&gt;, which behaves better than NewReno on long fat pipes. Not required, but it is on every server I&amp;nbsp;run.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;virtio_random_load&lt;/code&gt; matters on VMs: it exposes the host&amp;#8217;s &lt;span class="caps"&gt;RNG&lt;/span&gt; to the guest so the kernel does not have to wait on entropy at&amp;nbsp;boot.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;kern.racct.enable=1&lt;/code&gt; turns on resource accounting, which is&amp;nbsp;what &lt;code&gt;rctl&lt;/code&gt; and per-jail accounting depend on. Worth doing on any jail host, even if you do not yet enforce&amp;nbsp;limits.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;security.bsd.allow_destructive_dtrace=0&lt;/code&gt; keeps DTrace strictly observational. The default already nudges this way; I make it explicit so a future me cannot footgun a production box&amp;nbsp;with &lt;code&gt;dtrace -w&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Once &lt;code&gt;mac_do_load="YES"&lt;/code&gt; is in place,&amp;nbsp;a &lt;code&gt;kldstat&lt;/code&gt; after reboot confirms the module is&amp;nbsp;in:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@nethost:~ # kldstat
Id Refs Address                Size Name
 1   56 0xffffffff80200000  1f4dad0 kernel
 2    1 0xffffffff8214e000     4120 virtio_random.ko
 3    1 0xffffffff82153000     3618 cc_htcp.ko
 4    1 0xffffffff82157000   620c10 zfs.ko
 5    1 0xffffffff82778000     8808 cryptodev.ko
 6    1 0xffffffff82783000     88c8 mac_do.ko
[...]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you do not want to reboot to&amp;nbsp;test, &lt;code&gt;kldload mac_do&lt;/code&gt; will load it live and you can immediately set a rule&amp;nbsp;via &lt;code&gt;sysctl&lt;/code&gt;. Just remember to&amp;nbsp;put &lt;code&gt;mac_do_load="YES"&lt;/code&gt; in &lt;code&gt;loader.conf&lt;/code&gt; afterwards or the next reboot will quietly disarm the whole&amp;nbsp;thing.&lt;/p&gt;
&lt;h3 id="rule-configuration"&gt;Rule&amp;nbsp;configuration&lt;/h3&gt;
&lt;p&gt;Rules live&amp;nbsp;under &lt;code&gt;security.mac.do.rules&lt;/code&gt;. On the box this article is built around, the relevant chunk&amp;nbsp;of &lt;code&gt;/etc/sysctl.conf&lt;/code&gt; is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# mac_do configuration
security.mac.do.rules=&amp;quot;uid=1001&amp;gt;uid=0,gid=*,+gid=*&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That one line is the entire policy. Let us pick it&amp;nbsp;apart.&lt;/p&gt;
&lt;h2 id="the-rule-language"&gt;The Rule&amp;nbsp;Language&lt;/h2&gt;
&lt;p&gt;Each rule has two halves separated&amp;nbsp;by &lt;code&gt;&amp;gt;&lt;/code&gt;: the &lt;em&gt;from&lt;/em&gt; side, and the &lt;em&gt;to&lt;/em&gt; side. The from side matches the calling process&amp;#8217;s credentials. The to side describes what the caller is allowed to ask&amp;nbsp;for.&lt;/p&gt;
&lt;p&gt;The example above reads roughly&amp;nbsp;as:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;A process with &lt;span class="caps"&gt;UID&lt;/span&gt; 1001 may become &lt;span class="caps"&gt;UID&lt;/span&gt; 0, with any primary &lt;span class="caps"&gt;GID&lt;/span&gt;, and may keep or add any supplementary&amp;nbsp;groups.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Broken&amp;nbsp;out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;uid=1001&lt;/code&gt; is the from clause. Only my admin user&amp;nbsp;matches.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;uid=0&lt;/code&gt; is the requested target &lt;span class="caps"&gt;UID&lt;/span&gt;. Becoming&amp;nbsp;root.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;gid=*&lt;/code&gt; allows any &lt;em&gt;primary&lt;/em&gt; group &lt;span class="caps"&gt;ID&lt;/span&gt; on the target side. Without an explicit&amp;nbsp;target-side &lt;code&gt;gid=&lt;/code&gt; clause, &lt;code&gt;mac_do&lt;/code&gt; falls back to its default group-preservation rules, which is usually not what you want for a root-style&amp;nbsp;transition.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;+gid=*&lt;/code&gt; allows any &lt;em&gt;supplementary&lt;/em&gt; groups on the target side. Together&amp;nbsp;with &lt;code&gt;gid=*&lt;/code&gt;, this lifts all group-side constraints from the transition, so the new credentials can hold any group set the caller asks for. That is what you usually want when becoming&amp;nbsp;root.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Multiple rules can be concatenated&amp;nbsp;with &lt;code&gt;;&lt;/code&gt;. A more layered policy might look&amp;nbsp;like:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;security.mac.do.rules=&amp;quot;uid=1001&amp;gt;uid=0,gid=*,+gid=*;gid=0&amp;gt;uid=0,gid=*,+gid=*&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That allows my &lt;span class="caps"&gt;UID&lt;/span&gt; 1001 personally, and additionally any member of &lt;span class="caps"&gt;GID&lt;/span&gt; 0&amp;nbsp;(&lt;code&gt;wheel&lt;/code&gt;), to escalate to root. On boxes with multiple admins this is more maintainable than listing&amp;nbsp;UIDs.&lt;/p&gt;
&lt;p&gt;One sharp edge worth flagging, because I got it wrong in an earlier draft of this file and a reader caught it: the rule grammar takes &lt;em&gt;numeric&lt;/em&gt; IDs only.&amp;nbsp;Writing &lt;code&gt;gid=wheel&lt;/code&gt; looks reasonable and is what&amp;nbsp;every &lt;code&gt;sudoers&lt;/code&gt; reflex wants to type, but the kernel rejects&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl: security.mac.do.rules=gid=wheel&amp;gt;uid=0,gid=*,+gid=*: Invalid argument
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Use &lt;code&gt;gid=0&lt;/code&gt; for &lt;code&gt;wheel&lt;/code&gt;, and&amp;nbsp;check &lt;code&gt;/etc/group&lt;/code&gt; for any other group you want to reference. The parser does not resolve&amp;nbsp;names.&lt;/p&gt;
&lt;p&gt;You can also constrain the target much more tightly. To allow a deploy user to act as the application&amp;nbsp;user &lt;code&gt;app&lt;/code&gt; and nothing&amp;nbsp;else:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;security.mac.do.rules=&amp;quot;uid=2001&amp;gt;uid=1500,gid=1500,+gid=1500&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Here &lt;code&gt;2001&lt;/code&gt; is the deploy&amp;nbsp;account, &lt;code&gt;1500&lt;/code&gt; is the application&amp;#8217;s &lt;span class="caps"&gt;UID&lt;/span&gt; and &lt;span class="caps"&gt;GID&lt;/span&gt;, and no wildcards are present, so the deploy user can&amp;nbsp;become &lt;code&gt;app&lt;/code&gt; and&amp;nbsp;only &lt;code&gt;app&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The full grammar is&amp;nbsp;in &lt;code&gt;MAC_DO(4)&lt;/code&gt;. It is worth reading once cover to cover; the man page is genuinely good and includes the corner cases around supplementary groups, the meaning of&amp;nbsp;bare &lt;code&gt;gid=&lt;/code&gt; versus &lt;code&gt;+gid=&lt;/code&gt;, and how multiple rules compose. The summary I keep in my&amp;nbsp;head:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;From&amp;nbsp;side: &lt;code&gt;uid=&lt;/code&gt; and/or &lt;code&gt;gid=&lt;/code&gt; identifies &lt;em&gt;who can ask&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;To&amp;nbsp;side: &lt;code&gt;uid=&lt;/code&gt;, &lt;code&gt;gid=&lt;/code&gt;, and&amp;nbsp;optional &lt;code&gt;+gid=&lt;/code&gt; describes &lt;em&gt;what they may become&lt;/em&gt;.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;*&lt;/code&gt; means&amp;nbsp;any.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;+&lt;/code&gt; on the to side means &amp;#8220;may gain&amp;#8221;, as opposed to &amp;#8220;must match&amp;nbsp;exactly&amp;#8221;.&lt;/li&gt;
&lt;li&gt;Rules are effectively ORed together: if any rule authorises the requested credential transition, it is allowed. There is no allow/deny ordering to worry about, just a union of&amp;nbsp;permissions.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="persisting-versus-live"&gt;Persisting versus&amp;nbsp;live&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;/etc/sysctl.conf&lt;/code&gt; applies at boot. To change the policy without&amp;nbsp;rebooting:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;security.mac.do.rules&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;uid=1001&amp;gt;uid=0,gid=*,+gid=*&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The kernel parses the string immediately, and any failure to parse is reported back as&amp;nbsp;an &lt;code&gt;EINVAL&lt;/code&gt; from&amp;nbsp;the &lt;code&gt;sysctl&lt;/code&gt; call, with no partial state left behind. That makes it safe to iterate: a syntactically broken rule string is simply rejected, and the previous ruleset stays in&amp;nbsp;effect.&lt;/p&gt;
&lt;p&gt;To confirm what the kernel actually has&amp;nbsp;loaded:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;security.mac.do.rules
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Useful as a sanity check after editing, before you reach&amp;nbsp;for &lt;code&gt;mdo&lt;/code&gt; and discover it does&amp;nbsp;nothing.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;mac_do&lt;/code&gt; rules are also&amp;nbsp;jail-aware; &lt;code&gt;security.mac.do.rules&lt;/code&gt; is a jail-specific sysctl, so a rule set in the host is not automatically the same as the rule visible inside a jail. If you plan to&amp;nbsp;use &lt;code&gt;mdo&lt;/code&gt; inside jails, read the jail support section&amp;nbsp;of &lt;code&gt;mac_do(4)&lt;/code&gt; rather than assuming the host rule applies&amp;nbsp;everywhere.&lt;/p&gt;
&lt;h2 id="day-to-day-use"&gt;Day-to-Day&amp;nbsp;Use&lt;/h2&gt;
&lt;p&gt;On the hosts I have flipped&amp;nbsp;over, &lt;code&gt;mdo&lt;/code&gt; covers every case I used to&amp;nbsp;use &lt;code&gt;sudo&lt;/code&gt; for, with two&amp;nbsp;patterns:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Open a root shell&lt;/span&gt;
mdo

&lt;span class="c1"&gt;# Run a single command as root&lt;/span&gt;
mdo&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;By&amp;nbsp;default, &lt;code&gt;mdo&lt;/code&gt; behaves as&amp;nbsp;if &lt;code&gt;-u root&lt;/code&gt; had been specified,&amp;nbsp;so &lt;code&gt;mdo pkg upgrade&lt;/code&gt; requests a transition to &lt;span class="caps"&gt;UID&lt;/span&gt; 0. Other targets are available&amp;nbsp;with &lt;code&gt;-u&lt;/code&gt;, &lt;code&gt;-g&lt;/code&gt;, &lt;code&gt;-G&lt;/code&gt;, and the related flags described&amp;nbsp;in &lt;code&gt;mdo(1)&lt;/code&gt;, but the transition itself still has to be authorised by the&amp;nbsp;active &lt;code&gt;mac_do&lt;/code&gt; ruleset. In other words, the ruleset authorises &lt;em&gt;requested credentials&lt;/em&gt;; &lt;code&gt;mdo&lt;/code&gt; is not a sudoers-style command policy and does not pick a target on its&amp;nbsp;own.&lt;/p&gt;
&lt;p&gt;Behaviourally:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The new shell inherits the environment of the caller. That is closer&amp;nbsp;to &lt;code&gt;sudo -E&lt;/code&gt; than&amp;nbsp;to &lt;code&gt;sudo -i&lt;/code&gt;. If you want a clean login environment,&amp;nbsp;run &lt;code&gt;mdo -i&lt;/code&gt; (and yes, the flag exists;&amp;nbsp;check &lt;code&gt;mdo(1)&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;&lt;code&gt;PATH&lt;/code&gt; is preserved unless you do something about it. On a server this is fine; on anything where the&amp;nbsp;caller&amp;#8217;s &lt;code&gt;PATH&lt;/code&gt; is not trusted, you would want a wrapper that resets&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;There is no &lt;span class="caps"&gt;TTY&lt;/span&gt; ticket cache. Every invocation is independent. Annoying in interactive use, excellent for&amp;nbsp;scripts.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For Ansible-style&amp;nbsp;automation, &lt;code&gt;mdo&lt;/code&gt; is a become method by way&amp;nbsp;of &lt;code&gt;become_method: su&lt;/code&gt; with a tweaked command, or you can write a tiny custom become plugin. I have been using a wrapper script that&amp;nbsp;just &lt;code&gt;exec&lt;/code&gt;s &lt;code&gt;mdo&lt;/code&gt;, which works well enough for my fleet. A proper Ansible plugin is on the list, but the lack of an interactive password prompt means existing methods already cover most&amp;nbsp;cases.&lt;/p&gt;
&lt;h2 id="where-mdo-is-not-the-right-tool"&gt;Where mdo Is Not the Right&amp;nbsp;Tool&lt;/h2&gt;
&lt;p&gt;To be clear about the&amp;nbsp;trade-offs:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;If you need per-command policy (&amp;#8220;alice may&amp;nbsp;run &lt;code&gt;service nginx restart&lt;/code&gt; and nothing&amp;nbsp;else&amp;#8221;), &lt;code&gt;mac_do&lt;/code&gt; cannot express that. Stick&amp;nbsp;with &lt;code&gt;sudo&lt;/code&gt; or use &lt;span class="caps"&gt;OS&lt;/span&gt;-level packaging (a small setuid wrapper for the one operation, audited&amp;nbsp;carefully).&lt;/li&gt;
&lt;li&gt;If you need interactive password&amp;nbsp;authentication, &lt;code&gt;mdo&lt;/code&gt; does not give it to you. That is a deliberate model choice; the gate is meant to be at &lt;span class="caps"&gt;SSH&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;If you need extensive logging of &lt;em&gt;what&lt;/em&gt; was&amp;nbsp;run, &lt;code&gt;sudo&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s session logging is still&amp;nbsp;ahead. &lt;code&gt;mac_do&lt;/code&gt; logs the credential transition to the kernel log, but not the command line. Pair it&amp;nbsp;with &lt;code&gt;auditd&lt;/code&gt; and a&amp;nbsp;sensible &lt;code&gt;audit_control&lt;/code&gt; policy if you need command&amp;nbsp;auditing.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For multi-tenant systems, mixed-trust environments, or anything where you need fine-grained command-level policy and rotating&amp;nbsp;credentials, &lt;code&gt;sudo&lt;/code&gt; is still the right answer. For a single-admin server with &lt;span class="caps"&gt;SSH&lt;/span&gt;-key access and a clean separation between an admin account and&amp;nbsp;root, &lt;code&gt;mdo&lt;/code&gt; is hard to argue&amp;nbsp;with.&lt;/p&gt;
&lt;h2 id="optional-the-hardening-block-around-it"&gt;Optional: The Hardening Block Around&amp;nbsp;It&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;mac_do&lt;/code&gt; rule lives&amp;nbsp;in &lt;code&gt;/etc/sysctl.conf&lt;/code&gt; alongside a handful of other settings I treat as &amp;#8220;always on&amp;#8221;. They are not strictly&amp;nbsp;about &lt;code&gt;mdo&lt;/code&gt;, but they are the surrounding context of a hardened FreeBSD host, and they are cheap. The full file on this host&amp;nbsp;is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0
kern.randompid=1
vfs.zfs.vdev.min_auto_ashift=12

kern.elf64.aslr.enable=1
kern.elf32.aslr.enable=1

net.inet.tcp.blackhole=2
net.inet.udp.blackhole=1
net.inet.icmp.drop_redirect=1
net.inet.tcp.drop_synfin=1

net.inet.tcp.cc.algorithm=htcp

net.link.bridge.pfil_member=1
net.link.bridge.pfil_bridge=0
net.link.bridge.pfil_onlyip=1

# mac_do configuration
security.mac.do.rules=&amp;quot;uid=1001&amp;gt;uid=0,gid=*,+gid=*&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Quickly, line by line, because each of these has saved me from at least one bad&amp;nbsp;day:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;security.bsd.see_other_uids=0&lt;/code&gt; and &lt;code&gt;see_other_gids=0&lt;/code&gt; hide processes owned by other users&amp;nbsp;from &lt;code&gt;ps&lt;/code&gt;, &lt;code&gt;top&lt;/code&gt;, and friends. On a shared host or jail host this is the single highest-value privacy&amp;nbsp;toggle.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;see_jail_proc=0&lt;/code&gt; hides jailed processes from the host listing for non-root callers. Useful when you have a &amp;#8220;support&amp;#8221; user on the host that should not be able to enumerate tenant&amp;nbsp;workloads.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;unprivileged_read_msgbuf=0&lt;/code&gt; keeps &lt;code&gt;dmesg&lt;/code&gt; out of unprivileged hands. The kernel ring buffer leaks more than you would think, especially on&amp;nbsp;boot.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;unprivileged_proc_debug=0&lt;/code&gt; blocks unprivileged use&amp;nbsp;of &lt;code&gt;ptrace&lt;/code&gt; and friends. This is the same idea as Linux&amp;#8217;s&amp;nbsp;Yama &lt;code&gt;ptrace_scope=1&lt;/code&gt;, and it stops one whole class of credential-stealing&amp;nbsp;primitive.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;kern.randompid=1&lt;/code&gt; randomises &lt;span class="caps"&gt;PID&lt;/span&gt; allocation. Cheap, makes a small but real class of timing attacks&amp;nbsp;harder.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;kern.elf64.aslr.enable=1&lt;/code&gt; and &lt;code&gt;kern.elf32.aslr.enable=1&lt;/code&gt; turn on &lt;span class="caps"&gt;ASLR&lt;/span&gt; for &lt;span class="caps"&gt;ELF&lt;/span&gt; binaries. On FreeBSD 15 the defaults are already friendly, but I set them explicitly so they cannot regress under&amp;nbsp;me.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;net.inet.tcp.blackhole=2&lt;/code&gt; and &lt;code&gt;udp.blackhole=1&lt;/code&gt; make the host stop sending &lt;span class="caps"&gt;RST&lt;/span&gt;/&lt;span class="caps"&gt;ICMP&lt;/span&gt;-unreachable on closed ports. Scanners get silence instead of confirmation. Easy win against opportunistic&amp;nbsp;scans.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;net.inet.icmp.drop_redirect=1&lt;/code&gt; drops &lt;span class="caps"&gt;ICMP&lt;/span&gt; redirect packets, which an attacker on the local segment could otherwise use to repoint your&amp;nbsp;routes.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;net.inet.tcp.drop_synfin=1&lt;/code&gt; drops&amp;nbsp;malformed &lt;code&gt;SYN|FIN&lt;/code&gt; packets, an old fingerprinting and evasion&amp;nbsp;trick.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;net.link.bridge.pfil_*&lt;/code&gt; causes &lt;code&gt;pf&lt;/code&gt; to filter traffic on bridge member interfaces, which is what you want as soon as you put jails on a&amp;nbsp;bridge.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;code&gt;kern.securelevel&lt;/code&gt; is the one knob from this family that is &lt;em&gt;not&lt;/em&gt; in this file, deliberately. On a server that I still want to be able to maintain remotely, I&amp;nbsp;keep &lt;code&gt;securelevel&lt;/code&gt; at the default. On appliance-style boxes (&lt;span class="caps"&gt;DNS&lt;/span&gt; resolvers, a small router) I bump it&amp;nbsp;via &lt;code&gt;/etc/rc.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;kern_securelevel_enable=&amp;quot;YES&amp;quot;
kern_securelevel=&amp;quot;2&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;At level 2, no filesystem may be mounted &amp;#8220;less&amp;nbsp;secure&amp;#8221;, &lt;code&gt;schg&lt;/code&gt; flags become immutable, and the system clock can only move forward. It is an excellent lockdown for boxes you do not log into often, and a foot-cannon for everything&amp;nbsp;else.&lt;/p&gt;
&lt;h2 id="wrap"&gt;Wrap&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;mdo&lt;/code&gt; is small. The userland command is a thin wrapper, the policy is a single sysctl string, and the kernel module is unobtrusive. That smallness is exactly why it has&amp;nbsp;displaced &lt;code&gt;sudo&lt;/code&gt; on most of my fleet: there is less to configure, less to keep up to date, and less to get&amp;nbsp;wrong.&lt;/p&gt;
&lt;p&gt;The trade-off is&amp;nbsp;real. &lt;code&gt;mac_do&lt;/code&gt; is identity-based and intentionally has no per-command policy or interactive auth. If your model needs those,&amp;nbsp;keep &lt;code&gt;sudo&lt;/code&gt;. If your model already treats &lt;span class="caps"&gt;SSH&lt;/span&gt; as the gate and root as a credential the admin account is allowed to&amp;nbsp;assume, &lt;code&gt;mdo&lt;/code&gt; is a near-perfect fit and one fewer port to&amp;nbsp;track.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;MAC_DO(4)&lt;/code&gt; is the resource to read next; the rule grammar has more depth than the examples above show, particularly&amp;nbsp;around &lt;code&gt;gid=&lt;/code&gt; semantics and how multiple rules compose. Set up a test rule, kick it&amp;nbsp;with &lt;code&gt;sysctl&lt;/code&gt;, and see how it behaves before committing it&amp;nbsp;to &lt;code&gt;sysctl.conf&lt;/code&gt;. The kernel will reject anything malformed without applying it, so iteration is genuinely&amp;nbsp;safe.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="security"/><category term="mac"/><category term="mac_do"/><category term="privilege-delegation"/><category term="sudo"/><category term="doas"/><category term="hardening"/><category term="sysctl"/></entry><entry><title>FreeBSD Resource Monitoring, Accounting, and Troubleshooting</title><link href="https://hofstede.it/freebsd-resource-monitoring-accounting-and-troubleshooting/" rel="alternate"/><published>2026-05-12T00:00:00+02:00</published><updated>2026-05-12T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-05-12:/freebsd-resource-monitoring-accounting-and-troubleshooting/</id><summary type="html">&lt;p&gt;A practical tour of FreeBSD&amp;#8217;s built-in tools for watching &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, disk I/O, and network activity, finding the process or jail that is eating your server, and applying per-jail accounting and limits with kern.racct and&amp;nbsp;rctl.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;The server feels slow.&amp;#8221; It is the most common ticket text in the world, and the least useful. Before you can fix anything you need to know whether the box is &lt;span class="caps"&gt;CPU&lt;/span&gt; bound, memory pressured, waiting on disk, saturating a &lt;span class="caps"&gt;NIC&lt;/span&gt;, or simply running a runaway process in one jail that is starving everyone else. On jail hosts, the second half of the problem is attribution: not just what is overloaded, but which jail is&amp;nbsp;responsible.&lt;/p&gt;
&lt;p&gt;FreeBSD ships with an unusually complete troubleshooting toolkit out of the box. This article is a practical tour: how to watch &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, disk, and network in real time, how to drill from a host-level symptom down to the process responsible, and how to&amp;nbsp;use &lt;code&gt;kern.racct&lt;/code&gt; and &lt;code&gt;rctl&lt;/code&gt; to both &lt;em&gt;measure&lt;/em&gt; and &lt;em&gt;cap&lt;/em&gt; what individual jails consume. A short closing section points&amp;nbsp;at &lt;code&gt;jail_exporter&lt;/code&gt; for those who want long-term Prometheus data, but the focus here is the in-box troubleshooting&amp;nbsp;toolkit.&lt;/p&gt;
&lt;p&gt;This article intentionally sticks to tools present in the FreeBSD base system. There is nothing wrong&amp;nbsp;with &lt;code&gt;htop&lt;/code&gt; from ports, but every command below works on a freshly installed host with no extra packages. For the lower-level cases that this toolkit cannot&amp;nbsp;reach, &lt;code&gt;dtrace&lt;/code&gt; and &lt;code&gt;pmcstat&lt;/code&gt; are the next step, and are the subject of a follow-up&amp;nbsp;article.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#cpu-and-load"&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt; and Load&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#top"&gt;top&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#uptime-w-and-load-average"&gt;uptime, w, and load&amp;nbsp;average&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#vmstat"&gt;vmstat&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#pause-systat-as-a-live-dashboard"&gt;Pause: systat as a Live&amp;nbsp;Dashboard&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#memory"&gt;Memory&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#swap"&gt;Swap&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#vmstat-z-uma-zones"&gt;vmstat -z: &lt;span class="caps"&gt;UMA&lt;/span&gt;&amp;nbsp;zones&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#disk-io"&gt;Disk I/O&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#iostat"&gt;iostat&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#gstat"&gt;gstat&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#zfs"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#top-m-io"&gt;top -m&amp;nbsp;io&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#network"&gt;Network&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#interface-basics"&gt;Interface&amp;nbsp;basics&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#live-throughput"&gt;Live&amp;nbsp;throughput&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#sockets-and-connections"&gt;Sockets and&amp;nbsp;connections&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#protocol-stats"&gt;Protocol&amp;nbsp;stats&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#routes"&gt;Routes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#pf-firewall-state-rules-and-counters"&gt;pf: firewall state, rules, and&amp;nbsp;counters&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#live-network-views"&gt;Live network&amp;nbsp;views&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#packet-capture"&gt;Packet&amp;nbsp;capture&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#per-process-inspection"&gt;Per-Process&amp;nbsp;Inspection&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#finding-bottlenecks-a-workflow"&gt;Finding Bottlenecks: A&amp;nbsp;Workflow&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#per-jail-accounting-and-limits-with-kernracct"&gt;Per-Jail Accounting and Limits with kern.racct&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#enabling-racct"&gt;Enabling&amp;nbsp;racct&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#reading-per-jail-usage"&gt;Reading per-jail&amp;nbsp;usage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#setting-limits"&gt;Setting&amp;nbsp;limits&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#persisting-rules"&gt;Persisting&amp;nbsp;rules&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-practical-caution-about-deny"&gt;A practical caution about&amp;nbsp;deny&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#seeing-jails-in-top"&gt;Seeing jails in&amp;nbsp;top&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-note-on-jail_exporter-and-prometheus"&gt;A Note on jail_exporter and&amp;nbsp;Prometheus&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#wrap-up"&gt;Wrap-up&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="cpu-and-load"&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt; and&amp;nbsp;Load&lt;/h2&gt;
&lt;h3 id="top"&gt;top&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;top&lt;/code&gt; is the workhorse. A handful of flags turn it from a curiosity into a real diagnostic&amp;nbsp;tool:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Per-CPU breakdown at the top of the screen&lt;/span&gt;
top&lt;span class="w"&gt; &lt;/span&gt;-P

&lt;span class="c1"&gt;# Show threads, not just processes (useful for multi-threaded daemons)&lt;/span&gt;
top&lt;span class="w"&gt; &lt;/span&gt;-H

&lt;span class="c1"&gt;# Add a JID column so you can see which jail each process belongs to&lt;/span&gt;
&lt;span class="c1"&gt;# (available on modern FreeBSD releases; older systems may not have it)&lt;/span&gt;
top&lt;span class="w"&gt; &lt;/span&gt;-j

&lt;span class="c1"&gt;# Switch to I/O view instead of CPU&lt;/span&gt;
top&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;io

&lt;span class="c1"&gt;# Include system (kernel) processes such as idle, intr, geom&lt;/span&gt;
top&lt;span class="w"&gt; &lt;/span&gt;-S
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You can combine&amp;nbsp;flags: &lt;code&gt;top -SHPj&lt;/code&gt; gives you per-&lt;span class="caps"&gt;CPU&lt;/span&gt; stats, threads, system processes, and a &lt;span class="caps"&gt;JID&lt;/span&gt; column at once, which is what I usually run on a busy jail&amp;nbsp;host.&lt;/p&gt;
&lt;p&gt;Most of the answer is usually in the header&amp;nbsp;line:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;CPU:  4.2% user,  0.0% nice, 12.7% system,  1.1% interrupt, 82.0% idle
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;High &lt;code&gt;system&lt;/code&gt; plus&amp;nbsp;high &lt;code&gt;interrupt&lt;/code&gt; usually means kernel work: network packet processing, &lt;span class="caps"&gt;ZFS&lt;/span&gt;, or a chatty driver.&amp;nbsp;High &lt;code&gt;user&lt;/code&gt; is application code. Sustained interrupt time on a small box is often network or storage interrupt&amp;nbsp;processing; &lt;code&gt;vmstat -i&lt;/code&gt; shows which &lt;span class="caps"&gt;IRQ&lt;/span&gt; is&amp;nbsp;busy.&lt;/p&gt;
&lt;h3 id="uptime-w-and-load-average"&gt;uptime, w, and load&amp;nbsp;average&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;uptime
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;11&lt;/span&gt;:42AM&lt;span class="w"&gt;  &lt;/span&gt;up&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;87&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;days,&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;:14,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;users,&lt;span class="w"&gt; &lt;/span&gt;load&lt;span class="w"&gt; &lt;/span&gt;averages:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.84,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.71,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.55
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Load averages on FreeBSD count runnable plus uninterruptible processes, same as Linux. A 1-minute load far above the 15-minute load means something just started hammering the box; the reverse means whatever it was is winding down. Compare load&amp;nbsp;to &lt;code&gt;hw.ncpu&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;hw.ncpu
hw.ncpu:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Sustained load&amp;nbsp;over &lt;code&gt;hw.ncpu&lt;/code&gt; is your first sign you should&amp;nbsp;care.&lt;/p&gt;
&lt;h3 id="vmstat"&gt;vmstat&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;vmstat 1&lt;/code&gt; gives a one-second-resolution snapshot of &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory pressure, and context&amp;nbsp;switching:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vmstat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;procs&lt;span class="w"&gt;  &lt;/span&gt;memory&lt;span class="w"&gt;       &lt;/span&gt;page&lt;span class="w"&gt;                    &lt;/span&gt;disks&lt;span class="w"&gt;     &lt;/span&gt;faults&lt;span class="w"&gt;         &lt;/span&gt;cpu
&lt;span class="w"&gt; &lt;/span&gt;r&lt;span class="w"&gt; &lt;/span&gt;b&lt;span class="w"&gt; &lt;/span&gt;w&lt;span class="w"&gt;  &lt;/span&gt;avm&lt;span class="w"&gt;   &lt;/span&gt;fre&lt;span class="w"&gt;  &lt;/span&gt;flt&lt;span class="w"&gt;  &lt;/span&gt;re&lt;span class="w"&gt;  &lt;/span&gt;pi&lt;span class="w"&gt;  &lt;/span&gt;po&lt;span class="w"&gt;    &lt;/span&gt;fr&lt;span class="w"&gt;   &lt;/span&gt;sr&lt;span class="w"&gt; &lt;/span&gt;ad0&lt;span class="w"&gt; &lt;/span&gt;da0&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;sy&lt;span class="w"&gt;    &lt;/span&gt;cs&lt;span class="w"&gt; &lt;/span&gt;us&lt;span class="w"&gt; &lt;/span&gt;sy&lt;span class="w"&gt; &lt;/span&gt;id
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;18G&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.2G&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;421&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;612&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="m"&gt;8&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;4k&lt;span class="w"&gt;    &lt;/span&gt;8k&lt;span class="w"&gt;   &lt;/span&gt;12k&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;96&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;What to look&amp;nbsp;at:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;r&lt;/code&gt; (runnable): sustained values&amp;nbsp;above &lt;code&gt;hw.ncpu&lt;/code&gt; mean &lt;span class="caps"&gt;CPU&lt;/span&gt;&amp;nbsp;saturation&lt;/li&gt;
&lt;li&gt;&lt;code&gt;b&lt;/code&gt; (blocked): processes sleeping in the kernel waiting on resources, often disk or filesystem&amp;nbsp;I/O&lt;/li&gt;
&lt;li&gt;&lt;code&gt;pi&lt;/code&gt;/&lt;code&gt;po&lt;/code&gt; (page in/out from swap): non-zero values mean you are actually paging, which is almost always&amp;nbsp;bad&lt;/li&gt;
&lt;li&gt;&lt;code&gt;cs&lt;/code&gt; (context switches per second): unusually high values indicate lock contention or a misbehaving event&amp;nbsp;loop&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="pause-systat-as-a-live-dashboard"&gt;Pause: systat as a Live&amp;nbsp;Dashboard&lt;/h2&gt;
&lt;p&gt;Before diving into memory, disk, and network in isolation, it is worth&amp;nbsp;introducing &lt;code&gt;systat&lt;/code&gt;. It is a curses-based dashboard built into the base system, with a subview for almost every subsystem covered&amp;nbsp;below:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systat&lt;span class="w"&gt; &lt;/span&gt;-vmstat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="c1"&gt;# CPU, memory, paging, disk, top processes, all on one screen&lt;/span&gt;
systat&lt;span class="w"&gt; &lt;/span&gt;-iostat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
systat&lt;span class="w"&gt; &lt;/span&gt;-ifstat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
systat&lt;span class="w"&gt; &lt;/span&gt;-tcp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
systat&lt;span class="w"&gt; &lt;/span&gt;-netstat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
systat&lt;span class="w"&gt; &lt;/span&gt;-swap&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
systat&lt;span class="w"&gt; &lt;/span&gt;-pigs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="c1"&gt;# top CPU consumers, condensed&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Each section below references the relevant subview. In day-to-day troubleshooting I keep a tmux window&amp;nbsp;with &lt;code&gt;systat -vmstat 1&lt;/code&gt; on the left&amp;nbsp;and &lt;code&gt;gstat -p&lt;/code&gt; on the right; the combined view tells you in under five seconds whether the problem is &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, disk, or&amp;nbsp;network.&lt;/p&gt;
&lt;h2 id="memory"&gt;Memory&lt;/h2&gt;
&lt;p&gt;FreeBSD&amp;#8217;s memory accounting does not map cleanly to &amp;#8220;free&amp;#8221; in the Linux sense.&amp;nbsp;The &lt;code&gt;top&lt;/code&gt; memory header is the most digestible&amp;nbsp;view:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Mem: 2.1G Active, 5.6G Inact, 1.8G Laundry, 14G Wired, 512M Buf, 1.0G Free
ARC: 8.4G Total, 2.1G MFU, 5.9G MRU, 12K Anon, 102M Header, 312M Other
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Active&lt;/strong&gt;: in use right&amp;nbsp;now&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Inact&lt;/strong&gt;: not recently used but still cached. Available for&amp;nbsp;reclaim&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Laundry&lt;/strong&gt;: dirty pages waiting to be written out before&amp;nbsp;reuse&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Wired&lt;/strong&gt;: kernel memory that cannot be paged out. On &lt;span class="caps"&gt;ZFS&lt;/span&gt; systems this is often dominated by &lt;span class="caps"&gt;ARC&lt;/span&gt;&amp;nbsp;allocations&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Free&lt;/strong&gt;: truly free. Usually small on a healthy box, because unused memory is wasted&amp;nbsp;memory&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;On &lt;span class="caps"&gt;ZFS&lt;/span&gt; systems the &lt;strong&gt;Wired&lt;/strong&gt; number is dominated by the &lt;span class="caps"&gt;ARC&lt;/span&gt;. That is fine. Watch the &lt;strong&gt;&lt;span class="caps"&gt;ARC&lt;/span&gt;&lt;/strong&gt; line for what it is actually caching. The &lt;span class="caps"&gt;ARC&lt;/span&gt; shrinks automatically under memory pressure; you only need to&amp;nbsp;tune &lt;code&gt;vfs.zfs.arc_max&lt;/code&gt; if you see a workload starving for &lt;span class="caps"&gt;RAM&lt;/span&gt; while &lt;span class="caps"&gt;ARC&lt;/span&gt; stays&amp;nbsp;huge.&lt;/p&gt;
&lt;h3 id="swap"&gt;Swap&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;swapinfo&lt;span class="w"&gt; &lt;/span&gt;-h
Device&lt;span class="w"&gt;          &lt;/span&gt;1K-blocks&lt;span class="w"&gt;     &lt;/span&gt;Used&lt;span class="w"&gt;    &lt;/span&gt;Avail&lt;span class="w"&gt; &lt;/span&gt;Capacity
/dev/gpt/swap0&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="m"&gt;8388604&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;142336&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;8246268&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;%

&lt;span class="c1"&gt;# equivalent&lt;/span&gt;
pstat&lt;span class="w"&gt; &lt;/span&gt;-s
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Swap usage on its own is not automatically a problem; FreeBSD will happily leave cold pages on swap indefinitely. What you actually care about is steadily &lt;em&gt;increasing&lt;/em&gt; swap usage together with active paging&amp;nbsp;(&lt;code&gt;pi&lt;/code&gt;/&lt;code&gt;po&lt;/code&gt; in &lt;code&gt;vmstat&lt;/code&gt;). When that pattern shows up,&amp;nbsp;use &lt;code&gt;top -o res&lt;/code&gt; (sort by resident size)&amp;nbsp;or &lt;code&gt;top -o size&lt;/code&gt; (virtual size) to find who is consuming&amp;nbsp;memory.&lt;/p&gt;
&lt;h3 id="vmstat-z-uma-zones"&gt;vmstat -z: &lt;span class="caps"&gt;UMA&lt;/span&gt;&amp;nbsp;zones&lt;/h3&gt;
&lt;p&gt;When the kernel itself is leaking or&amp;nbsp;oversubscribed, &lt;code&gt;vmstat -z&lt;/code&gt; shows per-zone allocations and&amp;nbsp;failures:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vmstat&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;head&lt;span class="w"&gt; &lt;/span&gt;-1&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;vmstat&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort&lt;span class="w"&gt; &lt;/span&gt;-t:&lt;span class="w"&gt; &lt;/span&gt;-k3&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;head
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A&amp;nbsp;non-zero &lt;code&gt;FAIL&lt;/code&gt; column on a zone (mbuf clusters, vnode, file descriptors) tells you exactly what kernel resource is exhausted. This is the usual diagnostic for &amp;#8220;the box stopped accepting connections but &lt;span class="caps"&gt;CPU&lt;/span&gt; is&amp;nbsp;idle.&amp;#8221;&lt;/p&gt;
&lt;h2 id="disk-io"&gt;Disk&amp;nbsp;I/O&lt;/h2&gt;
&lt;h3 id="iostat"&gt;iostat&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;iostat&lt;span class="w"&gt; &lt;/span&gt;-x&lt;span class="w"&gt; &lt;/span&gt;-w&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-x&lt;/code&gt; form gives extended stats: ops/s, &lt;span class="caps"&gt;KB&lt;/span&gt;/op, ms/op, and percent-busy per device. Anything sustained above roughly 80% busy with&amp;nbsp;high &lt;code&gt;ms/op&lt;/code&gt; on a spinning disk, or above 90% on an &lt;span class="caps"&gt;SSD&lt;/span&gt;, is a real&amp;nbsp;bottleneck.&lt;/p&gt;
&lt;h3 id="gstat"&gt;gstat&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;gstat&lt;/code&gt; is the &lt;span class="caps"&gt;GEOM&lt;/span&gt; equivalent and is usually the friendlier tool because it shows every &lt;span class="caps"&gt;GEOM&lt;/span&gt; consumer (partitions, encrypted providers, mirrors) not just raw&amp;nbsp;disks:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gstat&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="c1"&gt;# only physical providers&lt;/span&gt;
gstat&lt;span class="w"&gt; &lt;/span&gt;-I&lt;span class="w"&gt; &lt;/span&gt;1s&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ada|nvd&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# filter by name, 1-second interval&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Watch&amp;nbsp;the &lt;code&gt;%busy&lt;/code&gt; column. On modern NVMe you will rarely see it climb; on &lt;span class="caps"&gt;SATA&lt;/span&gt; SSDs and HDDs it is a good first indicator of storage&amp;nbsp;pressure.&lt;/p&gt;
&lt;h3 id="zfs"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt;&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;zpool&lt;span class="w"&gt; &lt;/span&gt;iostat&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;iostat&lt;span class="w"&gt; &lt;/span&gt;-lq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# latency and queue depth per vdev&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Per-vdev latency is what tells you whether a degraded mirror or a single slow drive is dragging the pool down. The queue length view&amp;nbsp;(&lt;code&gt;-q&lt;/code&gt;) is what catches &amp;#8220;the pool is fine on average but tail latency is&amp;nbsp;terrible.&amp;#8221;&lt;/p&gt;
&lt;h3 id="top-m-io"&gt;top -m&amp;nbsp;io&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;top&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;io
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Switches &lt;code&gt;top&lt;/code&gt; from &lt;span class="caps"&gt;CPU&lt;/span&gt; sort into I/O sort. You see per-process &lt;span class="caps"&gt;VCSW&lt;/span&gt; (voluntary ctx switches), &lt;span class="caps"&gt;IVCSW&lt;/span&gt;, &lt;span class="caps"&gt;READ&lt;/span&gt;, &lt;span class="caps"&gt;WRITE&lt;/span&gt;, &lt;span class="caps"&gt;FAULT&lt;/span&gt;, &lt;span class="caps"&gt;TOTAL&lt;/span&gt;. This is often the first place to look when you need to identify the process hammering the&amp;nbsp;disks.&lt;/p&gt;
&lt;h2 id="network"&gt;Network&lt;/h2&gt;
&lt;h3 id="interface-basics"&gt;Interface&amp;nbsp;basics&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ifconfig
ifconfig&lt;span class="w"&gt; &lt;/span&gt;-L&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="c1"&gt;# show link-local IPv6&lt;/span&gt;
netstat&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="c1"&gt;# cumulative packet and error counters per interface&lt;/span&gt;
netstat&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;-h&lt;span class="w"&gt; &lt;/span&gt;-w&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="c1"&gt;# human readable, 1-second deltas&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;netstat -i -w 1&lt;/code&gt; is usually the quickest check for &amp;#8220;is my &lt;span class="caps"&gt;NIC&lt;/span&gt; actually moving packets right now and is it dropping any?&amp;#8221;&amp;nbsp;Non-zero &lt;code&gt;Idrop&lt;/code&gt; or &lt;code&gt;Odrop&lt;/code&gt; columns are bad and worth investigating&amp;nbsp;with &lt;code&gt;sysctl dev.&amp;lt;iface&amp;gt;&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="live-throughput"&gt;Live&amp;nbsp;throughput&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systat&lt;span class="w"&gt; &lt;/span&gt;-ifstat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="c1"&gt;# per-interface bits/sec, peak, ASCII bar chart&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is&amp;nbsp;the &lt;code&gt;ifstat&lt;/code&gt; subview referenced&amp;nbsp;earlier; &lt;code&gt;systat -tcp 1&lt;/code&gt;, &lt;code&gt;systat -netstat 1&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;systat -ip 1&lt;/code&gt; give live equivalents at the transport and &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;layers.&lt;/p&gt;
&lt;h3 id="sockets-and-connections"&gt;Sockets and&amp;nbsp;connections&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sockstat&lt;span class="w"&gt; &lt;/span&gt;-4&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# IPv4 listening sockets&lt;/span&gt;
sockstat&lt;span class="w"&gt; &lt;/span&gt;-4&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# IPv4 connected sockets&lt;/span&gt;
sockstat&lt;span class="w"&gt; &lt;/span&gt;-46&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;head&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="c1"&gt;# both stacks&lt;/span&gt;
sockstat&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="c1"&gt;# TCP only&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;sockstat&lt;/code&gt; is the FreeBSD answer&amp;nbsp;to &lt;code&gt;lsof -i&lt;/code&gt; and &lt;code&gt;ss&lt;/code&gt;. It maps every socket back to the owning command, &lt;span class="caps"&gt;PID&lt;/span&gt;, and user, which is usually what you need when chasing &amp;#8220;who opened this&amp;nbsp;port.&amp;#8221;&lt;/p&gt;
&lt;p&gt;For raw connection counts and &lt;span class="caps"&gt;TCP&lt;/span&gt; state&amp;nbsp;distribution:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;netstat&lt;span class="w"&gt; &lt;/span&gt;-an&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;NR&amp;gt;2 {print $6}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;uniq&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort&lt;span class="w"&gt; &lt;/span&gt;-rn
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Lots&amp;nbsp;of &lt;code&gt;TIME_WAIT&lt;/code&gt; is normal on a busy server. Lots&amp;nbsp;of &lt;code&gt;CLOSE_WAIT&lt;/code&gt; usually means the application is not closing accepted sockets and you have a&amp;nbsp;leak.&lt;/p&gt;
&lt;h3 id="protocol-stats"&gt;Protocol&amp;nbsp;stats&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;netstat&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="c1"&gt;# TCP protocol counters since boot&lt;/span&gt;
netstat&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ip&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="c1"&gt;# IP layer&lt;/span&gt;
netstat&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="c1"&gt;# mbuf usage, the single most important kernel network counter&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;netstat -m&lt;/code&gt; shows mbuf cluster allocation and, critically, denied/delayed allocations. If those are climbing you are running out of network buffers, which manifests as mysterious connection failures while &lt;span class="caps"&gt;CPU&lt;/span&gt; and link both look idle. This is a reliable signal that gets missed by people who only watch the link&amp;nbsp;layer.&lt;/p&gt;
&lt;h3 id="routes"&gt;Routes&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;netstat&lt;span class="w"&gt; &lt;/span&gt;-rn&lt;span class="w"&gt;                    &lt;/span&gt;&lt;span class="c1"&gt;# routing table, numeric&lt;/span&gt;
route&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;default&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# which gateway will this host actually use&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="pf-firewall-state-rules-and-counters"&gt;pf: firewall state, rules, and&amp;nbsp;counters&lt;/h3&gt;
&lt;p&gt;If &lt;code&gt;pf&lt;/code&gt; is enabled, a surprising amount of &amp;#8220;the network is broken&amp;#8221; tickets resolve at the firewall. The&amp;nbsp;base &lt;code&gt;pfctl&lt;/code&gt; tool exposes everything you&amp;nbsp;need:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-si&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# global info: status, counters, state table size and limit&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-sr&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="c1"&gt;# active ruleset with per-rule packet and byte counters&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-vvsr&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="c1"&gt;# same, plus rule numbers and evaluation/eval-skip counts&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-sn&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# NAT rules&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-ss&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# active states (the connection tracking table)&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-ss&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;wc&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="c1"&gt;# quick count of states, compare to limit in &amp;#39;pfctl -si&amp;#39;&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-sm&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# memory limits (states, src-nodes, frags, tables)&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;timeouts&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="c1"&gt;# current timeout values&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;Tables&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# list configured tables&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;name&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# contents of a single table&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The combination&amp;nbsp;of &lt;code&gt;pfctl -vvsr&lt;/code&gt; plus &lt;code&gt;pfctl -si&lt;/code&gt; answers most &amp;#8220;is the firewall counting this traffic at all, and which rule matched?&amp;#8221; questions. If a&amp;nbsp;rule&amp;#8217;s &lt;code&gt;Evaluations&lt;/code&gt; counter is climbing but&amp;nbsp;its &lt;code&gt;Packets&lt;/code&gt; counter is not, traffic is reaching pf but matching a different&amp;nbsp;rule.&lt;/p&gt;
&lt;p&gt;For validating a config without loading&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-nf&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.conf&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# parse and check, do not apply&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And for surgical&amp;nbsp;interventions:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;198&lt;/span&gt;.51.100.7&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="c1"&gt;# kill all states for a host&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;198&lt;/span&gt;.51.100.7&lt;span class="w"&gt; &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.0.0.1&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# kill states between two endpoints&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;pfctl -F state&lt;/code&gt; (flush the entire state table) exists but is almost never the right answer on a live host. Use the&amp;nbsp;targeted &lt;code&gt;-k&lt;/code&gt; form.&lt;/p&gt;
&lt;p&gt;The companion to all of this&amp;nbsp;is &lt;code&gt;pflog&lt;/code&gt;, a pseudo-interface that captures packets matched&amp;nbsp;by &lt;code&gt;log&lt;/code&gt;-tagged&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;tcpdump&lt;span class="w"&gt; &lt;/span&gt;-nettti&lt;span class="w"&gt; &lt;/span&gt;pflog0&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# live view of what pf is logging&lt;/span&gt;
tcpdump&lt;span class="w"&gt; &lt;/span&gt;-nr&lt;span class="w"&gt; &lt;/span&gt;/var/log/pflog&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;host 198.51.100.7&amp;#39;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# search the persisted log&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Add &lt;code&gt;log&lt;/code&gt; to&amp;nbsp;any &lt;code&gt;block&lt;/code&gt; or &lt;code&gt;pass&lt;/code&gt; rule you want visibility&amp;nbsp;into. &lt;code&gt;pflog&lt;/code&gt; itself is lightweight, and rules without&amp;nbsp;the &lt;code&gt;log&lt;/code&gt; keyword incur no logging overhead; logging is usually cheap at moderate traffic levels, but high-rate logging rules can become expensive on busy&amp;nbsp;links.&lt;/p&gt;
&lt;h3 id="live-network-views"&gt;Live network&amp;nbsp;views&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systat&lt;span class="w"&gt; &lt;/span&gt;-tcp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="c1"&gt;# TCP-level activity (states, retransmits)&lt;/span&gt;
systat&lt;span class="w"&gt; &lt;/span&gt;-netstat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="c1"&gt;# active connections, like &amp;#39;netstat&amp;#39; refreshed live&lt;/span&gt;
systat&lt;span class="w"&gt; &lt;/span&gt;-ip&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# IP layer counters&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="packet-capture"&gt;Packet&amp;nbsp;capture&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;tcpdump&lt;span class="w"&gt; &lt;/span&gt;-ni&lt;span class="w"&gt; &lt;/span&gt;igb0&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;tcp port 443&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Quick, surgical packet capture.&amp;nbsp;The &lt;code&gt;-n&lt;/code&gt; skips name resolution (always on a busy host)&amp;nbsp;and &lt;code&gt;-c&lt;/code&gt; limits the capture so you do not flood the&amp;nbsp;terminal.&lt;/p&gt;
&lt;h2 id="per-process-inspection"&gt;Per-Process&amp;nbsp;Inspection&lt;/h2&gt;
&lt;p&gt;When the high level views point to a single process, drill down&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Verbose, wide, with command line&lt;/span&gt;
ps&lt;span class="w"&gt; &lt;/span&gt;-auxww

&lt;span class="c1"&gt;# Process hierarchy / forest view&lt;/span&gt;
ps&lt;span class="w"&gt; &lt;/span&gt;-axjf

&lt;span class="c1"&gt;# Structured output for scripting&lt;/span&gt;
ps&lt;span class="w"&gt; &lt;/span&gt;-auxww&lt;span class="w"&gt; &lt;/span&gt;--libxo&lt;span class="w"&gt; &lt;/span&gt;json&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;jq&lt;span class="w"&gt; &lt;/span&gt;...

&lt;span class="c1"&gt;# Threads for a given process&lt;/span&gt;
procstat&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;pid&amp;gt;

&lt;span class="c1"&gt;# Open file descriptors for a PID&lt;/span&gt;
procstat&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;pid&amp;gt;
fstat&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;pid&amp;gt;

&lt;span class="c1"&gt;# Kernel-side stack (what is the process blocked on?)&lt;/span&gt;
procstat&lt;span class="w"&gt; &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;pid&amp;gt;

&lt;span class="c1"&gt;# Resource limits in effect&lt;/span&gt;
procstat&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;pid&amp;gt;

&lt;span class="c1"&gt;# Memory map&lt;/span&gt;
procstat&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;pid&amp;gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;procstat -k&lt;/code&gt; is particularly useful: if a process is hung, the kernel stack tells you whether it is stuck&amp;nbsp;in &lt;code&gt;nanosleep&lt;/code&gt;, &lt;code&gt;kqueue_wait&lt;/code&gt;, &lt;code&gt;g_waitfor_event&lt;/code&gt;, or somewhere genuinely interesting like a &lt;span class="caps"&gt;ZFS&lt;/span&gt; or &lt;span class="caps"&gt;VFS&lt;/span&gt; call. Requires root privileges on most&amp;nbsp;systems.&lt;/p&gt;
&lt;h2 id="finding-bottlenecks-a-workflow"&gt;Finding Bottlenecks: A&amp;nbsp;Workflow&lt;/h2&gt;
&lt;p&gt;A repeatable triage flow for &amp;#8220;what is using my&amp;nbsp;server&amp;#8221;:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Load and &lt;span class="caps"&gt;CPU&lt;/span&gt;.&lt;/strong&gt; &lt;code&gt;uptime&lt;/code&gt;,&amp;nbsp;then &lt;code&gt;top -SHPj&lt;/code&gt;. Compare 1-minute load&amp;nbsp;to &lt;code&gt;hw.ncpu&lt;/code&gt;. Look at the per-&lt;span class="caps"&gt;CPU&lt;/span&gt; line for hotspots and&amp;nbsp;at &lt;code&gt;system&lt;/code&gt; vs &lt;code&gt;user&lt;/code&gt; vs &lt;code&gt;interrupt&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Memory.&lt;/strong&gt; Look at&amp;nbsp;the &lt;code&gt;top&lt;/code&gt; memory header.&amp;nbsp;Is &lt;code&gt;Free&lt;/code&gt; near zero &lt;em&gt;and&lt;/em&gt; &lt;code&gt;Inact&lt;/code&gt;/&lt;code&gt;Laundry&lt;/code&gt; near zero &lt;em&gt;and&lt;/em&gt; swap rising? That is real pressure. Otherwise, &amp;#8220;low free memory&amp;#8221; is not a problem on&amp;nbsp;FreeBSD.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Disk.&lt;/strong&gt; &lt;code&gt;gstat -p&lt;/code&gt; or &lt;code&gt;iostat -x 1&lt;/code&gt;.&amp;nbsp;High &lt;code&gt;%busy&lt;/code&gt; with&amp;nbsp;high &lt;code&gt;ms/op&lt;/code&gt; is the smoking&amp;nbsp;gun.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Network.&lt;/strong&gt; &lt;code&gt;netstat -i -w 1&lt;/code&gt; for throughput and&amp;nbsp;errors, &lt;code&gt;netstat -m&lt;/code&gt; for buffer&amp;nbsp;exhaustion, &lt;code&gt;sockstat -4 -c | wc -l&lt;/code&gt; for connection&amp;nbsp;counts.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Drill down.&lt;/strong&gt; Once you know the subsystem,&amp;nbsp;switch &lt;code&gt;top&lt;/code&gt; into the right mode&amp;nbsp;(&lt;code&gt;-m io&lt;/code&gt;) or sort key&amp;nbsp;(&lt;code&gt;-o res&lt;/code&gt;, &lt;code&gt;-o size&lt;/code&gt;), or&amp;nbsp;use &lt;code&gt;procstat -k &amp;lt;pid&amp;gt;&lt;/code&gt; on&amp;nbsp;suspects.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Jail attribution.&lt;/strong&gt; On a multi-tenant jail&amp;nbsp;host, &lt;code&gt;top -j&lt;/code&gt; plus &lt;code&gt;rctl -hu jail:&amp;lt;name&amp;gt;&lt;/code&gt; (next section) tells you which jail is&amp;nbsp;responsible.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="per-jail-accounting-and-limits-with-kernracct"&gt;Per-Jail Accounting and Limits with&amp;nbsp;kern.racct&lt;/h2&gt;
&lt;p&gt;Everything above tells you what the &lt;em&gt;host&lt;/em&gt; is doing. On a jail host you often need the same view per jail, both for capacity planning and for catching the noisy neighbour. That is&amp;nbsp;what &lt;code&gt;kern.racct&lt;/code&gt; (accounting)&amp;nbsp;and &lt;code&gt;rctl&lt;/code&gt; (control) are&amp;nbsp;for.&lt;/p&gt;
&lt;h3 id="enabling-racct"&gt;Enabling&amp;nbsp;racct&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;RACCT&lt;/code&gt; and &lt;code&gt;RCTL&lt;/code&gt; are compiled into the &lt;span class="caps"&gt;GENERIC&lt;/span&gt; kernel but disabled at boot, because their per-thread bookkeeping has a small cost. Enable them&amp;nbsp;in &lt;code&gt;/boot/loader.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;kern.racct.enable=1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is a boot-time tunable. A reboot is required; you cannot toggle it at runtime. After&amp;nbsp;reboot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;kern.racct.enable
kern.racct.enable:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Once enabled, accounting applies to every jail automatically. No per-jail opt-in is&amp;nbsp;needed.&lt;/p&gt;
&lt;h3 id="reading-per-jail-usage"&gt;Reading per-jail&amp;nbsp;usage&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;rctl&lt;span class="w"&gt; &lt;/span&gt;-hu&lt;span class="w"&gt; &lt;/span&gt;jail:web1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cputime=2h
datasize=0
stacksize=0
coredumpsize=0
memoryuse=412M
memorylocked=0
maxproc=37
openfiles=412
vmemoryuse=1G
pseudoterminals=0
swapuse=0
nthr=58
wallclock=3d
pcpu=12
readbps=0
writebps=0
readiops=0
writeiops=0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;-h&lt;/code&gt; is human readable; drop it for raw byte/second numbers if you want to script against the&amp;nbsp;output. &lt;code&gt;rctl&lt;/code&gt; is the stable user-facing interface to racct data; internally it uses&amp;nbsp;the &lt;code&gt;rctl_get_racct(2)&lt;/code&gt; syscall. There is no&amp;nbsp;equivalent &lt;code&gt;sysctl&lt;/code&gt; interface despite occasional rumours to the&amp;nbsp;contrary.&lt;/p&gt;
&lt;h3 id="setting-limits"&gt;Setting&amp;nbsp;limits&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;rctl&lt;/code&gt; rules use the&amp;nbsp;form:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;subject:identifier:resource:action=amount
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Subjects&amp;nbsp;are &lt;code&gt;process&lt;/code&gt;, &lt;code&gt;user&lt;/code&gt;, &lt;code&gt;loginclass&lt;/code&gt;,&amp;nbsp;or &lt;code&gt;jail&lt;/code&gt;. For jails, the identifier is the jail name. Actions you will actually&amp;nbsp;use:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;deny&lt;/code&gt; blocks further allocation when the limit is&amp;nbsp;hit&lt;/li&gt;
&lt;li&gt;&lt;code&gt;log&lt;/code&gt; writes to the system log but does not&amp;nbsp;enforce&lt;/li&gt;
&lt;li&gt;&lt;code&gt;devctl&lt;/code&gt; sends&amp;nbsp;a &lt;code&gt;devd(8)&lt;/code&gt; notification you can hook with custom&amp;nbsp;scripts&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Examples:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Cap one jail at 1 GB resident memory&lt;/span&gt;
rctl&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;jail:web1:memoryuse:deny&lt;span class="o"&gt;=&lt;/span&gt;1G

&lt;span class="c1"&gt;# Cap at 50% of a single CPU (pcpu is percent of one core, not total host CPU,&lt;/span&gt;
&lt;span class="c1"&gt;# so on a 32-core box this is still only half of one core)&lt;/span&gt;
rctl&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;jail:web1:pcpu:deny&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;50&lt;/span&gt;

&lt;span class="c1"&gt;# Cap write throughput at 20 MB/s, 2000 iops&lt;/span&gt;
rctl&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;jail:web1:writebps:deny&lt;span class="o"&gt;=&lt;/span&gt;20M
rctl&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;jail:web1:writeiops:deny&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2000&lt;/span&gt;

&lt;span class="c1"&gt;# Cap process count&lt;/span&gt;
rctl&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;jail:web1:maxproc:deny&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;200&lt;/span&gt;

&lt;span class="c1"&gt;# Just log when a jail sustains &amp;gt; 100 MB/s read, do not enforce&lt;/span&gt;
rctl&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;jail:web1:readbps:log&lt;span class="o"&gt;=&lt;/span&gt;100M
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;List active&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;rctl&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;jail:web1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="persisting-rules"&gt;Persisting&amp;nbsp;rules&lt;/h3&gt;
&lt;p&gt;Rules added&amp;nbsp;with &lt;code&gt;rctl -a&lt;/code&gt; live until the next reboot. To persist them, put them&amp;nbsp;in &lt;code&gt;/etc/rctl.conf&lt;/code&gt; (one rule per line, same syntax) and enable the&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# /etc/rctl.conf
jail:web1:memoryuse:deny=1G
jail:web1:maxproc:deny=200
jail:web1:vmemoryuse:deny=2G
jail:db1:memoryuse:deny=4G
jail:db1:writeiops:log=5000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;rctl_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;rctl&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;service rctl start&lt;/code&gt; is a one-shot: it&amp;nbsp;parses &lt;code&gt;rctl.conf&lt;/code&gt; and pushes the rules into the kernel. After&amp;nbsp;editing &lt;code&gt;rctl.conf&lt;/code&gt;,&amp;nbsp;re-running &lt;code&gt;service rctl start&lt;/code&gt; (or restarting the service) re-applies rules from the&amp;nbsp;file.&lt;/p&gt;
&lt;h3 id="a-practical-caution-about-deny"&gt;A practical caution about&amp;nbsp;deny&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;deny&lt;/code&gt; is a hard wall. When a jail hits&amp;nbsp;its &lt;code&gt;memoryuse&lt;/code&gt; cap, allocations inside it start&amp;nbsp;returning &lt;code&gt;ENOMEM&lt;/code&gt;. Well behaved C programs handle that. Many large runtimes (the &lt;span class="caps"&gt;JVM&lt;/span&gt;, Node.js, some database engines) do not, and will abort or crash instead. Test in staging before deploying memory caps on production jails. For first deployments I usually start&amp;nbsp;with &lt;code&gt;log&lt;/code&gt; actions and watch what would have been denied for a week before switching&amp;nbsp;to &lt;code&gt;deny&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="seeing-jails-in-top"&gt;Seeing jails in&amp;nbsp;top&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;top&lt;span class="w"&gt; &lt;/span&gt;-j
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;adds&amp;nbsp;a &lt;code&gt;JID&lt;/code&gt; column. Combine&amp;nbsp;with &lt;code&gt;-m io&lt;/code&gt; to see which jail is doing the disk work. For most &amp;#8220;which jail is eating the server?&amp;#8221; questions this is enough; no extra tooling&amp;nbsp;required.&lt;/p&gt;
&lt;h2 id="a-note-on-jail_exporter-and-prometheus"&gt;A Note on jail_exporter and&amp;nbsp;Prometheus&lt;/h2&gt;
&lt;p&gt;The tools above give you live, in-the-moment views. If you want history, alerting, and dashboards you will eventually want to&amp;nbsp;feed &lt;code&gt;racct&lt;/code&gt; data into Prometheus. The community option is &lt;a href="https://github.com/phyber/jail_exporter"&gt;&lt;code&gt;jail_exporter&lt;/code&gt;&lt;/a&gt;, a small Rust exporter that&amp;nbsp;reads &lt;code&gt;racct&lt;/code&gt; and exposes per-jail metrics on an &lt;span class="caps"&gt;HTTP&lt;/span&gt; endpoint (default&amp;nbsp;port &lt;code&gt;9452&lt;/code&gt;, the conventional Prometheus exporter port for the&amp;nbsp;project).&lt;/p&gt;
&lt;p&gt;It is available as a port and&amp;nbsp;package:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;jail_exporter
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;jail_exporter_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;jail_exporter&lt;span class="w"&gt; &lt;/span&gt;start
curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;http://localhost:9452/metrics&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;^jail_
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I will not enumerate the metric set here, because it tracks the upstream &lt;span class="caps"&gt;README&lt;/span&gt; and any list I write would go stale. Read the &lt;span class="caps"&gt;README&lt;/span&gt;, scrape it from Prometheus, and graph&amp;nbsp;whichever &lt;code&gt;jail_racct_*&lt;/code&gt; series matter for your workload. Everything in the earlier triage workflow (&lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, I/O, process counts) is&amp;nbsp;exposed.&lt;/p&gt;
&lt;p&gt;This is a sidecar to the work above, not a replacement. When something is on fire right&amp;nbsp;now, &lt;code&gt;top -SHPj&lt;/code&gt;, &lt;code&gt;gstat&lt;/code&gt;, &lt;code&gt;netstat -m&lt;/code&gt;, &lt;code&gt;sockstat&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;rctl -hu&lt;/code&gt; will get you to the answer faster than any&amp;nbsp;dashboard.&lt;/p&gt;
&lt;h2 id="wrap-up"&gt;Wrap-up&lt;/h2&gt;
&lt;p&gt;FreeBSD&amp;#8217;s monitoring story is not glamorous, but it is unusually complete out of the box.&amp;nbsp;Between &lt;code&gt;top&lt;/code&gt;, &lt;code&gt;vmstat&lt;/code&gt;, &lt;code&gt;gstat&lt;/code&gt;, &lt;code&gt;systat&lt;/code&gt;, &lt;code&gt;netstat&lt;/code&gt;, &lt;code&gt;sockstat&lt;/code&gt;, &lt;code&gt;procstat&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;rctl&lt;/code&gt; you can usually answer &amp;#8220;what is using my server&amp;#8221; in a couple of minutes without installing anything.&amp;nbsp;Add &lt;code&gt;kern.racct&lt;/code&gt; and a&amp;nbsp;few &lt;code&gt;rctl.conf&lt;/code&gt; rules and you also get accurate per-jail attribution and the ability to cap noisy neighbours before they take the host&amp;nbsp;down.&lt;/p&gt;
&lt;p&gt;The next article in this series will look&amp;nbsp;at &lt;code&gt;dtrace&lt;/code&gt; and &lt;code&gt;pmcstat&lt;/code&gt; for the cases where this toolkit is not enough: low level latency analysis, lock contention, and hardware performance&amp;nbsp;counters.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=top&amp;amp;sektion=1"&gt;top(1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=systat&amp;amp;sektion=1"&gt;systat(1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=vmstat&amp;amp;sektion=8"&gt;vmstat(8)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=iostat&amp;amp;sektion=8"&gt;iostat(8)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=gstat&amp;amp;sektion=8"&gt;gstat(8)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=netstat&amp;amp;sektion=1"&gt;netstat(1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=sockstat&amp;amp;sektion=1"&gt;sockstat(1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=procstat&amp;amp;sektion=1"&gt;procstat(1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pfctl&amp;amp;sektion=8"&gt;pfctl(8)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pflog&amp;amp;sektion=4"&gt;pflog(4)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=rctl&amp;amp;sektion=8"&gt;rctl(8)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=racct&amp;amp;sektion=4"&gt;racct(4)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=rctl.conf&amp;amp;sektion=5"&gt;rctl.conf(5)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/phyber/jail_exporter"&gt;jail_exporter on&amp;nbsp;GitHub&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;FreeBSD Fundamentals:&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="performance"/><category term="monitoring"/><category term="troubleshooting"/><category term="jails"/><category term="racct"/><category term="rctl"/><category term="sysctl"/><category term="network"/><category term="observability"/></entry><entry><title>Automating FreeBSD Jails with cdist - Zero Dependencies Inside the Jail</title><link href="https://hofstede.it/automating-freebsd-jails-with-cdist-zero-dependencies-inside-the-jail/" rel="alternate"/><published>2026-04-12T00:00:00+02:00</published><updated>2026-04-12T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-04-12:/automating-freebsd-jails-with-cdist-zero-dependencies-inside-the-jail/</id><summary type="html">&lt;p&gt;cdist is refreshingly minimal - the target only needs &lt;span class="caps"&gt;POSIX&lt;/span&gt; sh, and the control machine speaks ssh. But cdist expects one ssh endpoint per host, and FreeBSD jails are not normally their own ssh targets. Two small Python wrappers plug cdist into jexec on the host, so configuration state flows into every jail without running a single daemon, agent, or Python interpreter inside the jail&amp;nbsp;itself.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Most configuration management tools still assume they own the target. Ansible ships Python modules over the wire and runs them in place. Salt wants a minion on every host. Chef wants a client, Puppet wants an&amp;nbsp;agent.&lt;/p&gt;
&lt;p&gt;&lt;img alt="cdist - FreeBSD" src="https://hofstede.it/images/2026-04-12-cdist-freebsd-jails.png" title="cdist - FreeBSD"&gt; &lt;/p&gt;
&lt;p&gt;Jails break that assumption in a satisfying way. A FreeBSD jail is supposed to be small - sometimes a single static binary,&amp;nbsp;an &lt;code&gt;rc.d&lt;/code&gt; script, and a few lines&amp;nbsp;in &lt;code&gt;rc.conf&lt;/code&gt;. Installing Python into every jail just so Ansible can run&amp;nbsp;its &lt;code&gt;setup&lt;/code&gt; module is, to borrow a phrase, the tail wagging the dog. I already wrote about a workaround for Ansible: &lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;the jailexec connection plugin&lt;/a&gt;, which SSHes to the jail host and&amp;nbsp;uses &lt;code&gt;jexec&lt;/code&gt; to tunnel commands into each jail. It works, and I still reach for it when I already have an Ansible setup. But many Ansible modules assume Python on the target, so in practice some of those jails still end up growing a Python&amp;nbsp;interpreter.&lt;/p&gt;
&lt;p&gt;Then I tried cdist, and everything got&amp;nbsp;smaller.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-cdist-actually-is"&gt;What cdist Actually&amp;nbsp;Is&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-two-hooks-that-make-this-interesting"&gt;The Two Hooks That Make This&amp;nbsp;Interesting&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#jexec-sshpy"&gt;jexec-ssh.py&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#jexec-scppy"&gt;jexec-scp.py&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#actually-running-it"&gt;Actually Running It&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#a-more-practical-example-a-custom-maintenance-type"&gt;A More Practical Example: A Custom Maintenance&amp;nbsp;Type&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-this-is-the-unix-way"&gt;Why This Is The Unix&amp;nbsp;Way&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#when-to-still-reach-for-ansible"&gt;When To Still Reach For&amp;nbsp;Ansible&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#getting-the-scripts"&gt;Getting The&amp;nbsp;Scripts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="what-cdist-actually-is"&gt;What cdist Actually&amp;nbsp;Is&lt;/h2&gt;
&lt;p&gt;cdist is a configuration management system written in Python that runs entirely on the control machine. On the target, it expects a&amp;nbsp;Bourne-style &lt;code&gt;sh&lt;/code&gt; environment. That is it. No agent, no client, no target-side runtime of any kind. cdist assembles a shell script locally, ships it over the wire, and watches stdout come back. The target does not know it is being managed any more than it would know it was being SSHed&amp;nbsp;into.&lt;/p&gt;
&lt;p&gt;That minimalism is the selling point. cdist&amp;#8217;s own tagline reads &amp;#8220;usable configuration management&amp;#8221;, but what I read between the lines is: the entire target-side runtime is whatever shell is already on the box. If you can ssh to it, you can cdist it. No bootstrap step, no package install, no &amp;#8220;please add our apt key first&amp;#8221;. For a FreeBSD jail that exists precisely to run one daemon and nothing else, that is&amp;nbsp;perfect.&lt;/p&gt;
&lt;p&gt;The model is also refreshingly simple conceptually. You write &lt;strong&gt;manifests&lt;/strong&gt; (declarative entry points) that invoke &lt;strong&gt;types&lt;/strong&gt; (reusable building&amp;nbsp;blocks: &lt;code&gt;__file&lt;/code&gt;, &lt;code&gt;__package_pkgng&lt;/code&gt;, &lt;code&gt;__service&lt;/code&gt;, &lt;code&gt;__line&lt;/code&gt;, and so on). Every type is itself a small directory of shell scripts -&amp;nbsp;an &lt;code&gt;explorer&lt;/code&gt; that reports current state,&amp;nbsp;a &lt;code&gt;gencode-remote&lt;/code&gt; that emits the shell commands to reach the desired state, and optional helpers. cdist runs the explorers on the target, generates a shell script locally, sends it over ssh, and runs it&amp;nbsp;with &lt;code&gt;sh -e&lt;/code&gt;. That is the whole&amp;nbsp;loop.&lt;/p&gt;
&lt;h2 id="the-two-hooks-that-make-this-interesting"&gt;The Two Hooks That Make This&amp;nbsp;Interesting&lt;/h2&gt;
&lt;p&gt;cdist&amp;#8217;s default transport is&amp;nbsp;OpenSSH: &lt;code&gt;ssh user@host sh -c '…'&lt;/code&gt; for&amp;nbsp;commands, &lt;code&gt;scp&lt;/code&gt; for files. But the transport is swappable. Two command-line flags override&amp;nbsp;it:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;--remote-exec PATH&lt;/code&gt; - a script that takes ssh-style options followed&amp;nbsp;by &lt;code&gt;&amp;lt;target&amp;gt; &amp;lt;command…&amp;gt;&lt;/code&gt; and runs the command on the target&amp;nbsp;somehow.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;--remote-copy PATH&lt;/code&gt; - a script that takes scp-style arguments and copies files to the target&amp;nbsp;somehow.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In this setup, the cdist target name is not a hostname at all; it is the jail&amp;nbsp;name. &lt;code&gt;JAIL_HOST&lt;/code&gt; tells the wrappers which real &lt;span class="caps"&gt;SSH&lt;/span&gt; endpoint to&amp;nbsp;use.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;API&lt;/span&gt; is dumb in the best way. cdist gives you the target name and a command line; the script does whatever it needs to. If you want to tunnel through a bastion, go ahead. If you want to talk to a switch over a serial console, go ahead. And if you want to &lt;span class="caps"&gt;SSH&lt;/span&gt; to a FreeBSD host and drop into a jail&amp;nbsp;with &lt;code&gt;jexec&lt;/code&gt; - which is exactly what I want - that is two small Python scripts. The only persistent requirement is host-side access&amp;nbsp;to &lt;code&gt;ssh&lt;/code&gt;, &lt;code&gt;scp&lt;/code&gt;, &lt;code&gt;jexec&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;jls&lt;/code&gt;; the jail itself remains untouched except for the configuration changes cdist&amp;nbsp;applies.&lt;/p&gt;
&lt;h2 id="jexec-sshpy"&gt;jexec-ssh.py&lt;/h2&gt;
&lt;p&gt;Here is the remote-exec wrapper in full. It is 60 lines, most of them&amp;nbsp;boilerplate:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/usr/bin/env python3&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;shlex&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sys&lt;/span&gt;

&lt;span class="n"&gt;_SSH_OPTS_WITH_VALUE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;frozenset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
    &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-o&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-F&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-i&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-l&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-p&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-c&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-m&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-L&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;strip_ssh_options&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;arg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;arg&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;_SSH_OPTS_WITH_VALUE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;arg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;jail_host&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;JAIL_HOST&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;jail_host&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;error: JAIL_HOST is not set&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;jail_user&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;JAIL_USER&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;root&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;args&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:]&lt;/span&gt;
    &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;strip_ssh_options&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;error: no target jail name on the command line&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;jailname&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;remote_payload&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot; &amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:])&lt;/span&gt;

    &lt;span class="n"&gt;jexec_cmd&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;jexec -u &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;shlex&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;jail_user&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;shlex&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;jailname&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; &amp;quot;&lt;/span&gt;
        &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/bin/sh -c &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;shlex&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;remote_payload&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execvp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;jail_host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;jexec_cmd&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;__main__&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The logic is the whole&amp;nbsp;story:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Read &lt;code&gt;JAIL_HOST&lt;/code&gt; from the environment. That is the ssh endpoint for the &lt;em&gt;host&lt;/em&gt;, not the&amp;nbsp;jail.&lt;/li&gt;
&lt;li&gt;Strip the ssh-style options cdist prepends - things&amp;nbsp;like &lt;code&gt;-o User=root&lt;/code&gt; - which are meaningful to a jail&amp;#8217;s (nonexistent) direct ssh endpoint, not to ours. Real connection options belong&amp;nbsp;in &lt;code&gt;~/.ssh/config&lt;/code&gt; for the&amp;nbsp;host.&lt;/li&gt;
&lt;li&gt;The first non-option argument is the jail name. Everything after that is the command cdist wants to&amp;nbsp;execute.&lt;/li&gt;
&lt;li&gt;Wrap the command&amp;nbsp;in &lt;code&gt;jexec -u &amp;lt;user&amp;gt; &amp;lt;jail&amp;gt; /bin/sh -c '…'&lt;/code&gt;, &lt;code&gt;shlex.quote&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;d so pipelines, redirections and globs survive transit through two&amp;nbsp;shells.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;execvp&lt;/code&gt; into the&amp;nbsp;real &lt;code&gt;ssh&lt;/code&gt;, replacing the Python process. ssh does the heavy lifting, and its exit code flows straight back to&amp;nbsp;cdist.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The command is crossing two shell boundaries: the host-side &lt;span class="caps"&gt;SSH&lt;/span&gt; command line and the&amp;nbsp;in-jail &lt;code&gt;sh -c&lt;/code&gt;. That is why the wrapper leans so hard&amp;nbsp;on &lt;code&gt;shlex.quote()&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;/bin/sh -c&lt;/code&gt; wrapper itself is not cosmetic. Without&amp;nbsp;it, &lt;code&gt;jexec&lt;/code&gt; hands the argv directly to whatever binary sits at position one, and cdist-generated constructs&amp;nbsp;like &lt;code&gt;for f in /etc/*.conf; do …; done&lt;/code&gt; die the moment a glob shows up. Running the payload through a&amp;nbsp;fresh &lt;code&gt;sh&lt;/code&gt; inside the jail restores the shell environment cdist&amp;nbsp;expects.&lt;/p&gt;
&lt;h2 id="jexec-scppy"&gt;jexec-scp.py&lt;/h2&gt;
&lt;p&gt;File transfer is slightly more involved, but only&amp;nbsp;slightly. &lt;code&gt;scp&lt;/code&gt; does not know what a jail is - but it does know what a host path is, and a jail&amp;#8217;s &amp;#8220;inside&amp;#8221; is always some path on the host&amp;nbsp;(&lt;code&gt;/zroot/jails/&amp;lt;name&amp;gt;&lt;/code&gt; is typical). We can&amp;nbsp;translate.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/usr/bin/env python3&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;os&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;subprocess&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;sys&lt;/span&gt;

&lt;span class="n"&gt;_SCP_OPTS_WITH_VALUE&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;frozenset&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-o&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-F&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-i&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-S&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-P&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;split_args&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;paths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[],&lt;/span&gt; &lt;span class="p"&gt;[]&lt;/span&gt;
    &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;arg&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;arg&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;_SCP_OPTS_WITH_VALUE&lt;/span&gt; &lt;span class="ow"&gt;and&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;extend&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;arg&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt;
            &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
        &lt;span class="k"&gt;elif&lt;/span&gt; &lt;span class="n"&gt;arg&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;arg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
        &lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;arg&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
            &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;paths&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;jail_root&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;jailname&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="n"&gt;out&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;check_output&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;/usr/sbin/jls&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-j&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;jailname&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;path&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
        &lt;span class="n"&gt;text&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="n"&gt;stderr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;subprocess&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;STDOUT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;rewrite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;:&amp;quot;&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;
    &lt;span class="n"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;rpath&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;:&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;/&amp;quot;&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;prefix&lt;/span&gt; &lt;span class="ow"&gt;or&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="n"&gt;path&lt;/span&gt;
    &lt;span class="n"&gt;root&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;jail_root&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;prefix&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;return&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rpath&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;lstrip&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;


&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;main&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;jail_host&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;environ&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;JAIL_HOST&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;jail_host&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;error: JAIL_HOST is not set&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;paths&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;split_args&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;argv&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;:])&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;exit&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;error: scp needs a source and a destination&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="n"&gt;translated&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;rewrite&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;jail_host&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;paths&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;os&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;execvp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;scp&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;scp&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;opts&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-r&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;-p&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;translated&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;


&lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="vm"&gt;__name__&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;__main__&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Walking through&amp;nbsp;it:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;For each argument, check if it looks&amp;nbsp;like &lt;code&gt;&amp;lt;jail&amp;gt;:/some/path&lt;/code&gt;. If not - local paths, host-qualified paths - leave it&amp;nbsp;alone.&lt;/li&gt;
&lt;li&gt;If it does, ssh to the host and&amp;nbsp;run &lt;code&gt;jls -j &amp;lt;jail&amp;gt; path&lt;/code&gt;. &lt;code&gt;jls&lt;/code&gt; is the canonical way to ask FreeBSD &amp;#8220;where does jail X live on disk right&amp;nbsp;now&amp;#8221;.&lt;/li&gt;
&lt;li&gt;Rewrite the argument&amp;nbsp;to &lt;code&gt;&amp;lt;JAIL_HOST&amp;gt;:&amp;lt;jailroot&amp;gt;/&amp;lt;path&amp;gt;&lt;/code&gt;, which is just a perfectly ordinary scp remote&amp;nbsp;path.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;execvp&lt;/code&gt; into the&amp;nbsp;system &lt;code&gt;scp&lt;/code&gt; with the rewritten&amp;nbsp;arguments.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;There is nothing magical&amp;nbsp;here. &lt;code&gt;scp&lt;/code&gt; was already perfectly capable of copying files&amp;nbsp;to &lt;code&gt;root@radon.example.com:/zroot/jails/testvnet/etc/motd&lt;/code&gt;. All the wrapper does is save the user the trouble of typing it and let cdist&amp;#8217;s file types behave as if the jail were a first-class ssh target. The simple implementation&amp;nbsp;asks &lt;code&gt;jls&lt;/code&gt; once per jail-qualified path; caching jail roots would be an easy optimization if you start moving lots of&amp;nbsp;files.&lt;/p&gt;
&lt;h2 id="actually-running-it"&gt;Actually Running&amp;nbsp;It&lt;/h2&gt;
&lt;p&gt;On a host with many jails, &lt;span class="caps"&gt;SSH&lt;/span&gt; connection setup can dominate runtime; enabling multiplexing on the host connection makes a dramatic difference. More on that in the repo&amp;#8217;s &lt;span class="caps"&gt;README&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;A minimal setup. First, a manifest&amp;nbsp;at &lt;code&gt;./conf/manifest/init&lt;/code&gt; - one line is enough to prove the whole pipeline works end to&amp;nbsp;end:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;__file&lt;span class="w"&gt; &lt;/span&gt;/tmp/cdist-wrapper-success&lt;span class="w"&gt; &lt;/span&gt;--state&lt;span class="w"&gt; &lt;/span&gt;present
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then point cdist at the two&amp;nbsp;wrappers:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~/tmp/cdist-test ❯ export JAIL_HOST=root@radon.edelga.se
~/tmp/cdist-test ❯ cdist config -c ./conf \
    --remote-exec ./jexec-ssh.py \
    --remote-copy ./jexec-scp.py \
    testvnet lg
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In this&amp;nbsp;example, &lt;code&gt;testvnet&lt;/code&gt; and &lt;code&gt;lg&lt;/code&gt; are jail names, not &lt;span class="caps"&gt;DNS&lt;/span&gt; names or &lt;span class="caps"&gt;SSH&lt;/span&gt; inventory entries. cdist is quiet on success: no output is good output. A sanity check on the host itself confirms&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@radon ~]# jexec testvnet ls -l /tmp/cdist-wrapper-success
-rw-------  1 root wheel 0 Apr 11 21:02 /tmp/cdist-wrapper-success
[root@radon ~]# jexec lg ls -l /tmp/cdist-wrapper-success
-rw-------  1 root wheel 0 Apr 11 21:18 /tmp/cdist-wrapper-success
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two jails, one cdist invocation, zero daemons, zero extra interpreters, nothing mutated outside the jails themselves. Replace the one-liner manifest with real cdist types&amp;nbsp;- &lt;code&gt;__package_pkgng&lt;/code&gt;, &lt;code&gt;__file&lt;/code&gt;, &lt;code&gt;__line&lt;/code&gt;, &lt;code&gt;__service&lt;/code&gt;, &lt;code&gt;__user&lt;/code&gt; - and you get the same thing at whatever scale you&amp;nbsp;need.&lt;/p&gt;
&lt;h3 id="a-more-practical-example-a-custom-maintenance-type"&gt;A More Practical Example: A Custom Maintenance&amp;nbsp;Type&lt;/h3&gt;
&lt;p&gt;The smoke test above proves the transport works. Here is a pragmatic host-maintenance example I use in practice: a custom type that runs base and package updates inside a jail. This is more of a maintenance action than a pure convergent state type, but it is a good illustration of how little machinery cdist needs when you do want to run shell-driven lifecycle&amp;nbsp;tasks.&lt;/p&gt;
&lt;p&gt;First, create the type&amp;#8217;s directory and&amp;nbsp;its &lt;code&gt;gencode-remote&lt;/code&gt; script. This is the shell that cdist will execute inside the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;conf/type/__freebsd_system_update
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# conf/type/__freebsd_system_update/gencode-remote&lt;/span&gt;

&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Checking for and installing FreeBSD base updates...&amp;quot;&lt;/span&gt;
&lt;span class="c1"&gt;# PAGER=cat prevents freebsd-update from dropping into an&lt;/span&gt;
&lt;span class="c1"&gt;# interactive pager and waiting for &amp;#39;q&amp;#39; on a headless target.&lt;/span&gt;
env&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PAGER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;freebsd-update&lt;span class="w"&gt; &lt;/span&gt;fetch&lt;span class="w"&gt; &lt;/span&gt;install

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Updating package catalogue and installing upgrades...&amp;quot;&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;update
pkg&lt;span class="w"&gt; &lt;/span&gt;upgrade&lt;span class="w"&gt; &lt;/span&gt;-y
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;+x&lt;span class="w"&gt; &lt;/span&gt;conf/type/__freebsd_system_update/gencode-remote
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then wire it into the manifest. cdist&amp;nbsp;exposes &lt;code&gt;$__target_os&lt;/code&gt; as an explorer, so the type only fires on FreeBSD&amp;nbsp;targets:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# conf/manifest/init&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$__target_os&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FreeBSD&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;__freebsd_system_update
&lt;span class="k"&gt;else&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Info: &lt;/span&gt;&lt;span class="nv"&gt;$__target_host&lt;/span&gt;&lt;span class="s2"&gt; is not FreeBSD, skipping updates.&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Run it the same way as&amp;nbsp;before:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~/tmp/cdist-test ❯ cdist config -c ./conf \
    --remote-exec ./jexec-ssh.py \
    --remote-copy ./jexec-scp.py \
    testvnet lg
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the whole workflow for writing a custom cdist type: a directory, a shell script, and a one-line invocation in the manifest. No &lt;span class="caps"&gt;YAML&lt;/span&gt;, no plugin protocol, no class hierarchy. The type is just shell, and the wrappers make sure that shell runs inside the&amp;nbsp;jail.&lt;/p&gt;
&lt;h2 id="why-this-is-the-unix-way"&gt;Why This Is The Unix&amp;nbsp;Way&lt;/h2&gt;
&lt;p&gt;I wrote these scripts one Monday afternoon. By dinnertime I had working state management across every jail on my FreeBSD hosts, without installing anything new on a single one of them. That is not a coincidence; it is what happens when every component in the stack knows exactly one job and does it&amp;nbsp;well.&lt;/p&gt;
&lt;p&gt;Count the&amp;nbsp;actors:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;ssh&lt;/code&gt; - authenticated, encrypted&amp;nbsp;transport&lt;/li&gt;
&lt;li&gt;&lt;code&gt;scp&lt;/code&gt; - file copy over&amp;nbsp;ssh&lt;/li&gt;
&lt;li&gt;&lt;code&gt;jexec&lt;/code&gt; - process execution inside a running jail&amp;#8217;s&amp;nbsp;namespace&lt;/li&gt;
&lt;li&gt;&lt;code&gt;jls&lt;/code&gt; - jail&amp;nbsp;inspection&lt;/li&gt;
&lt;li&gt;&lt;code&gt;sh&lt;/code&gt; - shell&amp;nbsp;execution&lt;/li&gt;
&lt;li&gt;&lt;code&gt;cdist&lt;/code&gt; - render desired state into a shell&amp;nbsp;script&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Each piece is narrow, composable, and independently useful. The two Python files in this repo do not replace any of them; they only connect them. Together that is roughly 120 lines of Python orchestrating several decades of battle-tested Unix primitives. None of those tools are tightly coupled to each other. None of them share a runtime. If I delete any one file from the pipeline, the rest still do their jobs. That is the thing people mean when they talk about &amp;#8220;the Unix philosophy&amp;#8221;, and it is the thing I keep coming back to every time I look at how modern tools solve the same&amp;nbsp;problems.&lt;/p&gt;
&lt;p&gt;Compare the dependency footprint inside the&amp;nbsp;jail:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;System&lt;/th&gt;
&lt;th&gt;Typically required inside the jail&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Ansible (default)&lt;/td&gt;
&lt;td&gt;Python interpreter for many modules, plus a normal remote access path&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ansible (jailexec)&lt;/td&gt;
&lt;td&gt;Python interpreter (for most modules)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Salt&lt;/td&gt;
&lt;td&gt;&lt;code&gt;salt-minion&lt;/code&gt;, Python, dependencies&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Puppet&lt;/td&gt;
&lt;td&gt;&lt;code&gt;puppet&lt;/code&gt; agent, Ruby, dependencies&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Chef&lt;/td&gt;
&lt;td&gt;&lt;code&gt;chef-client&lt;/code&gt;, Ruby, dependencies&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;cdist + these wrappers&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;&lt;span class="caps"&gt;POSIX&lt;/span&gt; &lt;code&gt;sh&lt;/code&gt;, which is already in base&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The same story plays out in the implementations themselves. My &lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;Ansible jailexec connection plugin&lt;/a&gt; is roughly 960 lines of Python, because Ansible&amp;#8217;s connection plugin &lt;span class="caps"&gt;API&lt;/span&gt; asks for a fair amount of scaffolding: argument translation, a file-transfer state machine, subprocess management, the whole plugin protocol. These two cdist wrappers together come to about 120 lines. That difference is not about quality; it is about how much each tool asks of the transport layer. cdist&amp;#8217;s hooks are deliberately narrow, so there is less to implement. And because cdist&amp;#8217;s types are shell, the jail does not need a Python interpreter at all, whereas the Ansible path still typically&amp;nbsp;does.&lt;/p&gt;
&lt;p&gt;The smallest running jail on my radon.edelga.se host is 61 &lt;span class="caps"&gt;MB&lt;/span&gt; on disk - a thin-jail nullfs view over a shared base, running one static binary. It still configures beautifully with this pipeline, because the pipeline asks it for nothing it was not already&amp;nbsp;providing.&lt;/p&gt;
&lt;h2 id="when-to-still-reach-for-ansible"&gt;When To Still Reach For&amp;nbsp;Ansible&lt;/h2&gt;
&lt;p&gt;This is not the universal&amp;nbsp;answer.&lt;/p&gt;
&lt;p&gt;cdist is opinionated about shell, which means that if your configuration domain is &amp;#8220;take this complicated &lt;span class="caps"&gt;JSON&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt;, reason about its responses, compute a delta, push it back&amp;#8221; - the kind of thing Ansible modules do in a page of Python each - you will end up writing that logic yourself&amp;nbsp;in &lt;code&gt;sh&lt;/code&gt;. That is fine for ten types; it is painful for a hundred. The moment you need rich remote introspection, structured data handling, or an ecosystem module that already solves a weird &lt;span class="caps"&gt;API&lt;/span&gt;, Ansible starts to earn its weight&amp;nbsp;again.&lt;/p&gt;
&lt;p&gt;If you already have a working Ansible setup across your infrastructure, the &lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;jailexec connection plugin&lt;/a&gt; is the path of least resistance for adding jails to it. You keep your playbooks, your roles, your inventory. I still use it where &amp;#8220;use what&amp;#8217;s already there&amp;#8221; beats &amp;#8220;be&amp;nbsp;minimal&amp;#8221;.&lt;/p&gt;
&lt;p&gt;But for a fresh build of jails on a new host - or for the handful of single-purpose jails I run on radon - cdist plus these two wrappers is the right tool. It makes &amp;#8220;managed state on a FreeBSD jail&amp;#8221; exactly as simple as &amp;#8220;I can ssh to the host and&amp;nbsp;run &lt;code&gt;jexec&lt;/code&gt;&amp;#8221;, and it asks nothing more of the jail than FreeBSD already&amp;nbsp;provides.&lt;/p&gt;
&lt;h2 id="getting-the-scripts"&gt;Getting The&amp;nbsp;Scripts&lt;/h2&gt;
&lt;p&gt;Both scripts live on &lt;a href="https://codeberg.org/Larvitz/jexec-cdist"&gt;Codeberg as jexec-cdist&lt;/a&gt;, &lt;span class="caps"&gt;CC0&lt;/span&gt; licensed, with a &lt;span class="caps"&gt;README&lt;/span&gt; that walks through the environment variables, the ssh multiplexing tip that turns a 15-second run into an instant one,&amp;nbsp;the &lt;code&gt;doas&lt;/code&gt; pattern for running as a non-root user on the host, and the handful of edge cases (nullfs unions, non-standard jail roots) where the path rewriting needs a small&amp;nbsp;nudge.&lt;/p&gt;
&lt;p&gt;Clone, &lt;code&gt;chmod +x&lt;/code&gt;, point cdist at them,&amp;nbsp;export &lt;code&gt;JAIL_HOST&lt;/code&gt;. That is the entire onboarding. The jails stay as clean as the day you created them - and that is the thing I wanted all&amp;nbsp;along.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.cdi.st/"&gt;cdist&lt;/a&gt; - the configuration management system this article is&amp;nbsp;about&lt;/li&gt;
&lt;li&gt;&lt;a href="https://codeberg.org/Larvitz/jexec-cdist"&gt;jexec-cdist on Codeberg&lt;/a&gt; - the two wrappers in this&amp;nbsp;article&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.cdi.st/manual/latest/man7/cdist-remote-exec-copy.html"&gt;cdist: writing your own remote-exec and remote-copy&lt;/a&gt; - the upstream documentation for the hooks these scripts plug&amp;nbsp;into&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/"&gt;Managing FreeBSD Jails with Ansible: the jailexec connection plugin&lt;/a&gt; - the Ansible equivalent, for&amp;nbsp;comparison&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;FreeBSD Foundationals: Jails&lt;/a&gt; - the jail fundamentals this article&amp;nbsp;assumes&lt;/li&gt;
&lt;li&gt;&lt;code&gt;jexec(8)&lt;/code&gt;, &lt;code&gt;jls(8)&lt;/code&gt; - the FreeBSD primitives doing the actual&amp;nbsp;work&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="jails"/><category term="cdist"/><category term="automation"/><category term="devops"/><category term="unix-philosophy"/><category term="jexec"/></entry><entry><title>Podman on FreeBSD: OCI Containers Without systemd</title><link href="https://hofstede.it/podman-on-freebsd-oci-containers-without-systemd/" rel="alternate"/><published>2026-04-06T00:00:00+02:00</published><updated>2026-04-06T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-04-06:/podman-on-freebsd-oci-containers-without-systemd/</id><summary type="html">&lt;p&gt;Podman runs on FreeBSD too - but without systemd, the workflow is different. This follow-up to my Linux Podman deep dive covers how to run both native FreeBSD and Linux &lt;span class="caps"&gt;OCI&lt;/span&gt; containers on FreeBSD, how container lifecycle management works without Quadlets, and how Podman complements Jails rather than replacing&amp;nbsp;them.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;My &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;previous article&lt;/a&gt; covered Podman in depth on Linux - Quadlets, systemd integration, secrets management, auto-updates. That article was explicitly a Linux story. But Podman also runs on FreeBSD, and the experience is different enough to deserve its own&amp;nbsp;treatment.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Podman FreeBSD" src="https://hofstede.it/images/2026-04-06-podman-freebsd.png" title="Podman FreeBSD"&gt; &lt;/p&gt;
&lt;p&gt;FreeBSD&amp;#8217;s Podman support has been maturing steadily. The FreeBSD port is current and actively maintained, with a dedicated port maintainer who participates in upstream Podman development. That said, the port is still labeled experimental and intended for evaluation and testing - so treat it as a powerful option with some rough edges rather than a drop-in Linux-equivalent experience. The absence of systemd changes the operational model fundamentally. There are no Quadlets, no systemd timers, no journald integration. What you get instead is Podman&amp;#8217;s core container runtime integrated with FreeBSD&amp;#8217;s own service management, and the ability to run both native FreeBSD containers and Linux containers side by&amp;nbsp;side.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#installing-podman-on-freebsd"&gt;Installing Podman on FreeBSD&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#prerequisites"&gt;Prerequisites&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#enabling-the-service"&gt;Enabling the&amp;nbsp;Service&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#verification"&gt;Verification&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#native-freebsd-oci-containers"&gt;Native FreeBSD &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;Containers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#linux-containers-via-the-linuxulator"&gt;Linux Containers via the Linuxulator&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#prerequisites_1"&gt;Prerequisites&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#running-linux-images"&gt;Running Linux&amp;nbsp;Images&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#limitations"&gt;Limitations&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#container-lifecycle-without-systemd"&gt;Container Lifecycle Without systemd&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#podmans-built-in-restart-policy"&gt;Podman&amp;#8217;s Built-in Restart&amp;nbsp;Policy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rcd-service-scripts"&gt;rc.d Service&amp;nbsp;Scripts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-you-lose-without-quadlets"&gt;What You Lose Without&amp;nbsp;Quadlets&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#logging"&gt;Logging&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#image-updates"&gt;Image&amp;nbsp;Updates&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#podman-and-jails-complementary-not-competing"&gt;Podman and Jails: Complementary, Not&amp;nbsp;Competing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#practical-tips"&gt;Practical Tips&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#networking"&gt;Networking&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#storage"&gt;Storage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#secrets"&gt;Secrets&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#when-to-use-what"&gt;When to Use&amp;nbsp;What&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#wrapping-up"&gt;Wrapping&amp;nbsp;Up&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="installing-podman-on-freebsd"&gt;Installing Podman on&amp;nbsp;FreeBSD&lt;/h2&gt;
&lt;p&gt;Podman is available in the FreeBSD ports tree and as a binary&amp;nbsp;package:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;podman
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There&amp;#8217;s also&amp;nbsp;a &lt;code&gt;podman-suite&lt;/code&gt; meta package that pulls in additional tooling (Buildah, Skopeo) if you need a more complete container&amp;nbsp;workflow.&lt;/p&gt;
&lt;p&gt;This pulls in the Podman binary, its dependencies, and the necessary &lt;span class="caps"&gt;OCI&lt;/span&gt; runtime components. On FreeBSD, Podman relies on FreeBSD-specific runtime support, typically&amp;nbsp;involving &lt;code&gt;ocijail&lt;/code&gt; - a FreeBSD-native &lt;span class="caps"&gt;OCI&lt;/span&gt; runtime that leverages jails under the hood to provide the container isolation boundary, rather&amp;nbsp;than &lt;code&gt;runc&lt;/code&gt; or &lt;code&gt;crun&lt;/code&gt; from the Linux world. Container data lives&amp;nbsp;under &lt;code&gt;/var/db/containers/&lt;/code&gt; by default&amp;nbsp;(not &lt;code&gt;/var/lib/containers/&lt;/code&gt; as on&amp;nbsp;Linux).&lt;/p&gt;
&lt;h3 id="prerequisites"&gt;Prerequisites&lt;/h3&gt;
&lt;p&gt;Before Podman will work, you need two things beyond the package itself: fdescfs and &lt;span class="caps"&gt;PF&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;fdescfs&lt;/strong&gt; - Podman&amp;#8217;s container monitor&amp;nbsp;(&lt;code&gt;conmon&lt;/code&gt;)&amp;nbsp;needs &lt;code&gt;fdescfs(5)&lt;/code&gt; mounted&amp;nbsp;on &lt;code&gt;/dev/fd&lt;/code&gt; to properly support container restart policies. If it&amp;#8217;s not already&amp;nbsp;mounted:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mount&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;fdescfs&lt;span class="w"&gt; &lt;/span&gt;fdesc&lt;span class="w"&gt; &lt;/span&gt;/dev/fd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Make it permanent&amp;nbsp;in &lt;code&gt;/etc/fstab&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;fdesc   /dev/fd         fdescfs         rw      0       0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; firewall&lt;/strong&gt; - Container networking relies on &lt;span class="caps"&gt;NAT&lt;/span&gt; to route container traffic out to the host&amp;#8217;s network. This requires &lt;span class="caps"&gt;PF&lt;/span&gt;. The Podman package ships a sample&amp;nbsp;configuration:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cp&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/containers/pf.conf.sample&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Edit &lt;code&gt;/etc/pf.conf&lt;/code&gt; and set&amp;nbsp;the &lt;code&gt;v4egress_if&lt;/code&gt; and &lt;code&gt;v6egress_if&lt;/code&gt; variables to your network interface(s), then enable &lt;span class="caps"&gt;PF&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
service&lt;span class="w"&gt; &lt;/span&gt;pf&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;To support port redirections from the host to services inside containers&amp;nbsp;(e.g., &lt;code&gt;podman run -p 8080:80&lt;/code&gt;), you also need&amp;nbsp;the &lt;code&gt;pf&lt;/code&gt; kernel module loaded and local filtering&amp;nbsp;enabled:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;pf_load=&amp;quot;YES&amp;quot;&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.conf
kldload&lt;span class="w"&gt; &lt;/span&gt;pf
sysctl&lt;span class="w"&gt; &lt;/span&gt;net.pf.filter_local&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;net.pf.filter_local=1&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/etc/sysctl.conf.local
service&lt;span class="w"&gt; &lt;/span&gt;pf&lt;span class="w"&gt; &lt;/span&gt;restart
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The sample &lt;span class="caps"&gt;PF&lt;/span&gt; configuration includes the&amp;nbsp;necessary &lt;code&gt;nat-anchor "cni-rdr/*"&lt;/code&gt; rule for redirect support. If you&amp;#8217;re integrating into an existing &lt;span class="caps"&gt;PF&lt;/span&gt; ruleset rather than using the sample, make sure that anchor is&amp;nbsp;present.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Storage&lt;/strong&gt; - If your system uses &lt;span class="caps"&gt;ZFS&lt;/span&gt; (and it should), create a dedicated dataset for&amp;nbsp;Podman:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;mountpoint&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/var/db/containers&lt;span class="w"&gt; &lt;/span&gt;zroot/containers
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If &lt;span class="caps"&gt;ZFS&lt;/span&gt; is not available, change the storage driver&amp;nbsp;to &lt;code&gt;vfs&lt;/code&gt; in &lt;code&gt;/usr/local/etc/containers/storage.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-I&lt;span class="w"&gt; &lt;/span&gt;.bak&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;s/driver = &amp;quot;zfs&amp;quot;/driver = &amp;quot;vfs&amp;quot;/&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/containers/storage.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="enabling-the-service"&gt;Enabling the&amp;nbsp;Service&lt;/h3&gt;
&lt;p&gt;The FreeBSD ports package includes rc.d integration for Podman and its &lt;span class="caps"&gt;API&lt;/span&gt;/socket&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;podman_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
service&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;API&lt;/span&gt;/socket service is particularly useful when you need Docker-compatible socket access or &lt;span class="caps"&gt;API&lt;/span&gt;-driven tooling (like Traefik&amp;#8217;s Docker provider or &lt;span class="caps"&gt;CI&lt;/span&gt; runners). The daemonless-vs-service story on FreeBSD is less clean-cut than the usual Linux Podman model - the FreeBSD port exposes distinct service knobs&amp;nbsp;like &lt;code&gt;podman_enable&lt;/code&gt; and &lt;code&gt;podman_service_enable&lt;/code&gt;, and the exact role of the service in local &lt;span class="caps"&gt;CLI&lt;/span&gt; operations depends on your configuration. When in doubt, enable&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="verification"&gt;Verification&lt;/h3&gt;
&lt;p&gt;With everything in place, verify with the Podman hello-world&amp;nbsp;image:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;quay.io/dougrabson/hello
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you see the Podman mascot, you&amp;#8217;re&amp;nbsp;good.&lt;/p&gt;
&lt;h2 id="native-freebsd-oci-containers"&gt;Native FreeBSD &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;Containers&lt;/h2&gt;
&lt;p&gt;This is the part that surprises people: Podman on FreeBSD can run containers built from native FreeBSD base images. These aren&amp;#8217;t Linux containers running through a compatibility layer - they&amp;#8217;re actual FreeBSD userland processes running in jail-based&amp;nbsp;isolation.&lt;/p&gt;
&lt;p&gt;The FreeBSD community maintains &lt;span class="caps"&gt;OCI&lt;/span&gt; images specifically for this&amp;nbsp;purpose:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;-it&lt;span class="w"&gt; &lt;/span&gt;docker.io/freebsd/freebsd-runtime:15.0&lt;span class="w"&gt; &lt;/span&gt;freebsd-version&lt;span class="w"&gt; &lt;/span&gt;-u
&lt;span class="m"&gt;15&lt;/span&gt;.0-RELEASE
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Inside that container, you&amp;#8217;re in FreeBSD userland&amp;nbsp;- &lt;code&gt;freebsd-version&lt;/code&gt; reports a real FreeBSD release, and standard FreeBSD tooling works as you&amp;#8217;d expect from the image contents. You can build and run FreeBSD software natively, with &lt;span class="caps"&gt;OCI&lt;/span&gt; image packaging and&amp;nbsp;distribution.&lt;/p&gt;
&lt;p&gt;Available base images&amp;nbsp;include:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;freebsd/freebsd-runtime&lt;/code&gt;&lt;/strong&gt; - minimal FreeBSD userland, suitable for running pre-built&amp;nbsp;binaries&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;freebsd/freebsd&lt;/code&gt;&lt;/strong&gt; - fuller base with development tools, suitable for building software inside the&amp;nbsp;container&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;You can build your own FreeBSD &lt;span class="caps"&gt;OCI&lt;/span&gt; images with a&amp;nbsp;standard &lt;code&gt;Containerfile&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;docker.io/freebsd/freebsd-runtime:15.0&lt;/span&gt;

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;nginx
&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;nginx.conf&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/nginx/nginx.conf
&lt;span class="k"&gt;EXPOSE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;80&lt;/span&gt;

&lt;span class="k"&gt;CMD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;nginx&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-g&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;daemon off;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Build and run it like any &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;image:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;my-freebsd-nginx&lt;span class="w"&gt; &lt;/span&gt;.
podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8080&lt;/span&gt;:80&lt;span class="w"&gt; &lt;/span&gt;my-freebsd-nginx
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is native performance - no emulation, no translation layer. The container process runs as a FreeBSD process inside a jail, with &lt;span class="caps"&gt;OCI&lt;/span&gt;-standard image packaging on&amp;nbsp;top.&lt;/p&gt;
&lt;h2 id="linux-containers-via-the-linuxulator"&gt;Linux Containers via the&amp;nbsp;Linuxulator&lt;/h2&gt;
&lt;p&gt;FreeBSD&amp;#8217;s Linux binary compatibility layer (the Linuxulator) extends to Podman.&amp;nbsp;With &lt;code&gt;linux64.ko&lt;/code&gt; loaded, Podman can run Linux &lt;span class="caps"&gt;OCI&lt;/span&gt; images - the same images you&amp;#8217;d pull from Docker Hub for any Linux-based&amp;nbsp;deployment.&lt;/p&gt;
&lt;h3 id="prerequisites_1"&gt;Prerequisites&lt;/h3&gt;
&lt;p&gt;Load the Linux kernel module on the&amp;nbsp;host:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Persistent across reboots&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;linux_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;linux64_load=&amp;quot;YES&amp;quot;&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/boot/loader.conf

&lt;span class="c1"&gt;# Load immediately&lt;/span&gt;
kldload&lt;span class="w"&gt; &lt;/span&gt;linux64
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you&amp;#8217;ve followed my &lt;a href="https://hofstede.it/running-a-factorio-headless-server-on-freebsd-with-the-linuxulator/"&gt;Factorio on FreeBSD&lt;/a&gt; article, this is the same Linuxulator setup - just applied to &lt;span class="caps"&gt;OCI&lt;/span&gt; containers instead of standalone&amp;nbsp;binaries.&lt;/p&gt;
&lt;h3 id="running-linux-images"&gt;Running Linux&amp;nbsp;Images&lt;/h3&gt;
&lt;p&gt;With the Linuxulator loaded, pulling and running Linux images works as you&amp;#8217;d&amp;nbsp;expect:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;--os&lt;span class="o"&gt;=&lt;/span&gt;linux&lt;span class="w"&gt; &lt;/span&gt;docker.io/library/alpine&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/etc/os-release&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;head&lt;span class="w"&gt; &lt;/span&gt;-1
&lt;span class="nv"&gt;NAME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Alpine Linux&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;--os=linux&lt;/code&gt; flag tells Podman to select the Linux variant of a multi-platform image. Inside the container, the process sees a Linux environment - glibc calls work, Linux-specific syscalls are translated by the&amp;nbsp;Linuxulator.&lt;/p&gt;
&lt;p&gt;This means many Linux-oriented &lt;span class="caps"&gt;OCI&lt;/span&gt; images can run on FreeBSD without modification, especially userland-heavy workloads that don&amp;#8217;t depend on Linux-specific kernel facilities. Your existing application server images, many database images, and standard web tooling are good&amp;nbsp;candidates.&lt;/p&gt;
&lt;h3 id="limitations"&gt;Limitations&lt;/h3&gt;
&lt;p&gt;One important practical difference from Linux is that current FreeBSD Podman deployments are typically run as root. Rootless Podman is not part of the FreeBSD workflow today, so the threat model is different from Linux setups where rootless operation is a major security&amp;nbsp;feature.&lt;/p&gt;
&lt;p&gt;The Linuxulator itself is remarkably complete, but it&amp;#8217;s not&amp;nbsp;perfect:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Syscall coverage&lt;/strong&gt;: The Linuxulator implements a large subset of Linux syscalls, but some less common ones may be missing. Most mainstream software works fine; highly Linux-specific tools (those that depend on cgroups v2 internals, eBPF, or&amp;nbsp;kernel-specific &lt;code&gt;/proc&lt;/code&gt; and &lt;code&gt;/sys&lt;/code&gt; interfaces) may&amp;nbsp;not.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Performance&lt;/strong&gt;: Native FreeBSD containers have zero translation overhead. Linux containers go through the Linuxulator&amp;#8217;s syscall translation, which adds some overhead - negligible for I/O-bound workloads, potentially noticeable for syscall-heavy&amp;nbsp;applications.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Kernel features&lt;/strong&gt;: Linux containers on FreeBSD don&amp;#8217;t have access to Linux-specific kernel features like cgroups (FreeBSD uses its own resource limiting&amp;nbsp;via &lt;code&gt;rctl&lt;/code&gt;), seccomp, or kernel namespaces. Isolation is provided by FreeBSD jails&amp;nbsp;instead.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="container-lifecycle-without-systemd"&gt;Container Lifecycle Without&amp;nbsp;systemd&lt;/h2&gt;
&lt;p&gt;Here&amp;#8217;s where the operational model diverges most from Linux. On Linux, Quadlets turn containers into systemd services with dependency management, restart policies, logging integration, and boot startup - all through the init system. FreeBSD doesn&amp;#8217;t have systemd, so none of that&amp;nbsp;exists.&lt;/p&gt;
&lt;p&gt;Instead, you have two&amp;nbsp;approaches.&lt;/p&gt;
&lt;h3 id="podmans-built-in-restart-policy"&gt;Podman&amp;#8217;s Built-in Restart&amp;nbsp;Policy&lt;/h3&gt;
&lt;p&gt;The simplest method.&amp;nbsp;Podman&amp;#8217;s &lt;code&gt;--restart&lt;/code&gt; flag works on FreeBSD just as it does on&amp;nbsp;Linux:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;my-app&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--restart&lt;span class="w"&gt; &lt;/span&gt;always&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8080&lt;/span&gt;:80&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;docker.io/library/nginx:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With &lt;code&gt;podman_enable=YES&lt;/code&gt; in &lt;code&gt;/etc/rc.conf&lt;/code&gt;, the Podman service starts at boot, and containers&amp;nbsp;with &lt;code&gt;--restart=always&lt;/code&gt; are automatically restarted. This gives you basic lifecycle management without writing any service&amp;nbsp;scripts.&lt;/p&gt;
&lt;p&gt;The restart policies are the same as&amp;nbsp;Docker&amp;#8217;s:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Policy&lt;/th&gt;
&lt;th&gt;Behavior&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;no&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Never restart (default)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;always&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Always restart, including at boot&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;on-failure[:max]&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Restart on non-zero exit, optional retry limit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;unless-stopped&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Like &lt;code&gt;always&lt;/code&gt;, but not if manually stopped&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For simple deployments, this is often sufficient. The container starts at boot, restarts on crash, and Podman handles the&amp;nbsp;lifecycle.&lt;/p&gt;
&lt;h3 id="rcd-service-scripts"&gt;rc.d Service&amp;nbsp;Scripts&lt;/h3&gt;
&lt;p&gt;For more control - dependency ordering, custom health checks, integration with FreeBSD&amp;#8217;s service framework - write an rc.d script. This is FreeBSD&amp;#8217;s native equivalent of a systemd unit&amp;nbsp;file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# PROVIDE: myapp&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: DAEMON podman&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;

.&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.subr

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;myapp&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rcvar&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;myapp_enable

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;

:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_enable&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_image&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;docker.io/library/nginx:latest&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_name&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;myapp&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;

&lt;span class="nv"&gt;start_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_start&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;stop_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_stop&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;status_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_status&amp;quot;&lt;/span&gt;

myapp_start&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Starting &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Clean up any orphaned container from a dirty shutdown&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--restart&lt;span class="w"&gt; &lt;/span&gt;on-failure:5&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8080&lt;/span&gt;:80&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_image&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

myapp_stop&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Stopping &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

myapp_status&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{{.State.Status}}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;myapp_name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; is not running.&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Save this&amp;nbsp;as &lt;code&gt;/usr/local/etc/rc.d/myapp&lt;/code&gt;, make it executable, and enable&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;+x&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/rc.d/myapp
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;myapp_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
service&lt;span class="w"&gt; &lt;/span&gt;myapp&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;# REQUIRE: DAEMON podman&lt;/code&gt; line ensures the Podman service is running before your container starts. You can chain dependencies between container services the same&amp;nbsp;way:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# In your database service script&lt;/span&gt;
&lt;span class="c1"&gt;# PROVIDE: myapp_db&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: DAEMON podman&lt;/span&gt;

&lt;span class="c1"&gt;# In your application service script&lt;/span&gt;
&lt;span class="c1"&gt;# PROVIDE: myapp_web&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: myapp_db&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This gives you explicit startup ordering through&amp;nbsp;FreeBSD&amp;#8217;s &lt;code&gt;rcorder&lt;/code&gt; system - the same mechanism that orders every other service on the&amp;nbsp;system.&lt;/p&gt;
&lt;h3 id="what-you-lose-without-quadlets"&gt;What You Lose Without&amp;nbsp;Quadlets&lt;/h3&gt;
&lt;p&gt;Let&amp;#8217;s be direct about the trade-offs. Compared to the Quadlet workflow on&amp;nbsp;Linux:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Capability&lt;/th&gt;
&lt;th&gt;Linux (Quadlets)&lt;/th&gt;
&lt;th&gt;FreeBSD (rc.d / restart policy)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Declarative container definition&lt;/td&gt;
&lt;td&gt;&lt;code&gt;.container&lt;/code&gt; files&lt;/td&gt;
&lt;td&gt;&lt;code&gt;podman run&lt;/code&gt; flags in scripts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dependency management&lt;/td&gt;
&lt;td&gt;systemd &lt;code&gt;After=&lt;/code&gt;, &lt;code&gt;Requires=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;rcorder &lt;code&gt;REQUIRE:&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Log integration&lt;/td&gt;
&lt;td&gt;journald&amp;nbsp;via &lt;code&gt;journalctl -u&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;stdout/stderr to files, or syslog&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Auto-updates&lt;/td&gt;
&lt;td&gt;&lt;code&gt;podman auto-update&lt;/code&gt; with systemd timer&lt;/td&gt;
&lt;td&gt;Custom scripting (pull + recreate via cron)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Health checks driving restarts&lt;/td&gt;
&lt;td&gt;&lt;code&gt;HealthOnFailure=stop&lt;/code&gt; + systemd restart&lt;/td&gt;
&lt;td&gt;Custom scripting&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Resource limits&lt;/td&gt;
&lt;td&gt;systemd cgroup directives&lt;/td&gt;
&lt;td&gt;&lt;code&gt;rctl&lt;/code&gt; rules on the jail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secrets injection&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Secret=&lt;/code&gt; directive&lt;/td&gt;
&lt;td&gt;&lt;code&gt;podman secret&lt;/code&gt; &lt;span class="caps"&gt;CLI&lt;/span&gt; (works the same)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The Quadlet workflow is genuinely more ergonomic for complex multi-container deployments. FreeBSD&amp;#8217;s approach requires more manual wiring. But it&amp;#8217;s the same tools FreeBSD administrators already use for everything else on the system - rc.d,&amp;nbsp;cron, &lt;code&gt;rctl&lt;/code&gt; - so the operational patterns are familiar even if they&amp;#8217;re less&amp;nbsp;integrated.&lt;/p&gt;
&lt;h3 id="logging"&gt;Logging&lt;/h3&gt;
&lt;p&gt;Without journald, container logs go to Podman&amp;#8217;s log driver. By default, Podman stores logs per container under its local container storage. You access them the usual&amp;nbsp;way:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;logs&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;my-app
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For centralized logging, configure the syslog log&amp;nbsp;driver:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--log-driver&lt;span class="w"&gt; &lt;/span&gt;syslog&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--log-opt&lt;span class="w"&gt; &lt;/span&gt;syslog-facility&lt;span class="o"&gt;=&lt;/span&gt;local0&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;my-app&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;docker.io/library/nginx:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This sends container output to FreeBSD&amp;#8217;s syslogd, where it integrates with your existing log infrastructure - rotation&amp;nbsp;via &lt;code&gt;newsyslog.conf&lt;/code&gt;, forwarding to a central syslog server, or whatever your setup&amp;nbsp;uses.&lt;/p&gt;
&lt;h3 id="image-updates"&gt;Image&amp;nbsp;Updates&lt;/h3&gt;
&lt;p&gt;Podman&amp;#8217;s&amp;nbsp;documented &lt;code&gt;auto-update&lt;/code&gt; workflow is systemd-centric - it&amp;#8217;s designed for containers running inside systemd units, and after pulling a newer image it restarts the systemd unit managing the container. The timer that triggers the check is only one piece; the restart/recreation mechanism itself depends on systemd. That entire chain doesn&amp;#8217;t exist on&amp;nbsp;FreeBSD.&lt;/p&gt;
&lt;p&gt;What you can do is script the image refresh and container recreation yourself. A cron job can check for newer images and log the&amp;nbsp;results:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/cron.d/podman-image-refresh&lt;/span&gt;
&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;*&lt;span class="w"&gt; &lt;/span&gt;root&lt;span class="w"&gt; &lt;/span&gt;/usr/local/sbin/podman-refresh.sh&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;logger&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;podman-refresh
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;But unlike the Linux auto-update flow, you need to handle the restart and recreation step yourself - pulling the new image, stopping the old container, and starting a fresh one with the updated image. A simple&amp;nbsp;approach:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# /usr/local/sbin/podman-refresh.sh&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;traefik&lt;span class="w"&gt; &lt;/span&gt;myapp&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{{.ImageName}}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;continue&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$image&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-q&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Writing manifest&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# Re-create via your rc.d script or however the container is defined&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;service&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;start
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is more manual than the Linux Quadlet auto-update experience, and it requires that your rc.d scripts or startup commands can fully recreate the container from scratch. It&amp;#8217;s operational overhead worth acknowledging: on Linux, Podman and systemd handle this loop for you; on FreeBSD, you own&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="podman-and-jails-complementary-not-competing"&gt;Podman and Jails: Complementary, Not&amp;nbsp;Competing&lt;/h2&gt;
&lt;p&gt;This is the important framing. FreeBSD already has the most mature &lt;span class="caps"&gt;OS&lt;/span&gt;-level container technology in existence: &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;Jails&lt;/a&gt;. They&amp;#8217;ve been around since FreeBSD 4.0, they provide kernel-enforced isolation, and they&amp;#8217;re deeply integrated with the rest of the operating system. So why would you want&amp;nbsp;Podman?&lt;/p&gt;
&lt;p&gt;The answer isn&amp;#8217;t &amp;#8220;Podman is better than Jails.&amp;#8221; It&amp;#8217;s that they solve different&amp;nbsp;problems:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Jails&lt;/strong&gt; are FreeBSD&amp;#8217;s native isolation primitive. A jail is essentially a lightweight FreeBSD instance with its own filesystem, network stack, process space, and user database. Jails are ideal&amp;nbsp;for:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Long-lived services that you build and maintain&amp;nbsp;yourself&lt;/li&gt;
&lt;li&gt;Workloads that benefit from direct access to FreeBSD&amp;#8217;s kernel features (&lt;span class="caps"&gt;ZFS&lt;/span&gt;, &lt;span class="caps"&gt;PF&lt;/span&gt;, &lt;code&gt;rctl&lt;/code&gt;,&amp;nbsp;DTrace)&lt;/li&gt;
&lt;li&gt;Environments where you want full FreeBSD package management inside the isolated&amp;nbsp;environment&lt;/li&gt;
&lt;li&gt;Multi-tenant hosting where each tenant gets a complete FreeBSD&amp;nbsp;environment&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Podman&lt;/strong&gt; brings &lt;span class="caps"&gt;OCI&lt;/span&gt; container compatibility. It&amp;#8217;s ideal&amp;nbsp;for:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Running third-party software distributed as &lt;span class="caps"&gt;OCI&lt;/span&gt;/Docker images without&amp;nbsp;repackaging&lt;/li&gt;
&lt;li&gt;Linux-only applications that have no FreeBSD port (via the&amp;nbsp;Linuxulator)&lt;/li&gt;
&lt;li&gt;Workflows that already use Dockerfiles and container&amp;nbsp;registries&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;CI&lt;/span&gt;/&lt;span class="caps"&gt;CD&lt;/span&gt; pipelines that produce &lt;span class="caps"&gt;OCI&lt;/span&gt; images as&amp;nbsp;artifacts&lt;/li&gt;
&lt;li&gt;Development environments that need to match Linux production&amp;nbsp;targets&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The practical scenario: you run your core infrastructure - your reverse proxy, your databases, your custom applications - in Jails, managed by Bastille or your preferred jail manager, with &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots, &lt;span class="caps"&gt;VNET&lt;/span&gt; networking, and &lt;span class="caps"&gt;PF&lt;/span&gt; firewall rules. Then a project requires a specific application that&amp;#8217;s only distributed as a Docker image, or your &lt;span class="caps"&gt;CI&lt;/span&gt; pipeline produces &lt;span class="caps"&gt;OCI&lt;/span&gt; images that need to run somewhere. Instead of setting up a Linux &lt;span class="caps"&gt;VM&lt;/span&gt; or wrestling with ports and dependencies, you run that image with Podman. The Jail handles what Jails do best; Podman handles what the &lt;span class="caps"&gt;OCI&lt;/span&gt; ecosystem does&amp;nbsp;best.&lt;/p&gt;
&lt;p&gt;In theory, Podman-inside-a-Jail sounds attractive for defense in depth, but I would treat it as experimental at best rather than a recommended deployment&amp;nbsp;pattern.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a jail for container workloads&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;containers&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;14&lt;/span&gt;.3-RELEASE&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.254.252.50&lt;span class="w"&gt; &lt;/span&gt;bastille0

&lt;span class="c1"&gt;# Install Podman inside the jail&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;containers&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;podman

&lt;span class="c1"&gt;# Enable and start Podman inside the jail&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;containers&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;podman_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
bastille&lt;span class="w"&gt; &lt;/span&gt;service&lt;span class="w"&gt; &lt;/span&gt;containers&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;start

&lt;span class="c1"&gt;# Run OCI containers inside the jail&lt;/span&gt;
bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;containers&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;myapp&lt;span class="w"&gt; &lt;/span&gt;docker.io/library/nginx:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This gives you defense-in-depth: the jail provides network isolation and resource limits at the FreeBSD kernel level, while Podman handles the &lt;span class="caps"&gt;OCI&lt;/span&gt; runtime&amp;nbsp;inside.&lt;/p&gt;
&lt;h2 id="practical-tips"&gt;Practical&amp;nbsp;Tips&lt;/h2&gt;
&lt;h3 id="networking"&gt;Networking&lt;/h3&gt;
&lt;p&gt;Podman networking on FreeBSD follows the same broad model as on Linux - user-defined networks, container name resolution, and host port publishing - but the backend may differ by port version and build configuration. Current FreeBSD ports have been observed using &lt;span class="caps"&gt;CNI&lt;/span&gt; rather than netavark. Check what your install is using&amp;nbsp;with &lt;code&gt;podman info&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;info&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{{.Host.NetworkBackend}}&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Regardless of backend, the operational patterns are the&amp;nbsp;same:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a network&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;network&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;mynet

&lt;span class="c1"&gt;# Run containers on that network&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;--network&lt;span class="w"&gt; &lt;/span&gt;mynet&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;db&lt;span class="w"&gt; &lt;/span&gt;postgres:17
podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;--network&lt;span class="w"&gt; &lt;/span&gt;mynet&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;app&lt;span class="w"&gt; &lt;/span&gt;myapp:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;DNS&lt;/span&gt; resolution between containers on the same network works by container&amp;nbsp;name.&lt;/p&gt;
&lt;p&gt;Remember that all of this depends on the &lt;span class="caps"&gt;PF&lt;/span&gt; setup from the installation section. Without &lt;span class="caps"&gt;PF&lt;/span&gt; doing &lt;span class="caps"&gt;NAT&lt;/span&gt;, container traffic has no route out of the container network. Port redirections&amp;nbsp;(&lt;code&gt;-p&lt;/code&gt; flag) flow through&amp;nbsp;the &lt;code&gt;cni-rdr&lt;/code&gt; &lt;span class="caps"&gt;PF&lt;/span&gt; anchors. If you&amp;#8217;re already running &lt;span class="caps"&gt;PF&lt;/span&gt; with your own ruleset (and you probably are if you&amp;#8217;ve read my &lt;a href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; guide&lt;/a&gt;), integrate the Podman anchors into your existing configuration rather than replacing it with the sample&amp;nbsp;file.&lt;/p&gt;
&lt;h3 id="storage"&gt;Storage&lt;/h3&gt;
&lt;p&gt;With the &lt;span class="caps"&gt;ZFS&lt;/span&gt; dataset from the installation section in place, Podman automatically uses the &lt;span class="caps"&gt;ZFS&lt;/span&gt; storage driver. This is one area where Podman on FreeBSD genuinely outshines the Linux experience: instead of fighting with overlayfs quirks or fuse-overlayfs for rootless setups, Podman creates real &lt;span class="caps"&gt;ZFS&lt;/span&gt; datasets for container image layers. You get copy-on-write, compression, checksumming, and snapshot support for free, all backed by the same storage stack you&amp;#8217;re already using for everything else on the system. You can snapshot the entire container storage tree, roll back after a bad update, or send it to another host&amp;nbsp;with &lt;code&gt;zfs send&lt;/code&gt; - all operations that feel natural on FreeBSD and awkward on&amp;nbsp;Linux.&lt;/p&gt;
&lt;h3 id="secrets"&gt;Secrets&lt;/h3&gt;
&lt;p&gt;Podman&amp;#8217;s secret store works identically on&amp;nbsp;FreeBSD:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pwgen&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;db_password&lt;span class="w"&gt; &lt;/span&gt;-
podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;--secret&lt;span class="w"&gt; &lt;/span&gt;db_password,type&lt;span class="o"&gt;=&lt;/span&gt;env,target&lt;span class="o"&gt;=&lt;/span&gt;DB_PASS&lt;span class="w"&gt; &lt;/span&gt;myapp:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;No differences here - the secrets management story from the &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/#secrets-management"&gt;Linux article&lt;/a&gt; applies&amp;nbsp;as-is.&lt;/p&gt;
&lt;h2 id="when-to-use-what"&gt;When to Use&amp;nbsp;What&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Recommended approach&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Custom FreeBSD service, full &lt;span class="caps"&gt;OS&lt;/span&gt; access needed&lt;/td&gt;
&lt;td&gt;Jail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Third-party app only available as Docker image&lt;/td&gt;
&lt;td&gt;Podman&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Linux-only software with no FreeBSD port&lt;/td&gt;
&lt;td&gt;Podman + Linuxulator&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Multi-tenant isolation with independent networks&lt;/td&gt;
&lt;td&gt;Jail (&lt;span class="caps"&gt;VNET&lt;/span&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;CI&lt;/span&gt;/&lt;span class="caps"&gt;CD&lt;/span&gt; pipeline producing &lt;span class="caps"&gt;OCI&lt;/span&gt; artifacts&lt;/td&gt;
&lt;td&gt;Podman&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Database with &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots for backup&lt;/td&gt;
&lt;td&gt;Jail (direct &lt;span class="caps"&gt;ZFS&lt;/span&gt; access)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Development parity with Linux production&lt;/td&gt;
&lt;td&gt;Podman + Linux containers&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mix of both needs&lt;/td&gt;
&lt;td&gt;Podman inside a Jail&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="wrapping-up"&gt;Wrapping&amp;nbsp;Up&lt;/h2&gt;
&lt;p&gt;Podman on FreeBSD isn&amp;#8217;t trying to replace Jails. It&amp;#8217;s filling a gap that Jails were never designed to fill: compatibility with the &lt;span class="caps"&gt;OCI&lt;/span&gt; container ecosystem. The ability&amp;nbsp;to &lt;code&gt;podman pull&lt;/code&gt; an image from Docker Hub and run it on FreeBSD - whether it&amp;#8217;s a native FreeBSD image or a Linux image through the Linuxulator - means you don&amp;#8217;t have to choose between FreeBSD&amp;#8217;s operational model and the container&amp;nbsp;ecosystem.&lt;/p&gt;
&lt;p&gt;The operational workflow is less polished than on Linux. Without systemd, you lose Quadlets and the tight init-system integration that makes Podman on Linux so compelling. But what you get in return is Podman integrated with FreeBSD&amp;#8217;s own tools: rc.d for service management, cron for&amp;nbsp;scheduling, &lt;code&gt;rctl&lt;/code&gt; for resource limits, &lt;span class="caps"&gt;PF&lt;/span&gt; for network policy, &lt;span class="caps"&gt;ZFS&lt;/span&gt; for storage. Different tools, same&amp;nbsp;result.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re running FreeBSD and you&amp;#8217;ve been eyeing the &lt;span class="caps"&gt;OCI&lt;/span&gt; ecosystem from the outside, Podman is your bridge&amp;nbsp;in.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://podman.io/docs/installation#installing-on-freebsd"&gt;Podman Installation: FreeBSD&lt;/a&gt; - official installation&amp;nbsp;guide&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/containers/podman/blob/main/docs/tutorials/podman-for-freebsd.md"&gt;Podman on FreeBSD - containers/podman&amp;nbsp;Wiki&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hub.docker.com/u/freebsd"&gt;FreeBSD &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;Images&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/linuxemu/"&gt;FreeBSD Handbook: Linux Binary&amp;nbsp;Compatibility&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/nicr9/ocijail"&gt;ocijail - FreeBSD &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;Runtime&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman in Production: Quadlets, Secrets, Auto-Updates, and Docker Compatibility&lt;/a&gt; - the Linux-focused companion&amp;nbsp;article&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;FreeBSD Foundationals: Jails&lt;/a&gt; - deep dive into FreeBSD&amp;#8217;s native&amp;nbsp;isolation&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/running-a-factorio-headless-server-on-freebsd-with-the-linuxulator/"&gt;Running Factorio on FreeBSD with the Linuxulator&lt;/a&gt; - another Linuxulator use&amp;nbsp;case&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="podman"/><category term="containers"/><category term="oci"/><category term="jails"/><category term="linuxulator"/><category term="docker"/></entry><entry><title>Dual-FIB Policy Routing on FreeBSD: Two Upstreams, One Server, Zero Confusion</title><link href="https://hofstede.it/dual-fib-policy-routing-on-freebsd-two-upstreams-one-server-zero-confusion/" rel="alternate"/><published>2026-03-23T00:00:00+01:00</published><updated>2026-03-23T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-03-23:/dual-fib-policy-routing-on-freebsd-two-upstreams-one-server-zero-confusion/</id><summary type="html">&lt;p&gt;How to run a FreeBSD server with two completely independent internet uplinks - a physical provider and a &lt;span class="caps"&gt;BGP&lt;/span&gt; tunnel - using dual-&lt;span class="caps"&gt;FIB&lt;/span&gt; routing tables, &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s rtable and reply-to directives, and a single bridge that carries &lt;span class="caps"&gt;NAT&lt;/span&gt;&amp;#8217;d, routed, and pure public jail traffic&amp;nbsp;simultaneously.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Logo" src="https://hofstede.it/images/2026-03-23-freebsd-dual-fib-routing-two-upstreams.png" title="Dual-FIB Routing: Header image"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-problem-one-server-two-gateways"&gt;The Problem: One Server, Two&amp;nbsp;Gateways&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#architecture-overview"&gt;Architecture&amp;nbsp;Overview&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-physical-upstream-netcup-fib-0"&gt;The Physical Upstream: Netcup (&lt;span class="caps"&gt;FIB&lt;/span&gt;&amp;nbsp;0)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-logical-upstream-bgp-tunnel-fib-1"&gt;The Logical Upstream: &lt;span class="caps"&gt;BGP&lt;/span&gt; Tunnel (&lt;span class="caps"&gt;FIB&lt;/span&gt;&amp;nbsp;1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#fib-1-building-a-complete-routing-table"&gt;&lt;span class="caps"&gt;FIB&lt;/span&gt; 1: Building a Complete Routing&amp;nbsp;Table&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#jail-networking-and-the-bridge"&gt;Jail Networking and the&amp;nbsp;Bridge&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#pf-where-the-routing-decision-happens"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt;: Where the Routing Decision Happens&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#normalization"&gt;Normalization&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#translation-nat-and-rdr"&gt;Translation (&lt;span class="caps"&gt;NAT&lt;/span&gt; and &lt;span class="caps"&gt;RDR&lt;/span&gt;)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#default-policies"&gt;Default&amp;nbsp;Policies&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#egress-the-route-to-safety-net"&gt;Egress: The route-to Safety&amp;nbsp;Net&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#egress-jail-policy-routing-via-rtable"&gt;Egress: Jail Policy Routing via&amp;nbsp;rtable&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#ingress-tunnel-encapsulation"&gt;Ingress: Tunnel&amp;nbsp;Encapsulation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#ingress-the-reply-to-imperative"&gt;Ingress: The reply-to&amp;nbsp;Imperative&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-pure-public-vnet-jail"&gt;The Pure Public &lt;span class="caps"&gt;VNET&lt;/span&gt;&amp;nbsp;Jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-complete-flow"&gt;The Complete Flow&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#inbound-to-a-bgp-addressed-service-ipv6"&gt;Inbound to a &lt;span class="caps"&gt;BGP&lt;/span&gt;-addressed service&amp;nbsp;(IPv6)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#outbound-from-a-pure-public-jail-ipv4"&gt;Outbound from a pure public jail&amp;nbsp;(IPv4)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#natd-jail-egress-ipv4-standard-path"&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt;&amp;#8217;d jail egress (IPv4, standard&amp;nbsp;path)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#verification"&gt;Verification&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#from-the-host-testing-both-fibs"&gt;From the host: testing both&amp;nbsp;FIBs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#from-inside-the-pure-public-jail"&gt;From inside the pure public&amp;nbsp;jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#from-external-clients-both-paths-work"&gt;From external clients: both paths&amp;nbsp;work&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#security-considerations"&gt;Security&amp;nbsp;Considerations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#lessons-learned"&gt;Lessons&amp;nbsp;Learned&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;The &lt;a href="https://hofstede.it/running-your-own-as-bgp-on-freebsd-with-frr-gre-tunnels-and-policy-routing/"&gt;previous articles&lt;/a&gt; in this series covered the &lt;span class="caps"&gt;BGP&lt;/span&gt; router side: obtaining an &lt;span class="caps"&gt;AS&lt;/span&gt;, peering with upstream providers, and building a tunnel overlay. This article covers the other end - the downstream server that consumes that tunnel and needs to serve traffic from two completely different &lt;span class="caps"&gt;IP&lt;/span&gt; ranges through two completely different paths, simultaneously, without either one interfering with the&amp;nbsp;other.&lt;/p&gt;
&lt;p&gt;The server in question&amp;nbsp;is &lt;code&gt;radon&lt;/code&gt;, a Netcup &lt;span class="caps"&gt;VPS&lt;/span&gt; running FreeBSD with Bastille jails. It has two internet uplinks: a physical connection to Netcup&amp;#8217;s network and a &lt;span class="caps"&gt;GIF&lt;/span&gt; tunnel to my &lt;span class="caps"&gt;BGP&lt;/span&gt; router&amp;nbsp;(&lt;code&gt;hobgp&lt;/code&gt;). Jails on this server use three distinct routing paradigms - private &lt;span class="caps"&gt;NAT&lt;/span&gt;, natively routed &lt;span class="caps"&gt;BGP&lt;/span&gt; IPv6, and pure public routed &lt;span class="caps"&gt;BGP&lt;/span&gt; IPv4 - all on the same bridge&amp;nbsp;interface.&lt;/p&gt;
&lt;p&gt;The mechanism that makes this work is &lt;strong&gt;dual-&lt;span class="caps"&gt;FIB&lt;/span&gt; policy routing&lt;/strong&gt;: two independent routing tables in the kernel, with &lt;span class="caps"&gt;PF&lt;/span&gt; deciding which table handles which traffic based on source address. It&amp;#8217;s elegant once you understand it, and surprisingly simple to configure once you&amp;#8217;ve seen it&amp;nbsp;done.&lt;/p&gt;
&lt;h2 id="the-problem-one-server-two-gateways"&gt;The Problem: One Server, Two&amp;nbsp;Gateways&lt;/h2&gt;
&lt;p&gt;A server with a single default gateway has simple routing: everything goes out the same door. But what happens when you want traffic from different source addresses to take different&amp;nbsp;paths?&lt;/p&gt;
&lt;p&gt;On &lt;code&gt;radon&lt;/code&gt;, the physical interface&amp;nbsp;(&lt;code&gt;vtnet0&lt;/code&gt;) is assigned Netcup&amp;#8217;s &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;space: &lt;code&gt;152.53.147.5&lt;/code&gt; for IPv4&amp;nbsp;and &lt;code&gt;2a0a:4cc0:c1:2f90::2&lt;/code&gt; for IPv6. The default gateway points to Netcup&amp;#8217;s router. Standard traffic - package updates, &lt;span class="caps"&gt;DNS&lt;/span&gt; queries, NATed jail traffic - flows through this&amp;nbsp;path.&lt;/p&gt;
&lt;p&gt;But &lt;code&gt;radon&lt;/code&gt; also has addresses from my own &lt;span class="caps"&gt;AS201379&lt;/span&gt;: &lt;code&gt;194.28.98.217&lt;/code&gt; (shared via loopback for port&amp;nbsp;forwarding), &lt;code&gt;194.28.98.216&lt;/code&gt; (directly assigned to a jail), and the&amp;nbsp;entire &lt;code&gt;2a06:9801:1c:1000::/64&lt;/code&gt; IPv6 subnet routed to jails. This address space is announced to the internet through my &lt;span class="caps"&gt;BGP&lt;/span&gt; router. Traffic arriving for these addresses traverses the &lt;span class="caps"&gt;BGP&lt;/span&gt; tunnel. If replies to that traffic exit through Netcup&amp;#8217;s default gateway instead of the tunnel, the provider drops them as spoofed - the source &lt;span class="caps"&gt;IP&lt;/span&gt; doesn&amp;#8217;t belong to Netcup&amp;#8217;s&amp;nbsp;network.&lt;/p&gt;
&lt;p&gt;The kernel&amp;#8217;s single routing table (&lt;span class="caps"&gt;FIB&lt;/span&gt; 0) knows one default route: Netcup. It has no concept of &amp;#8220;this source address should use a different exit.&amp;#8221; That&amp;#8217;s where dual-&lt;span class="caps"&gt;FIB&lt;/span&gt; comes&amp;nbsp;in.&lt;/p&gt;
&lt;h2 id="architecture-overview"&gt;Architecture&amp;nbsp;Overview&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                        ┌─────────────────────────────────────────────┐
                        │           Public Internet                   │
                        └────────┬──────────────────────┬─────────────┘
                                 │                      │
                         Netcup Network           BGP (AS201379)
                         152.53.144.0/22          via hobgp router
                                 │                      │
                                 │               ┌──────┴──────────┐
                                 │               │  hobgp (Core)   │
                                 │               │  128.140.64.181 │
                                 │               └──────┬──────────┘
                                 │                      │
                                 │              GIF tunnel (proto 41)
                                 │              IPv6-in-IPv4 encap
     ┌───────────────────────────┴──────────────────────┴─────────────────────┐
     │                        radon.edelga.se                                 │
     │                                                                        │
     │  vtnet0 ──── FIB 0 (Default)                                           │
     │  152.53.147.5/22        &amp;lt;- Standard internet, NAT, mgmt                │
     │  2a0a:4cc0:c1:2f90::2/64                                               │
     │                                                                        │
     │  gif0 ────── FIB 1 (Tunnel)                                            │
     │  194.28.98.216/29       &amp;lt;- BGP-addressed traffic                       │
     │  2a06:9801:1c:ffff::2/128                                              │
     │                                ┌─────────────────────────────────────┐ │
     │  bastille0 ════════════════════│        Jail Bridge                  │ │
     │  10.254.254.1/24               │                                     │ │
     │  2a06:9801:1c:1000::1/64       │  ┌────────┐  ┌─────────────────────┐│ │
     │                                │  │ Caddy  │  │ testvnet            ││ │
     │  lo0:                          │  │ .254.10│  │194.28.98.216        ││ │
     │  194.28.98.217/32 (shared)     │  │ (NAT)  │  │2a06:9801:1c:1000::99││ │
     │                                │  └────────┘  │   (Pure Public)     ││ │
     │                                │              └─────────────────────┘│ │
     │                                └─────────────────────────────────────┘ │
     └────────────────────────────────────────────────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The key insight: &lt;span class="caps"&gt;FIB&lt;/span&gt; 0 handles &amp;#8220;normal&amp;#8221; traffic (Netcup-addressed). &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 handles &lt;span class="caps"&gt;BGP&lt;/span&gt;-addressed traffic (&lt;span class="caps"&gt;AS201379&lt;/span&gt;). &lt;span class="caps"&gt;PF&lt;/span&gt; is the glue that assigns packets to the correct &lt;span class="caps"&gt;FIB&lt;/span&gt; based on source or destination&amp;nbsp;address.&lt;/p&gt;
&lt;h2 id="the-physical-upstream-netcup-fib-0"&gt;The Physical Upstream: Netcup (&lt;span class="caps"&gt;FIB&lt;/span&gt;&amp;nbsp;0)&lt;/h2&gt;
&lt;p&gt;The physical interface lives in the default routing table. This is the server&amp;#8217;s &amp;#8220;normal&amp;#8221; internet&amp;nbsp;connection:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Primary physical interface - Netcup VPS&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 152.53.147.5 netmask 255.255.252.0 -lro -tso&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;152.53.144.1&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;ifconfig_vtnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2a0a:4cc0:c1:2f90::2 prefixlen 64&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fe80::1%vtnet0&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;The &lt;code&gt;-lro -tso&lt;/code&gt; flags deserve a callout.&lt;/strong&gt; &lt;span class="caps"&gt;LRO&lt;/span&gt; (Large Receive Offload) and &lt;span class="caps"&gt;TSO&lt;/span&gt; (&lt;span class="caps"&gt;TCP&lt;/span&gt; Segmentation Offload) are hardware acceleration features on the virtio &lt;span class="caps"&gt;NIC&lt;/span&gt;. They work fine for traffic that terminates at the host. But when the host forwards packets - as a router does for jail traffic - &lt;strong&gt;these offloads break &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s ability to correctly &lt;span class="caps"&gt;NAT&lt;/span&gt; and checksum forwarded packets.&lt;/strong&gt; The symptom is maddening: connections from the host work, but NATed jail traffic silently fails with bad checksums. &lt;strong&gt;Disable both whenever a FreeBSD host acts as a&amp;nbsp;gateway.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;SLAAC&lt;/span&gt; is disabled globally to avoid conflicts with statically routed tunnel&amp;nbsp;addresses:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;ipv6_activate_all_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="the-logical-upstream-bgp-tunnel-fib-1"&gt;The Logical Upstream: &lt;span class="caps"&gt;BGP&lt;/span&gt; Tunnel (&lt;span class="caps"&gt;FIB&lt;/span&gt;&amp;nbsp;1)&lt;/h2&gt;
&lt;p&gt;The &lt;span class="caps"&gt;GIF&lt;/span&gt; tunnel connects to the &lt;span class="caps"&gt;BGP&lt;/span&gt; router&amp;nbsp;at &lt;code&gt;128.140.64.181&lt;/code&gt;. This is where the multi-&lt;span class="caps"&gt;FIB&lt;/span&gt; magic&amp;nbsp;lives:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bridge0 gif0&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;ifconfig_gif0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fib 1 tunnel 152.53.147.5 128.140.64.181 tunnelfib 0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_gif0_alias0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.255.255.6 10.255.255.5&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_gif0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2a06:9801:1c:ffff::2 2a06:9801:1c:ffff::1 prefixlen 128&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This single line&amp;nbsp;- &lt;code&gt;fib 1 tunnel 152.53.147.5 128.140.64.181 tunnelfib 0&lt;/code&gt; - is the most important configuration directive on the entire server. It contains two directives that work in&amp;nbsp;concert:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;fib 1&lt;/code&gt;&lt;/strong&gt;: The tunnel interface itself lives in routing table 1. Traffic arriving&amp;nbsp;on &lt;code&gt;gif0&lt;/code&gt; and traffic routed&amp;nbsp;out &lt;code&gt;gif0&lt;/code&gt; consults &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 for routing&amp;nbsp;decisions.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;tunnelfib 0&lt;/code&gt;&lt;/strong&gt;: The outer IPv4 encapsulation wrapper&amp;nbsp;(the &lt;code&gt;152.53.147.5 → 128.140.64.181&lt;/code&gt; packet) uses &lt;span class="caps"&gt;FIB&lt;/span&gt; 0 - the default routing table - to find the path to the &lt;span class="caps"&gt;BGP&lt;/span&gt;&amp;nbsp;router.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Why both?&amp;nbsp;Without &lt;code&gt;tunnelfib 0&lt;/code&gt;, the encapsulated IPv4 packets would consult &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 for their route&amp;nbsp;to &lt;code&gt;128.140.64.181&lt;/code&gt;. But &lt;span class="caps"&gt;FIB&lt;/span&gt; 1&amp;#8217;s default route points&amp;nbsp;at &lt;code&gt;gif0&lt;/code&gt; itself (the tunnel). That creates a recursive loop: to send a packet out the tunnel, you&amp;#8217;d need to use the&amp;nbsp;tunnel. &lt;code&gt;tunnelfib 0&lt;/code&gt; breaks the recursion by telling the outer encapsulation to use the normal Netcup&amp;nbsp;route.&lt;/p&gt;
&lt;p&gt;The IPv4 point-to-point addresses&amp;nbsp;(&lt;code&gt;10.255.255.6/5&lt;/code&gt;) are used for the tunnel&amp;#8217;s inner addressing. The IPv6 link addresses&amp;nbsp;(&lt;code&gt;2a06:9801:1c:ffff::2/1&lt;/code&gt;) come from our own /48 - they&amp;#8217;re already routable within our &lt;span class="caps"&gt;BGP&lt;/span&gt;&amp;nbsp;infrastructure.&lt;/p&gt;
&lt;h2 id="fib-1-building-a-complete-routing-table"&gt;&lt;span class="caps"&gt;FIB&lt;/span&gt; 1: Building a Complete Routing&amp;nbsp;Table&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;FIB&lt;/span&gt; 1 starts empty. It knows nothing about the server&amp;#8217;s local networks. Every route must be explicitly added - if &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 doesn&amp;#8217;t have a route to a destination, traffic in that routing table gets&amp;nbsp;dropped.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;static_routes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fib1_v6_default fib1_v6_jailnet fib1_v4_default fib1_v4_jailnet fib1_v4_host217 fib1_v4_jail216 v4_jail216&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# 1. Default Routes (How FIB 1 reaches the internet)&lt;/span&gt;
&lt;span class="nv"&gt;route_fib1_v6_default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-6 default -interface gif0 -fib 1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;route_fib1_v4_default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-inet default 10.255.255.5 -fib 1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# 2. Subnet Routes (How FIB 1 finds the bridge)&lt;/span&gt;
&lt;span class="nv"&gt;route_fib1_v6_jailnet&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-6 2a06:9801:1c:1000::/64 -interface bastille0 -fib 1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;route_fib1_v4_jailnet&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-net 10.254.254.0/24 -interface bastille0 -fib 1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# 3. Host Routes (Where FIB 1 finds specific public IPs)&lt;/span&gt;
&lt;span class="nv"&gt;route_fib1_v4_host217&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-host 194.28.98.217 -interface lo0 -fib 1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;route_fib1_v4_jail216&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-host 194.28.98.216 -interface bastille0 -fib 1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;route_v4_jail216&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-host 194.28.98.216 -interface bastille0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="c1"&gt;# FIB 0 also needs this — explained in the Pure Public section below&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Each route category serves a specific&amp;nbsp;purpose:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Default routes&lt;/strong&gt; tell &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 how to reach the internet. IPv6 exits&amp;nbsp;through &lt;code&gt;gif0&lt;/code&gt; directly (the tunnel interface). IPv4&amp;nbsp;uses &lt;code&gt;10.255.255.5&lt;/code&gt; as the next-hop - that&amp;#8217;s the &lt;span class="caps"&gt;BGP&lt;/span&gt; router&amp;#8217;s end of the tunnel&amp;#8217;s inner IPv4&amp;nbsp;addressing.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Subnet routes&lt;/strong&gt; are necessary because &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 doesn&amp;#8217;t automatically know&amp;nbsp;that &lt;code&gt;bastille0&lt;/code&gt; has the jail network attached. Without these, return traffic arriving on the tunnel for jail addresses would try to exit via the default route (back out the tunnel) instead of being delivered locally to the&amp;nbsp;bridge.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Host routes&lt;/strong&gt; handle the public&amp;nbsp;IPs. &lt;code&gt;194.28.98.217&lt;/code&gt; is&amp;nbsp;on &lt;code&gt;lo0&lt;/code&gt; (the host&amp;#8217;s shared loopback &lt;span class="caps"&gt;IP&lt;/span&gt; for port&amp;nbsp;forwarding). &lt;code&gt;194.28.98.216&lt;/code&gt; is&amp;nbsp;on &lt;code&gt;bastille0&lt;/code&gt; (directly assigned to&amp;nbsp;the &lt;code&gt;testvnet&lt;/code&gt; jail). Both FIBs need to know where these addresses live - note&amp;nbsp;that &lt;code&gt;v4_jail216&lt;/code&gt; adds the same route to &lt;span class="caps"&gt;FIB&lt;/span&gt; 0 as well, so that the default routing table can also reach the&amp;nbsp;jail.&lt;/p&gt;
&lt;h2 id="jail-networking-and-the-bridge"&gt;Jail Networking and the&amp;nbsp;Bridge&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;bastille0&lt;/code&gt; bridge is the single point where all jail routing paradigms&amp;nbsp;converge:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;ifconfig_bridge0_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.254.254.1/24&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2a06:9801:1c:1000::1 prefixlen 64&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Three types of traffic share this&amp;nbsp;bridge:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Private &lt;span class="caps"&gt;NAT&lt;/span&gt; (10.254.254.0/24)&lt;/strong&gt;: Jails like Caddy&amp;nbsp;(&lt;code&gt;10.254.254.10&lt;/code&gt;) get private IPv4 addresses, NATed to the Netcup &lt;span class="caps"&gt;IP&lt;/span&gt; when&amp;nbsp;exiting &lt;code&gt;vtnet0&lt;/code&gt;. This is standard jail networking - nothing special&amp;nbsp;here.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;BGP&lt;/span&gt; IPv6 Routed (2a06:9801:1c:1000::/64)&lt;/strong&gt;: Jails also receive addresses from our &lt;span class="caps"&gt;BGP&lt;/span&gt; prefix. These are globally routable, announced by &lt;span class="caps"&gt;AS201379&lt;/span&gt;, and traffic for them arrives via the &lt;span class="caps"&gt;GIF&lt;/span&gt; tunnel. No &lt;span class="caps"&gt;NAT&lt;/span&gt; - pure end-to-end IPv6 as&amp;nbsp;designed.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;BGP&lt;/span&gt; IPv4 Routed (194.28.98.x)&lt;/strong&gt;:&amp;nbsp;The &lt;code&gt;testvnet&lt;/code&gt; jail&amp;nbsp;has &lt;code&gt;194.28.98.216/32&lt;/code&gt; - a real public IPv4 address, no &lt;span class="caps"&gt;NAT&lt;/span&gt;, directly routed through both&amp;nbsp;FIBs.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The shared host &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;(&lt;code&gt;194.28.98.217&lt;/code&gt;)&amp;nbsp;on &lt;code&gt;lo0&lt;/code&gt; is a different pattern. It&amp;#8217;s not assigned to any jail; instead, &lt;span class="caps"&gt;PF&lt;/span&gt; uses &lt;span class="caps"&gt;RDR&lt;/span&gt; rules to forward ports from this &lt;span class="caps"&gt;IP&lt;/span&gt; to internal jails. This lets Caddy serve content from a &lt;span class="caps"&gt;BGP&lt;/span&gt;-routed IPv4 address while the jail itself only has a&amp;nbsp;private &lt;code&gt;10.254.254.10&lt;/code&gt; address.&lt;/p&gt;
&lt;h2 id="pf-where-the-routing-decision-happens"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt;: Where the Routing Decision&amp;nbsp;Happens&lt;/h2&gt;
&lt;p&gt;The &lt;span class="caps"&gt;PF&lt;/span&gt; configuration is where dual-&lt;span class="caps"&gt;FIB&lt;/span&gt; stops being a kernel feature and becomes a routing policy. Let&amp;#8217;s walk through each&amp;nbsp;layer.&lt;/p&gt;
&lt;h3 id="normalization"&gt;Normalization&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;scrub in all fragment reassemble
scrub all max-mss 1220
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;MSS&lt;/span&gt; clamp at 1220 accounts for tunnel encapsulation overhead. &lt;span class="caps"&gt;MSS&lt;/span&gt; (Maximum Segment Size) dictates the maximum size of the &lt;span class="caps"&gt;TCP&lt;/span&gt; payload. To calculate it, subtract the inner &lt;span class="caps"&gt;IP&lt;/span&gt; header and the inner &lt;span class="caps"&gt;TCP&lt;/span&gt; header from the &lt;span class="caps"&gt;MTU&lt;/span&gt;. For IPv6 traffic inside the tunnel, clamped to the safe minimum &lt;span class="caps"&gt;MTU&lt;/span&gt; of 1280 bytes, you subtract the IPv6 header (40 bytes) and the &lt;span class="caps"&gt;TCP&lt;/span&gt; header (20 bytes): 1280 - 40 - 20 = 1220. Without this, large &lt;span class="caps"&gt;TCP&lt;/span&gt; segments get silently dropped or fragmented at the tunnel, causing mysterious stalls where small requests succeed but large transfers&amp;nbsp;hang.&lt;/p&gt;
&lt;h3 id="translation-nat-and-rdr"&gt;Translation (&lt;span class="caps"&gt;NAT&lt;/span&gt; and &lt;span class="caps"&gt;RDR&lt;/span&gt;)&lt;/h3&gt;
&lt;p&gt;Translation happens before filtering. This is critical to understand - filter rules for &lt;span class="caps"&gt;RDR&lt;/span&gt; traffic must match the &lt;strong&gt;translated&lt;/strong&gt; internal &lt;span class="caps"&gt;IP&lt;/span&gt;, not the public &lt;span class="caps"&gt;IP&lt;/span&gt;.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;# Outbound NAT: private jails → Netcup IP&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="cp"&gt;# Port forwarding on the Netcup IP (vtnet0)&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v4&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1965&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;gemini_v4&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;53&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;powerdns_v4&lt;/span&gt;

&lt;span class="cp"&gt;# Port forwarding on the BGP host IP (arrives via tunnel)&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;host_v4_routed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v4&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The last rule is different - traffic&amp;nbsp;to &lt;code&gt;194.28.98.217:80/443&lt;/code&gt; arrives on the tunnel interface and gets redirected to the same Caddy jail. This means Caddy serves web traffic regardless of whether it arrived via Netcup or the &lt;span class="caps"&gt;BGP&lt;/span&gt; tunnel, even though Caddy only has a private &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;address.&lt;/p&gt;
&lt;h3 id="default-policies"&gt;Default&amp;nbsp;Policies&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;block quick from &amp;lt;bruteforce&amp;gt;
block drop in log all
block drop out log all
antispoof quick for { $ext_if, bastille0 }
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Default deny in both directions. Bruteforce entries get an instant drop before any evaluation. Antispoof prevents packets from entering interfaces they don&amp;#8217;t belong&amp;nbsp;to.&lt;/p&gt;
&lt;h3 id="egress-the-route-to-safety-net"&gt;Egress:&amp;nbsp;The &lt;code&gt;route-to&lt;/code&gt; Safety&amp;nbsp;Net&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pass out quick on $ext_if route-to ($tun_if $tun_gw_v4) inet from $host_v4_routed to any keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is a safety net rule that catches an edge case. When local processes on the host respond from the shared &lt;span class="caps"&gt;BGP&lt;/span&gt; &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;(&lt;code&gt;194.28.98.217&lt;/code&gt;), those replies naturally try to exit&amp;nbsp;via &lt;code&gt;vtnet0&lt;/code&gt; - the host&amp;#8217;s default route in &lt;span class="caps"&gt;FIB&lt;/span&gt; 0. But Netcup would drop traffic sourced from an &lt;span class="caps"&gt;IP&lt;/span&gt; that doesn&amp;#8217;t belong to their&amp;nbsp;network.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;route-to&lt;/code&gt; directive intercepts: &amp;#8220;if you see a packet about to&amp;nbsp;exit &lt;code&gt;vtnet0&lt;/code&gt; with&amp;nbsp;source &lt;code&gt;194.28.98.217&lt;/code&gt;, redirect it to the tunnel interface instead.&amp;#8221; It overrides the routing decision at the &lt;span class="caps"&gt;PF&lt;/span&gt; level, forcing the packet&amp;nbsp;into &lt;code&gt;gif0&lt;/code&gt; toward &lt;code&gt;10.255.255.5&lt;/code&gt; (the &lt;span class="caps"&gt;BGP&lt;/span&gt;&amp;nbsp;router).&lt;/p&gt;
&lt;h3 id="egress-jail-policy-routing-via-rtable"&gt;Egress: Jail Policy Routing&amp;nbsp;via &lt;code&gt;rtable&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;This is the core mechanism. When a jail sends traffic from a &lt;span class="caps"&gt;BGP&lt;/span&gt; address, &lt;span class="caps"&gt;PF&lt;/span&gt; must force it into &lt;span class="caps"&gt;FIB&lt;/span&gt;&amp;nbsp;1:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;BGP&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;addressed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;jail&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;traffic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;routing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;bgp_net_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rtable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pure&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;public&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;jail&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;traffic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;routing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;testvnet_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rtable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;rtable 1&lt;/code&gt; directive is the key. When a packet&amp;nbsp;from &lt;code&gt;2a06:9801:1c:1000::/64&lt;/code&gt; or &lt;code&gt;194.28.98.216&lt;/code&gt; arrives at the bridge, &lt;span class="caps"&gt;PF&lt;/span&gt; assigns it to routing table 1. The kernel then consults &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 for the forwarding decision - and &lt;span class="caps"&gt;FIB&lt;/span&gt; 1&amp;#8217;s default route points&amp;nbsp;out &lt;code&gt;gif0&lt;/code&gt; to the &lt;span class="caps"&gt;BGP&lt;/span&gt; router. The packet gets encapsulated and sent through the&amp;nbsp;tunnel.&lt;/p&gt;
&lt;p&gt;Without &lt;code&gt;rtable 1&lt;/code&gt;, these packets would use &lt;span class="caps"&gt;FIB&lt;/span&gt; 0&amp;#8217;s default route and exit through Netcup - where they&amp;#8217;d be dropped as&amp;nbsp;spoofed.&lt;/p&gt;
&lt;p&gt;There&amp;#8217;s also a &lt;span class="caps"&gt;DNS64&lt;/span&gt; translation rule for jails needing IPv4 destinations via&amp;nbsp;IPv6:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pass in quick on bastille0 inet6 from $bgp_net_v6 to 64:ff9b::/96 af-to inet from ($ext_if) keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And a standard &lt;span class="caps"&gt;NAT&lt;/span&gt; egress rule for private&amp;nbsp;jails:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pass in quick on bastille0 from &amp;lt;jails_v4&amp;gt; to ! $jail_net keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;! $jail_net&lt;/code&gt; negation allows jails to reach the internet but prevents them from directly accessing other jails - a micro-segmentation pattern that documents and enforces the network&amp;nbsp;architecture.&lt;/p&gt;
&lt;h3 id="ingress-tunnel-encapsulation"&gt;Ingress: Tunnel&amp;nbsp;Encapsulation&lt;/h3&gt;
&lt;p&gt;The &lt;span class="caps"&gt;GIF&lt;/span&gt; tunnel uses protocol 41 (IPv6-in-IPv4). &lt;span class="caps"&gt;PF&lt;/span&gt; must allow these encapsulated packets to enter and exit the physical&amp;nbsp;interface:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pass in  quick on $ext_if proto { 41, ipencap } from 128.140.64.181 to ($ext_if)
pass out quick on $ext_if proto { 41, ipencap } from ($ext_if) to 128.140.64.181
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Strictly locked to the &lt;span class="caps"&gt;BGP&lt;/span&gt; router&amp;#8217;s &lt;span class="caps"&gt;IP&lt;/span&gt; - no one else should be sending protocol 41 to this&amp;nbsp;server.&lt;/p&gt;
&lt;h3 id="ingress-the-reply-to-imperative"&gt;Ingress:&amp;nbsp;The &lt;code&gt;reply-to&lt;/code&gt; Imperative&lt;/h3&gt;
&lt;p&gt;All inbound connections from the &lt;span class="caps"&gt;BGP&lt;/span&gt; tunnel must be tagged&amp;nbsp;with &lt;code&gt;reply-to&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tunnel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ingress&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_gw_v4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;host_v4_routed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_gw_v4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;trusted_v4&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;host_v4_routed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;30822&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_gw_v4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;udp&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;frontend_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_gw_v4&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;testvnet_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tunnel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ingress&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;bgp_hub_ip&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;bgp_net_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;echorep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;toobig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;timex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;paramprob&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;bgp_hub_ip&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;udp&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;frontend_v6&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;bgp_hub_ip&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;udp&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;powerdns_v6&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;53&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;tun_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;bgp_hub_ip&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;gemini_v6&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1965&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;reply-to&lt;/code&gt; instructs &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s state machine: &amp;#8220;when you see a reply to this connection, send it back&amp;nbsp;out &lt;code&gt;$tun_if&lt;/code&gt; to the specified next-hop, regardless of what the routing table says.&amp;#8221; Without it, the kernel would consult &lt;span class="caps"&gt;FIB&lt;/span&gt; 0 for return traffic (the jail doesn&amp;#8217;t live in &lt;span class="caps"&gt;FIB&lt;/span&gt; 1), and replies would exit&amp;nbsp;through &lt;code&gt;vtnet0&lt;/code&gt; - getting dropped by Netcup as&amp;nbsp;spoofed.&lt;/p&gt;
&lt;p&gt;The IPv4 and IPv6 reply-to targets differ because they use different tunnel endpoint addresses:
- &lt;strong&gt;IPv4&lt;/strong&gt;: &lt;code&gt;$tun_gw_v4&lt;/code&gt; = &lt;code&gt;10.255.255.5&lt;/code&gt; (the inner IPv4 point-to-point address)
- &lt;strong&gt;IPv6&lt;/strong&gt;: &lt;code&gt;$bgp_hub_ip&lt;/code&gt; = &lt;code&gt;2a06:9801:1c:ffff::1&lt;/code&gt; (the inner IPv6 link&amp;nbsp;address)&lt;/p&gt;
&lt;h2 id="the-pure-public-vnet-jail"&gt;The Pure Public &lt;span class="caps"&gt;VNET&lt;/span&gt;&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;testvnet&lt;/code&gt; jail is the most interesting routing case. It has a real public IPv4 address&amp;nbsp;(&lt;code&gt;194.28.98.216/32&lt;/code&gt;) and a &lt;span class="caps"&gt;BGP&lt;/span&gt; IPv6 address&amp;nbsp;(&lt;code&gt;2a06:9801:1c:1000::99&lt;/code&gt;), both assigned directly to the&amp;nbsp;jail&amp;#8217;s &lt;code&gt;vnet0&lt;/code&gt; interface:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Inside the jail&lt;/span&gt;
vnet0:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1008843&lt;/span&gt;&amp;lt;UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST,LOWER_UP&amp;gt;
&lt;span class="w"&gt;        &lt;/span&gt;inet&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;194&lt;/span&gt;.28.98.216&lt;span class="w"&gt; &lt;/span&gt;netmask&lt;span class="w"&gt; &lt;/span&gt;0xffffffff&lt;span class="w"&gt; &lt;/span&gt;broadcast&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;194&lt;/span&gt;.28.98.216
&lt;span class="w"&gt;        &lt;/span&gt;inet6&lt;span class="w"&gt; &lt;/span&gt;2a06:9801:1c:1000::99&lt;span class="w"&gt; &lt;/span&gt;prefixlen&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;64&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This jail has no &lt;span class="caps"&gt;NAT&lt;/span&gt;. Outbound traffic&amp;nbsp;from &lt;code&gt;194.28.98.216&lt;/code&gt; hits the bridge, &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;nbsp;matches &lt;code&gt;from $testvnet_v4 → rtable 1&lt;/code&gt;, and the packet exits via the tunnel. Inbound traffic arrives on the tunnel, &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;nbsp;matches &lt;code&gt;to $testvnet_v4&lt;/code&gt; with &lt;code&gt;reply-to&lt;/code&gt;, and the packet is delivered to the jail on the bridge. Return traffic follows&amp;nbsp;the &lt;code&gt;reply-to&lt;/code&gt; state back through the&amp;nbsp;tunnel.&lt;/p&gt;
&lt;p&gt;Both FIBs need host routes for this address. &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 needs it because tunnel traffic must find the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;route_fib1_v4_jail216&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-host 194.28.98.216 -interface bastille0 -fib 1&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;FIB&lt;/span&gt; 0 needs it because proto 41 decapsulation happens before &lt;span class="caps"&gt;PF&lt;/span&gt; routing decisions - the kernel must know where to deliver the inner&amp;nbsp;packet:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;route_v4_jail216&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-host 194.28.98.216 -interface bastille0&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="the-complete-flow"&gt;The Complete&amp;nbsp;Flow&lt;/h2&gt;
&lt;h3 id="inbound-to-a-bgp-addressed-service-ipv6"&gt;Inbound to a &lt;span class="caps"&gt;BGP&lt;/span&gt;-addressed service&amp;nbsp;(IPv6)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt; 1. Client sends packet to 2a06:9801:1c:1000::10 (Caddy jail)
 2. Packet traverses the internet → AS201379 → hobgp BGP router
 3. hobgp forwards through GIF tunnel to radon (proto 41 encapsulated)
 4. radon receives proto 41 on vtnet0, decapsulates → gif0
 5. PF match: reply-to ($tun_if $bgp_hub_ip), creates state entry
 6. Forwarded to bastille0 → Caddy jail
 7. Caddy responds, packet exits on bastille0
 8. PF state table triggers reply-to: send via gif0 to bgp_hub_ip
 9. gif0 encapsulates (proto 41) using tunnelfib 0 → vtnet0 → hobgp
10. hobgp receives, forwards to upstream → internet → client
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="outbound-from-a-pure-public-jail-ipv4"&gt;Outbound from a pure public jail&amp;nbsp;(IPv4)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;1. testvnet jail sends packet from 194.28.98.216
2. Packet arrives on bastille0
3. PF match: &amp;quot;from $testvnet_v4 → rtable 1&amp;quot;
4. Kernel routes via FIB 1 → default route → gif0
5. gif0 encapsulates using tunnelfib 0 → vtnet0 → hobgp
6. hobgp receives, forwards to internet (source: 194.28.98.216)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="natd-jail-egress-ipv4-standard-path"&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt;&amp;#8217;d jail egress (IPv4, standard&amp;nbsp;path)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;1. Caddy jail sends packet from 10.254.254.10
2. Packet arrives on bastille0
3. PF match: &amp;quot;from &amp;lt;jails_v4&amp;gt; to ! $jail_net&amp;quot; (standard egress)
4. Kernel routes via FIB 0 → default route → vtnet0
5. NAT translates source to 152.53.147.5
6. Packet exits to Netcup → internet
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="verification"&gt;Verification&lt;/h2&gt;
&lt;h3 id="from-the-host-testing-both-fibs"&gt;From the host: testing both&amp;nbsp;FIBs&lt;/h3&gt;
&lt;p&gt;The &lt;code&gt;setfib&lt;/code&gt; command selects which routing table a command&amp;nbsp;uses:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Via Netcup (FIB 0) - physical uplink&lt;/span&gt;
root@radon:~&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# setfib 0 fetch -4 -o - -q https://ifconfig.co&lt;/span&gt;
&lt;span class="m"&gt;152&lt;/span&gt;.53.147.5

root@radon:~&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# setfib 0 fetch -6 -o - -q https://ifconfig.co&lt;/span&gt;
2a0a:4cc0:c1:2f90::2

&lt;span class="c1"&gt;# Via BGP tunnel (FIB 1) - bound to the routed IP&lt;/span&gt;
root@radon:~&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# setfib 1 fetch --bind-address=194.28.98.217 -4 -o - -q https://ifconfig.co&lt;/span&gt;
&lt;span class="m"&gt;194&lt;/span&gt;.28.98.217

root@radon:~&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# setfib 1 fetch -6 -o - -q https://ifconfig.co&lt;/span&gt;
2a06:9801:1c:ffff::2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;FIB&lt;/span&gt; 0 returns Netcup&amp;#8217;s addresses. &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 returns &lt;span class="caps"&gt;BGP&lt;/span&gt; addresses. Two completely independent paths to the internet from the same&amp;nbsp;machine.&lt;/p&gt;
&lt;h3 id="from-inside-the-pure-public-jail"&gt;From inside the pure public&amp;nbsp;jail&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@radon:~&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# bastille cmd testvnet fetch -4 -o - -q https://ifconfig.co&lt;/span&gt;
&lt;span class="m"&gt;194&lt;/span&gt;.28.98.216

root@radon:~&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# bastille cmd testvnet fetch -6 -o - -q https://ifconfig.co&lt;/span&gt;
2a06:9801:1c:1000::99
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The jail sees its own real public &lt;span class="caps"&gt;IP&lt;/span&gt; - not a NATed address. Both IPv4 and IPv6 traffic exits through the &lt;span class="caps"&gt;BGP&lt;/span&gt; tunnel and appears on the internet with the correct source&amp;nbsp;addresses.&lt;/p&gt;
&lt;h3 id="from-external-clients-both-paths-work"&gt;From external clients: both paths&amp;nbsp;work&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Via Netcup (default DNS, resolves to 152.53.147.5)&lt;/span&gt;
~&lt;span class="w"&gt; &lt;/span&gt;❯&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-Iv4&lt;span class="w"&gt; &lt;/span&gt;http://radon.edelga.se
*&lt;span class="w"&gt; &lt;/span&gt;IPv4:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;152&lt;/span&gt;.53.147.5
*&lt;span class="w"&gt; &lt;/span&gt;Connected&lt;span class="w"&gt; &lt;/span&gt;to&lt;span class="w"&gt; &lt;/span&gt;radon.edelga.se&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="m"&gt;152&lt;/span&gt;.53.147.5&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;80&lt;/span&gt;
&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;HTTP/1.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;308&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;Permanent&lt;span class="w"&gt; &lt;/span&gt;Redirect
&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;Server:&lt;span class="w"&gt; &lt;/span&gt;Caddy

&lt;span class="c1"&gt;# Via BGP tunnel (resolves to 2a06:9801:1c:1000::10)&lt;/span&gt;
~&lt;span class="w"&gt; &lt;/span&gt;❯&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-Iv6&lt;span class="w"&gt; &lt;/span&gt;http://blog.hofstede.it
*&lt;span class="w"&gt; &lt;/span&gt;IPv6:&lt;span class="w"&gt; &lt;/span&gt;2a06:9801:1c:1000::10
*&lt;span class="w"&gt; &lt;/span&gt;Connected&lt;span class="w"&gt; &lt;/span&gt;to&lt;span class="w"&gt; &lt;/span&gt;blog.hofstede.it&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;2a06:9801:1c:1000::10&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;80&lt;/span&gt;
&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;HTTP/1.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;308&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;Permanent&lt;span class="w"&gt; &lt;/span&gt;Redirect
&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;Server:&lt;span class="w"&gt; &lt;/span&gt;Caddy

&lt;span class="c1"&gt;# Via BGP tunnel (IPv4, routed through 194.28.98.217)&lt;/span&gt;
~&lt;span class="w"&gt; &lt;/span&gt;❯&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-Iv4&lt;span class="w"&gt; &lt;/span&gt;http://194.28.98.217
*&lt;span class="w"&gt; &lt;/span&gt;Connected&lt;span class="w"&gt; &lt;/span&gt;to&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;194&lt;/span&gt;.28.98.217&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="m"&gt;194&lt;/span&gt;.28.98.217&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;80&lt;/span&gt;
&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;HTTP/1.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;308&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;Permanent&lt;span class="w"&gt; &lt;/span&gt;Redirect
&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;Server:&lt;span class="w"&gt; &lt;/span&gt;Caddy
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Three different entry points, all reaching the same Caddy jail - which only has&amp;nbsp;a &lt;code&gt;10.254.254.10&lt;/code&gt; private address. The same service is accessible through Netcup&amp;#8217;s network (physical) and through the &lt;span class="caps"&gt;BGP&lt;/span&gt; tunnel (logical), and Caddy doesn&amp;#8217;t know or care which path the request&amp;nbsp;took.&lt;/p&gt;
&lt;h2 id="security-considerations"&gt;Security&amp;nbsp;Considerations&lt;/h2&gt;
&lt;p&gt;Dual-&lt;span class="caps"&gt;FIB&lt;/span&gt; setups require extra security attention because more routing paths mean more attack&amp;nbsp;surface:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;kern_securelevel=2&lt;/code&gt;&lt;/strong&gt; prevents loading kernel modules and writing to raw disk devices post-boot. Even if root is compromised, the attacker can&amp;#8217;t modify the running kernel or bypass the&amp;nbsp;firewall.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; is restricted to known IPs on a non-standard port&lt;/strong&gt; with aggressive rate limiting.&amp;nbsp;The &lt;code&gt;overload &amp;lt;bruteforce&amp;gt; flush global&lt;/code&gt; combination instantly blocks attackers and kills their existing&amp;nbsp;connections:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;30822&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; via the tunnel&amp;nbsp;uses &lt;code&gt;reply-to&lt;/code&gt; to ensure the management session stays within the tunnel path - an attacker on Netcup&amp;#8217;s network can&amp;#8217;t even see the tunnel-side &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;port.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Antispoof&lt;/strong&gt; blocks packets with impossible source addresses for each interface, preventing attackers from injecting traffic that pretends to come from the internal&amp;nbsp;network.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Monitoring is bound to the internal bridge only&lt;/strong&gt;, keeping Prometheus exporters invisible from the&amp;nbsp;internet:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;node_exporter_listen_address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.254.254.1:9100&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;jail_exporter_listen_address&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.254.254.1:9452&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="lessons-learned"&gt;Lessons&amp;nbsp;Learned&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;tunnelfib&lt;/code&gt; is the unsung hero.&lt;/strong&gt;&amp;nbsp;Without &lt;code&gt;tunnelfib 0&lt;/code&gt; on the &lt;span class="caps"&gt;GIF&lt;/span&gt; interface, the tunnel would try to route its own encapsulated packets through itself. The error mode is a silent hang - no error messages, no crash, just packets disappearing into recursive routing. If your &lt;span class="caps"&gt;GIF&lt;/span&gt; tunnel comes up but passes no traffic,&amp;nbsp;check &lt;code&gt;tunnelfib&lt;/code&gt; first.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;FIB&lt;/span&gt; 1 needs manual completeness.&lt;/strong&gt; Unlike &lt;span class="caps"&gt;FIB&lt;/span&gt; 0, which automatically gets interface routes when interfaces come up, &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 starts empty. If you can ping the internet from &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 but jails can&amp;#8217;t reply, you&amp;#8217;re missing a subnet route that tells &lt;span class="caps"&gt;FIB&lt;/span&gt; 1 where the bridge is. The symptom is asymmetric: inbound works&amp;nbsp;(because &lt;code&gt;reply-to&lt;/code&gt; handles replies), but jail-initiated connections&amp;nbsp;fail.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;reply-to&lt;/code&gt; and &lt;code&gt;rtable&lt;/code&gt; solve different halves of the same&amp;nbsp;problem.&lt;/strong&gt; &lt;code&gt;reply-to&lt;/code&gt; handles inbound connections: &amp;#8220;replies to tunnel traffic must exit via the&amp;nbsp;tunnel.&amp;#8221; &lt;code&gt;rtable&lt;/code&gt; handles outbound connections: &amp;#8220;traffic from &lt;span class="caps"&gt;BGP&lt;/span&gt; addresses must route via &lt;span class="caps"&gt;FIB&lt;/span&gt; 1.&amp;#8221; You need both - one without the other produces subtle&amp;nbsp;failures.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Disable hardware offloading on forwarding hosts.&lt;/strong&gt; &lt;span class="caps"&gt;LRO&lt;/span&gt; and &lt;span class="caps"&gt;TSO&lt;/span&gt; on virtio NICs cause &lt;span class="caps"&gt;PF&lt;/span&gt; to compute incorrect checksums for forwarded packets. The failure is invisible at the host level - everything looks fine locally - but downstream receivers silently drop the corrupted packets. &lt;strong&gt;The &lt;code&gt;-lro -tso&lt;/code&gt; flags are non-negotiable&amp;nbsp;when &lt;code&gt;gateway_enable="YES"&lt;/code&gt;.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;MSS&lt;/span&gt; clamping is not optional with tunnels.&lt;/strong&gt; Every encapsulation layer reduces the effective &lt;span class="caps"&gt;MTU&lt;/span&gt;.&amp;nbsp;Without &lt;code&gt;scrub all max-mss 1220&lt;/code&gt;, &lt;span class="caps"&gt;TCP&lt;/span&gt; connections that transfer more than ~1220 bytes of payload per segment will stall. The insidious part: small requests (&lt;span class="caps"&gt;DNS&lt;/span&gt;, &lt;span class="caps"&gt;HTTP&lt;/span&gt; redirects) work fine, making the problem seem intermittent until someone tries to download a&amp;nbsp;file.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A single bridge can carry multiple routing paradigms.&lt;/strong&gt; Private NATed traffic, natively routed IPv6, and pure public IPv4 all coexist&amp;nbsp;on &lt;code&gt;bastille0&lt;/code&gt; without any &lt;span class="caps"&gt;VLAN&lt;/span&gt; tagging or interface separation. &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s source-based rules handle the routing differentiation. This simplicity is a feature - fewer moving parts means fewer failure&amp;nbsp;modes.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Dual-&lt;span class="caps"&gt;FIB&lt;/span&gt; policy routing on FreeBSD is the kind of feature that sounds complex until you see it in practice. Two routing tables, a handful of &lt;span class="caps"&gt;PF&lt;/span&gt; directives&amp;nbsp;(&lt;code&gt;rtable&lt;/code&gt;, &lt;code&gt;reply-to&lt;/code&gt;, &lt;code&gt;route-to&lt;/code&gt;), and some static routes. That&amp;#8217;s the entire mechanism that lets a single server speak from two completely different address spaces through two completely different internet&amp;nbsp;paths.&lt;/p&gt;
&lt;p&gt;The configuration shown here is what runs in production. The same Caddy jail&amp;nbsp;serves &lt;code&gt;blog.hofstede.it&lt;/code&gt; via IPv6 through the &lt;span class="caps"&gt;BGP&lt;/span&gt; tunnel&amp;nbsp;and &lt;code&gt;radon.edelga.se&lt;/code&gt; via IPv4 through Netcup - simultaneously, transparently, without knowing the difference.&amp;nbsp;The &lt;code&gt;testvnet&lt;/code&gt; jail has a real public IPv4 address that&amp;#8217;s globally routable through &lt;span class="caps"&gt;AS201379&lt;/span&gt;, with no &lt;span class="caps"&gt;NAT&lt;/span&gt; anywhere in the path. And the whole thing coexists peacefully with standard NATed jail traffic on the same&amp;nbsp;bridge.&lt;/p&gt;
&lt;p&gt;Is this setup necessary for running a blog? No. But it&amp;#8217;s the same pattern that production multi-homed infrastructure uses at much larger scale. The tools are the same - FIBs, &lt;span class="caps"&gt;PF&lt;/span&gt;, tunnels - just running on a single virtual machine instead of a rack of routers. FreeBSD makes this accessible because these features are first-class kernel primitives, not bolted-on&amp;nbsp;afterthoughts.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=setfib"&gt;FreeBSD setfib(1)&lt;/a&gt; - Multiple routing table&amp;nbsp;selection&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=gif"&gt;FreeBSD gif(4)&lt;/a&gt; - Generic tunnel interface,&amp;nbsp;including &lt;code&gt;tunnelfib&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pf.conf"&gt;FreeBSD pf.conf(5)&lt;/a&gt; - &lt;span class="caps"&gt;PF&lt;/span&gt; configuration&amp;nbsp;reference&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/firewalls/"&gt;FreeBSD Handbook: Firewalls (&lt;span class="caps"&gt;PF&lt;/span&gt;)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/running-your-own-as-bgp-on-freebsd-with-frr-gre-tunnels-and-policy-routing/"&gt;Running Your Own &lt;span class="caps"&gt;AS&lt;/span&gt;: &lt;span class="caps"&gt;BGP&lt;/span&gt; on FreeBSD&lt;/a&gt; - The &lt;span class="caps"&gt;BGP&lt;/span&gt; router side of this&amp;nbsp;setup&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/running-your-own-as-going-multi-homed-with-ibgp-and-three-transits/"&gt;Going Multi-Homed with iBGP&lt;/a&gt; - The multi-PoP&amp;nbsp;expansion&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; Firewall on FreeBSD: A Practical Guide&lt;/a&gt; - &lt;span class="caps"&gt;PF&lt;/span&gt; foundation&amp;nbsp;concepts&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;The internet doesn&amp;#8217;t care that all of this runs on one virtual machine. Packets arrive, get routed through the correct table, hit the correct jail, and the replies leave through the correct tunnel. Two FIBs, one bridge, zero confusion. That&amp;#8217;s the beauty of policy routing done right - the complexity is in the configuration, not in the&amp;nbsp;operation.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="networking"/><category term="pf"/><category term="bgp"/><category term="jails"/><category term="tunneling"/><category term="policy-routing"/></entry><entry><title>FreeBSD Foundationals: ZFS - The Last Filesystem You’ll Ever Need</title><link href="https://hofstede.it/freebsd-foundationals-zfs-the-last-filesystem-youll-ever-need/" rel="alternate"/><published>2026-03-13T00:00:00+01:00</published><updated>2026-03-13T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-03-13:/freebsd-foundationals-zfs-the-last-filesystem-youll-ever-need/</id><summary type="html">&lt;p&gt;The second in the FreeBSD Foundationals series. This one covers &lt;span class="caps"&gt;ZFS&lt;/span&gt; from philosophy to practice: why it exists, how pools and datasets work, what checksumming and self-healing actually do, how to tune recordsize, compression, and atime, how encryption works with key management, how snapshots and the hidden .zfs directory give you time travel, and how &lt;span class="caps"&gt;ZFS&lt;/span&gt; send/recv turns backup and migration into a solved problem. Includes a look at sanoid/syncoid for automated snapshot&amp;nbsp;management.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; was born at Sun Microsystems in 2004, open-sourced in 2005 as part of OpenSolaris, and has since become the default filesystem on FreeBSD. Not just the default in the installer - the default in production, the default in the Handbook, the default in the minds of people who have lost data exactly once and decided never again. (It&amp;#8217;s also available on Linux, where it works beautifully - just don&amp;#8217;t ask me how I know you can &lt;a href="https://hofstede.it/rhel-on-zfs-root-an-unholy-experiment/"&gt;run &lt;span class="caps"&gt;RHEL&lt;/span&gt; on a &lt;span class="caps"&gt;ZFS&lt;/span&gt; root pool&lt;/a&gt;. That was a crime, not a&amp;nbsp;tutorial.)&lt;/p&gt;
&lt;p&gt;This is the second article in the &lt;strong&gt;FreeBSD Foundationals&lt;/strong&gt; series. The &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;first one covered Jails&lt;/a&gt;. We&amp;#8217;re covering &lt;span class="caps"&gt;ZFS&lt;/span&gt; now because it&amp;#8217;s the foundation everything else sits on: your jails, your databases, your mail spools, your backups. Understanding &lt;span class="caps"&gt;ZFS&lt;/span&gt; isn&amp;#8217;t optional if you&amp;#8217;re running FreeBSD&amp;nbsp;seriously.&lt;/p&gt;
&lt;h2 id="why-zfs-exists"&gt;Why &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;nbsp;Exists&lt;/h2&gt;
&lt;p&gt;Traditional filesystems trust the hardware. They write data to disk, read it back later, and assume what comes back is what was written. This assumption is wrong more often than most people realize. Disks develop bad sectors. Controllers corrupt data in transit. &lt;span class="caps"&gt;RAM&lt;/span&gt; bit-flips go undetected. &lt;span class="caps"&gt;RAID&lt;/span&gt; controllers with battery-backed caches fail in ways that silently eat data. The industry term for this is &lt;strong&gt;silent data corruption&lt;/strong&gt;, and every filesystem that doesn&amp;#8217;t checksum its data is vulnerable to&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; was designed from the ground up with one overriding principle: &lt;strong&gt;your data should never be silently wrong&lt;/strong&gt;. Every block of data and metadata is checksummed. Every read is verified against that checksum. If the checksum doesn&amp;#8217;t match, &lt;span class="caps"&gt;ZFS&lt;/span&gt; knows the data is corrupt - and if redundancy exists (mirror or raidz), it automatically repairs the damage from a good copy without anyone having to notice or&amp;nbsp;intervene.&lt;/p&gt;
&lt;p&gt;But &lt;span class="caps"&gt;ZFS&lt;/span&gt; is not just a filesystem. It&amp;#8217;s a &lt;strong&gt;filesystem, volume manager, and software &lt;span class="caps"&gt;RAID&lt;/span&gt; implementation rolled into one&lt;/strong&gt;. Traditional Unix storage stacks look like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Traditional stack:              ZFS stack:

┌────────────────┐              ┌────────────────┐
│   Filesystem   │              │                │
│   (ext4/ufs)   │              │      ZFS       │
├────────────────┤              │                │
│ Volume Manager │              │  (filesystem + │
│   (LVM/gpart)  │              │  volume mgmt + │
├────────────────┤              │   RAID + ...   │
│  Software RAID │              │                │
│   (gmirror)    │              │                │
├────────────────┤              ├────────────────┤
│     Disks      │              │     Disks      │
└────────────────┘              └────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With a traditional stack, you need to coordinate between three or four separate layers, each with its own tools, its own failure modes, and its own ideas about how data is organized. &lt;span class="caps"&gt;ZFS&lt;/span&gt; eliminates this entire class of problems by managing everything from raw disks to individual files in a single, coherent&amp;nbsp;system.&lt;/p&gt;
&lt;h2 id="core-concepts-pools-and-datasets"&gt;Core Concepts: Pools and&amp;nbsp;Datasets&lt;/h2&gt;
&lt;h3 id="pools-the-storage-foundation"&gt;Pools: The Storage&amp;nbsp;Foundation&lt;/h3&gt;
&lt;p&gt;A &lt;span class="caps"&gt;ZFS&lt;/span&gt; &lt;strong&gt;pool&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;zpool&lt;/code&gt;) is the fundamental unit of storage. You give &lt;span class="caps"&gt;ZFS&lt;/span&gt; one or more disks (or partitions, or files - it doesn&amp;#8217;t care), and it creates a pool. All storage in &lt;span class="caps"&gt;ZFS&lt;/span&gt; comes from a pool. There are no partitions to resize, no logical volumes to extend, no separate &amp;#8220;allocate space, then format&amp;#8221;&amp;nbsp;steps.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a simple pool from a single disk&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank&lt;span class="w"&gt; &lt;/span&gt;da0

&lt;span class="c1"&gt;# Create a mirror (two-way redundancy)&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank&lt;span class="w"&gt; &lt;/span&gt;mirror&lt;span class="w"&gt; &lt;/span&gt;da0&lt;span class="w"&gt; &lt;/span&gt;da1

&lt;span class="c1"&gt;# Create a raidz1 pool (single-parity, like RAID-5)&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank&lt;span class="w"&gt; &lt;/span&gt;raidz1&lt;span class="w"&gt; &lt;/span&gt;da0&lt;span class="w"&gt; &lt;/span&gt;da1&lt;span class="w"&gt; &lt;/span&gt;da2

&lt;span class="c1"&gt;# Create a raidz2 pool (double-parity, like RAID-6)&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank&lt;span class="w"&gt; &lt;/span&gt;raidz2&lt;span class="w"&gt; &lt;/span&gt;da0&lt;span class="w"&gt; &lt;/span&gt;da1&lt;span class="w"&gt; &lt;/span&gt;da2&lt;span class="w"&gt; &lt;/span&gt;da3

&lt;span class="c1"&gt;# Check pool status&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;tank
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Pool topology matters and cannot be changed after creation. You choose your redundancy level when you create the pool. A mirror can tolerate one disk failure per mirror pair. Raidz1 tolerates one disk failure per vdev. Raidz2 tolerates two. Choose raidz2 for anything that matters - modern drives are massive, and resilvering (rebuilding) a &lt;span class="caps"&gt;16TB&lt;/span&gt; disk takes days of heavy I/O, which is exactly when a second aging drive is most likely to fail. The cost of one extra parity disk is nothing compared to a lost&amp;nbsp;pool.&lt;/p&gt;
&lt;p&gt;You can check the health of a pool at any&amp;nbsp;time:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;zpool&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;tank
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;  pool: tank
 state: ONLINE
  scan: scrub repaired 0B in 02:31:44 with 0 errors on Sun Mar  9 03:01:44 2026
config:

        NAME        STATE     READ WRITE CKSUM
        tank        ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0

errors: No known data errors
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;CKSUM&lt;/code&gt; column is the one that matters most. A non-zero value there means &lt;span class="caps"&gt;ZFS&lt;/span&gt; detected a checksum mismatch - silent corruption that most other filesystems would have delivered to your application without a&amp;nbsp;word.&lt;/p&gt;
&lt;h3 id="scrubs-proactive-integrity-verification"&gt;Scrubs: Proactive Integrity&amp;nbsp;Verification&lt;/h3&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; doesn&amp;#8217;t just verify data when you read it. You can (and should) run periodic &lt;strong&gt;scrubs&lt;/strong&gt; that read every block in the pool and verify its&amp;nbsp;checksum:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Start a scrub&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;scrub&lt;span class="w"&gt; &lt;/span&gt;tank

&lt;span class="c1"&gt;# Check scrub progress&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;tank
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On a redundant pool (mirror or raidz), a scrub will automatically repair any corruption it finds using the redundant copy. On a non-redundant pool, a scrub will detect the corruption and report it, but can&amp;#8217;t repair it - there&amp;#8217;s no good copy to repair&amp;nbsp;from.&lt;/p&gt;
&lt;p&gt;Schedule scrubs regularly. Weekly is common for servers. &lt;span class="caps"&gt;ZFS&lt;/span&gt; automatically pauses scrubs during heavy I/O to avoid degrading application performance, so don&amp;#8217;t be afraid to schedule them during the work week if your maintenance windows are limited. FreeBSD enables periodic &lt;span class="caps"&gt;ZFS&lt;/span&gt; scrubs by default via a cron entry&amp;nbsp;in &lt;code&gt;/etc/periodic.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# In /etc/periodic.conf (or check /etc/defaults/periodic.conf)&lt;/span&gt;
&lt;span class="nv"&gt;daily_scrub_zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="datasets-flexible-storage-without-partitions"&gt;Datasets: Flexible Storage Without&amp;nbsp;Partitions&lt;/h3&gt;
&lt;p&gt;A &lt;span class="caps"&gt;ZFS&lt;/span&gt; &lt;strong&gt;dataset&lt;/strong&gt; is roughly analogous to a traditional filesystem, but without fixed size. Datasets share the pool&amp;#8217;s storage dynamically. You don&amp;#8217;t pre-allocate space to a dataset - it grows as needed, up to whatever the pool (or an optional quota)&amp;nbsp;allows.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create datasets&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank/home
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank/home/chris
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank/jails
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank/postgresql
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Datasets are hierarchical. Properties set on a parent are inherited by children unless explicitly overridden. This inheritance model is one of &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;#8217;s most practical&amp;nbsp;features:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Set compression on the parent - all children inherit it&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;compression&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;zstd&lt;span class="w"&gt; &lt;/span&gt;tank

&lt;span class="c1"&gt;# Override compression for a specific child dataset&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;compression&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;lz4&lt;span class="w"&gt; &lt;/span&gt;tank/postgresql

&lt;span class="c1"&gt;# Check inherited vs. local properties&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;compression&lt;span class="w"&gt; &lt;/span&gt;tank/home
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;NAME        PROPERTY     VALUE           SOURCE
tank/home   compression  zstd            inherited from tank
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="key-dataset-properties"&gt;Key Dataset&amp;nbsp;Properties&lt;/h3&gt;
&lt;p&gt;These are the properties you&amp;#8217;ll actually care about in&amp;nbsp;practice:&lt;/p&gt;
&lt;h4 id="compression"&gt;compression&lt;/h4&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; supports transparent compression at the dataset level. Every block is compressed before being written to disk and decompressed on read. This sounds like it should be slower, but on modern CPUs the compression is so fast that the reduced I/O usually makes things &lt;em&gt;faster&lt;/em&gt;&amp;nbsp;overall.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Modern default - excellent compression ratio, very fast&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;compression&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;zstd&lt;span class="w"&gt; &lt;/span&gt;tank

&lt;span class="c1"&gt;# Alternative - slightly faster, slightly worse ratio&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;compression&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;lz4&lt;span class="w"&gt; &lt;/span&gt;tank/fast-storage

&lt;span class="c1"&gt;# Check compression effectiveness&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;compressratio&lt;span class="w"&gt; &lt;/span&gt;tank
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There&amp;#8217;s almost no reason to leave compression off. Even on data that doesn&amp;#8217;t compress well (encrypted files, already-compressed media), &lt;span class="caps"&gt;ZFS&lt;/span&gt; detects incompressible blocks and stores them uncompressed with negligible&amp;nbsp;overhead.&lt;/p&gt;
&lt;h4 id="recordsize"&gt;recordsize&lt;/h4&gt;
&lt;p&gt;The &lt;code&gt;recordsize&lt;/code&gt; property controls the maximum block size for a dataset. The default is &lt;span class="caps"&gt;128KB&lt;/span&gt;, which works well for general-purpose workloads. But specific workloads benefit from tuning&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# PostgreSQL: match the database page size (8KB)&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;recordsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;8k&lt;span class="w"&gt; &lt;/span&gt;tank/postgresql

&lt;span class="c1"&gt;# MySQL/InnoDB: match the InnoDB page size (16KB)&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;recordsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;16k&lt;span class="w"&gt; &lt;/span&gt;tank/mysql

&lt;span class="c1"&gt;# Large sequential files (media, backups, VM images): use 1MB&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;recordsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;1M&lt;span class="w"&gt; &lt;/span&gt;tank/backups
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The principle: match the recordsize to the I/O pattern of the application. Databases do many small, random reads and writes at their page size - a &lt;span class="caps"&gt;128KB&lt;/span&gt; recordsize means &lt;span class="caps"&gt;ZFS&lt;/span&gt; reads &lt;span class="caps"&gt;128KB&lt;/span&gt; to satisfy an &lt;span class="caps"&gt;8KB&lt;/span&gt; database page read.&amp;nbsp;Setting &lt;code&gt;recordsize=8k&lt;/code&gt; for PostgreSQL can dramatically improve random read&amp;nbsp;performance.&lt;/p&gt;
&lt;p&gt;For sequential workloads (backups, media storage, large file copies), a larger recordsize means fewer metadata operations and better compression&amp;nbsp;ratios.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Important:&lt;/strong&gt; recordsize only affects newly written data. Changing it on an existing dataset does not rewrite existing blocks. For databases, set the recordsize before importing&amp;nbsp;data.&lt;/p&gt;
&lt;h4 id="atime"&gt;atime&lt;/h4&gt;
&lt;p&gt;Every time a file is read, Unix traditionally updates its &amp;#8220;access time&amp;#8221;&amp;nbsp;(&lt;code&gt;atime&lt;/code&gt;) metadata. On a busy system, this means every read operation also triggers a write operation. &lt;span class="caps"&gt;ZFS&lt;/span&gt; inherits this behavior by default, but you should almost always disable&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Disable access time updates entirely&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;atime&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;off&lt;span class="w"&gt; &lt;/span&gt;tank

&lt;span class="c1"&gt;# Or use relative atime (only update when mtime/ctime is newer than atime)&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;relatime&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;on&lt;span class="w"&gt; &lt;/span&gt;tank
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Disabling &lt;code&gt;atime&lt;/code&gt; eliminates a write I/O for every read I/O. For a mail server scanning thousands of files per second, or a web server serving static assets, this is a significant performance improvement with essentially no downside. Almost nothing on a modern system depends on accurate&amp;nbsp;atime.&lt;/p&gt;
&lt;h4 id="quota-and-reservation"&gt;quota and&amp;nbsp;reservation&lt;/h4&gt;
&lt;p&gt;Quotas limit how much space a dataset can consume. Reservations guarantee a minimum amount of space. These are useful for preventing runaway datasets from starving&amp;nbsp;others:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Limit a dataset to 50GB&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;quota&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;50G&lt;span class="w"&gt; &lt;/span&gt;tank/home/chris

&lt;span class="c1"&gt;# Guarantee 10GB is always available for this dataset&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;reservation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;10G&lt;span class="w"&gt; &lt;/span&gt;tank/postgresql
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;refquota&lt;/code&gt; and &lt;code&gt;refreservation&lt;/code&gt; are the variants that exclude snapshots from the accounting.&amp;nbsp;Usually &lt;code&gt;refquota&lt;/code&gt; is what you want for user quotas - you don&amp;#8217;t want someone&amp;#8217;s disk usage to silently increase because of snapshots they didn&amp;#8217;t&amp;nbsp;take.&lt;/p&gt;
&lt;h2 id="self-healing-how-it-actually-works"&gt;Self-Healing: How It Actually&amp;nbsp;Works&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; checksums every block of data and metadata using &lt;span class="caps"&gt;SHA&lt;/span&gt;-256 (by default; other algorithms are available). The checksum is stored separately from the data it protects - in the parent block&amp;#8217;s metadata. This means corruption of a data block can&amp;#8217;t also corrupt its own checksum, which is the fatal flaw of systems that store checksums inline (like a &lt;span class="caps"&gt;TCP&lt;/span&gt; checksum covering the &lt;span class="caps"&gt;TCP&lt;/span&gt;&amp;nbsp;header).&lt;/p&gt;
&lt;p&gt;The verification&amp;nbsp;flow:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Write path:
  Application data → compress → checksum → write to disk
                                   ↓
                          store checksum in
                          parent metadata block

Read path:
  Read block from disk → verify checksum → decompress → deliver to application
       ↓                      ↓
  If checksum fails:    On redundant pool:
  &amp;quot;This block is bad&amp;quot;   → read from mirror/parity
                         → repair the bad copy
                         → deliver good data
                         → increment CKSUM error counter
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This happens transparently. Your application never sees the corrupt data. It gets the correct data, repaired silently, and the error is logged. You&amp;#8217;ll see it&amp;nbsp;in &lt;code&gt;zpool status&lt;/code&gt; output as a non-zero &lt;span class="caps"&gt;CKSUM&lt;/span&gt; count, which tells you a disk is developing problems before it fails&amp;nbsp;catastrophically.&lt;/p&gt;
&lt;p&gt;On a non-redundant pool, &lt;span class="caps"&gt;ZFS&lt;/span&gt; can detect corruption but can&amp;#8217;t repair it. The read will fail with an I/O error, which is still better than silently returning corrupted data. At least you know the data is&amp;nbsp;bad.&lt;/p&gt;
&lt;h2 id="snapshots-instant-point-in-time-copies"&gt;Snapshots: Instant Point-in-Time&amp;nbsp;Copies&lt;/h2&gt;
&lt;p&gt;A &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshot is a read-only point-in-time copy of a dataset. Snapshots are essentially free to create - they take no additional space initially, because they share all their data with the live dataset. Space is only consumed as the live dataset changes and the snapshot needs to preserve the old&amp;nbsp;blocks.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a snapshot&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@before-upgrade

&lt;span class="c1"&gt;# Create a snapshot of all datasets under a hierarchy&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;tank/jails@2026-03-13

&lt;span class="c1"&gt;# List snapshots&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;snapshot

&lt;span class="c1"&gt;# See how much space a snapshot is consuming&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;name,used,refer&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="rolling-back"&gt;Rolling&amp;nbsp;Back&lt;/h3&gt;
&lt;p&gt;If something goes wrong, you can roll a dataset back to a snapshot&amp;nbsp;instantly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Roll back to the snapshot (destroys all changes since the snapshot)&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;rollback&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@before-upgrade
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note: &lt;code&gt;zfs rollback&lt;/code&gt; can only roll back to the most recent snapshot. If you have multiple snapshots and want to go back further, you need to destroy the intermediate snapshots first (or&amp;nbsp;use &lt;code&gt;-r&lt;/code&gt; to do it&amp;nbsp;automatically):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Roll back, destroying intermediate snapshots&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;rollback&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@before-upgrade
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This makes snapshots invaluable for system upgrades. Snapshot before you upgrade, upgrade, test. If the upgrade broke something, roll back. The entire operation takes seconds, not hours of restoring from&amp;nbsp;backup.&lt;/p&gt;
&lt;h3 id="the-beauty-of-zfs-accessing-snapshots-directly"&gt;The Beauty of /.zfs: Accessing Snapshots&amp;nbsp;Directly&lt;/h3&gt;
&lt;p&gt;Here&amp;#8217;s something that surprises people who are new to &lt;span class="caps"&gt;ZFS&lt;/span&gt;: every dataset has a&amp;nbsp;hidden &lt;code&gt;.zfs&lt;/code&gt; directory at its mount point. Inside it,&amp;nbsp;under &lt;code&gt;.zfs/snapshot/&lt;/code&gt;, every snapshot is accessible as a normal read-only directory&amp;nbsp;tree.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# The .zfs directory is hidden from normal ls output&lt;/span&gt;
ls&lt;span class="w"&gt; &lt;/span&gt;-la&lt;span class="w"&gt; &lt;/span&gt;/tank/home/chris/.zfs/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;total 0
dr-xr-xr-x  2 root  wheel  2 Mar 13 19:00 .
drwxr-xr-x  5 chris chris 12 Mar 13 19:30 ..
drwxr-xr-x  5 chris chris 12 Mar 10 08:00 snapshot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# List all snapshots as directories&lt;/span&gt;
ls&lt;span class="w"&gt; &lt;/span&gt;/tank/home/chris/.zfs/snapshot/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;2026-03-10    2026-03-11    2026-03-12    2026-03-13
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Just... cd into a snapshot and find your file&lt;/span&gt;
&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/tank/home/chris/.zfs/snapshot/2026-03-12/
ls&lt;span class="w"&gt; &lt;/span&gt;Documents/
&lt;span class="c1"&gt;# → there&amp;#39;s the file you accidentally deleted yesterday&lt;/span&gt;
cp&lt;span class="w"&gt; &lt;/span&gt;Documents/important-report.pdf&lt;span class="w"&gt; &lt;/span&gt;/tank/home/chris/Documents/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;No restore tools. No backup software. No waiting. You browse the snapshot like a normal directory, find the file you need, and copy it back.&amp;nbsp;The &lt;code&gt;.zfs&lt;/code&gt; directory doesn&amp;#8217;t show up in&amp;nbsp;normal &lt;code&gt;ls&lt;/code&gt; output (it&amp;#8217;s hidden by default), but you can access it directly by name. This means regular users can recover their own accidentally deleted files without root access or administrator&amp;nbsp;intervention.&lt;/p&gt;
&lt;p&gt;This is arguably &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;#8217;s most user-facing killer feature. With automated snapshots (see the sanoid section below), users have a continuously available time machine for their&amp;nbsp;files.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;.zfs&lt;/code&gt; directory can be made visible in directory listings if you&amp;nbsp;prefer:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;snapdir&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;visible&lt;span class="w"&gt; &lt;/span&gt;tank/home
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="zfs-native-encryption"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; Native&amp;nbsp;Encryption&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; supports native dataset-level encryption, added in OpenZFS 0.8. Encryption is per-dataset, transparent to applications, and managed entirely through &lt;span class="caps"&gt;ZFS&lt;/span&gt; commands. No separate tools, no dm-crypt, no &lt;span class="caps"&gt;GELI&lt;/span&gt; - just &lt;span class="caps"&gt;ZFS&lt;/span&gt;.&lt;/p&gt;
&lt;h3 id="creating-encrypted-datasets"&gt;Creating Encrypted&amp;nbsp;Datasets&lt;/h3&gt;
&lt;p&gt;Encryption is set at dataset creation time and cannot be added to an existing unencrypted dataset (you&amp;#8217;d need to create a new encrypted dataset and move the&amp;nbsp;data):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create an encrypted dataset with a passphrase&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;encryption&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;aes-256-gcm&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keylocation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;prompt&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keyformat&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;passphrase&lt;span class="w"&gt; &lt;/span&gt;tank/private

&lt;span class="c1"&gt;# Create an encrypted dataset with a key file&lt;/span&gt;
dd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/dev/random&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;of&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/root/keys/tank-private.key&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;bs&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;count&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;encryption&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;aes-256-gcm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keylocation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;file:///root/keys/tank-private.key&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keyformat&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;raw&lt;span class="w"&gt; &lt;/span&gt;tank/secure
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Child datasets inherit encryption from their parent.&amp;nbsp;If &lt;code&gt;tank/private&lt;/code&gt; is&amp;nbsp;encrypted, &lt;code&gt;tank/private/documents&lt;/code&gt; and &lt;code&gt;tank/private/photos&lt;/code&gt; are automatically encrypted with the same&amp;nbsp;key:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# These inherit encryption from tank/private&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank/private/documents
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;tank/private/photos
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="key-management"&gt;Key&amp;nbsp;Management&lt;/h3&gt;
&lt;p&gt;When a system boots, encrypted datasets are not automatically mounted because &lt;span class="caps"&gt;ZFS&lt;/span&gt; doesn&amp;#8217;t have the decryption key yet. You need to load the key before the dataset becomes&amp;nbsp;accessible:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Load the key for a single dataset&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;load-key&lt;span class="w"&gt; &lt;/span&gt;tank/private

&lt;span class="c1"&gt;# Load keys for all encrypted datasets&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;load-key&lt;span class="w"&gt; &lt;/span&gt;-a

&lt;span class="c1"&gt;# Mount after loading key&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;mount&lt;span class="w"&gt; &lt;/span&gt;tank/private

&lt;span class="c1"&gt;# Check encryption status&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;keystatus&lt;span class="w"&gt; &lt;/span&gt;tank/private
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;NAME           PROPERTY    VALUE        SOURCE
tank/private   keystatus   available    -
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For servers that need encrypted datasets available at boot without manual intervention, you can store the key file on a separate device - for example, a &lt;span class="caps"&gt;USB&lt;/span&gt; stick that stays plugged into the server but could be removed and secured&amp;nbsp;independently:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Key file on a separate USB device mounted at boot&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keylocation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;file:///mnt/usbkey/tank-private.key&lt;span class="w"&gt; &lt;/span&gt;tank/private
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you store the key file on the boot pool itself, be aware that this provides &lt;strong&gt;no protection against physical theft&lt;/strong&gt; of the drive - the thief gets both the encrypted data and the key to decrypt it. The only scenario where a same-disk key file helps is disk disposal or &lt;span class="caps"&gt;RMA&lt;/span&gt;, where you wipe or destroy the boot partition but send the data disks out with encrypted blocks a third party can&amp;#8217;t&amp;nbsp;read.&lt;/p&gt;
&lt;p&gt;For meaningful data-at-rest protection, either keep the key on a physically separate device, or use passphrase-based keys and accept the manual unlock step at&amp;nbsp;boot.&lt;/p&gt;
&lt;h3 id="changing-keys"&gt;Changing&amp;nbsp;Keys&lt;/h3&gt;
&lt;p&gt;You can change the encryption key (or switch between passphrase and key file) without re-encrypting the&amp;nbsp;data:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Change from key file to passphrase&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;change-key&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keyformat&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;passphrase&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keylocation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;prompt&lt;span class="w"&gt; &lt;/span&gt;tank/private

&lt;span class="c1"&gt;# Change from passphrase to key file&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;change-key&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keyformat&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;raw&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;keylocation&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;file:///root/keys/new.key&lt;span class="w"&gt; &lt;/span&gt;tank/private
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is a key-wrapping operation - the actual data encryption key doesn&amp;#8217;t change, only the wrapper protecting it. It&amp;#8217;s instantaneous regardless of dataset&amp;nbsp;size.&lt;/p&gt;
&lt;h3 id="locking-datasets"&gt;Locking&amp;nbsp;Datasets&lt;/h3&gt;
&lt;p&gt;When you&amp;#8217;re done working with encrypted data, you can unload the key, making the dataset inaccessible until the key is loaded&amp;nbsp;again:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Unmount and unload the key&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;unmount&lt;span class="w"&gt; &lt;/span&gt;tank/private
zfs&lt;span class="w"&gt; &lt;/span&gt;unload-key&lt;span class="w"&gt; &lt;/span&gt;tank/private
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="zfs-sendrecv-the-killer-feature-for-backups"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; Send/Recv: The Killer Feature for&amp;nbsp;Backups&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;zfs send&lt;/code&gt; serializes a snapshot (or the difference between two snapshots) into a stream of&amp;nbsp;bytes. &lt;code&gt;zfs recv&lt;/code&gt; takes that stream and reconstructs the dataset on the other end. This is &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;#8217;s native backup and replication mechanism, and it&amp;#8217;s remarkably&amp;nbsp;powerful.&lt;/p&gt;
&lt;h3 id="basic-sendrecv"&gt;Basic&amp;nbsp;Send/Recv&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Send a full snapshot to a file&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@2026-03-13&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/backup/mailserver-2026-03-13.zfs

&lt;span class="c1"&gt;# Receive it into a different pool&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;recv&lt;span class="w"&gt; &lt;/span&gt;backup/jails/mailserver&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;&lt;span class="w"&gt; &lt;/span&gt;/backup/mailserver-2026-03-13.zfs
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="incremental-sends-only-the-changes"&gt;Incremental Sends: Only the&amp;nbsp;Changes&lt;/h3&gt;
&lt;p&gt;The real power is incremental sends. After the initial full send, subsequent sends only transfer the blocks that changed between two&amp;nbsp;snapshots:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# First: full send of the initial snapshot&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@monday&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;recv&lt;span class="w"&gt; &lt;/span&gt;backup/jails/mailserver

&lt;span class="c1"&gt;# Later: incremental send (only changes since Monday)&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@monday&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@tuesday&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;recv&lt;span class="w"&gt; &lt;/span&gt;backup/jails/mailserver
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-i&lt;/code&gt; flag tells &lt;span class="caps"&gt;ZFS&lt;/span&gt; to send only the delta between the two snapshots. On a &lt;span class="caps"&gt;500GB&lt;/span&gt; dataset where &lt;span class="caps"&gt;2GB&lt;/span&gt; changed since the last snapshot, the incremental send transfers ~&lt;span class="caps"&gt;2GB&lt;/span&gt;, not &lt;span class="caps"&gt;500GB&lt;/span&gt;. This makes daily off-site backups of large datasets entirely practical, even over modest network&amp;nbsp;links.&lt;/p&gt;
&lt;h3 id="sendrecv-over-ssh"&gt;Send/Recv Over &lt;span class="caps"&gt;SSH&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;For remote backups, pipe through &lt;span class="caps"&gt;SSH&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Send to a remote backup server&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@monday&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@tuesday&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;backup-server&lt;span class="w"&gt; &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;recv&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;mountpoint&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;none&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;canmount&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;off&lt;span class="w"&gt; &lt;/span&gt;backup/jails/mailserver

&lt;span class="c1"&gt;# For initial full replication with all properties&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;-R&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@tuesday&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;backup-server&lt;span class="w"&gt; &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;recv&lt;span class="w"&gt; &lt;/span&gt;-F&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;mountpoint&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;none&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;canmount&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;off&lt;span class="w"&gt; &lt;/span&gt;backup/jails/mailserver
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-F&lt;/code&gt; flag on the receiving end forces a rollback to match the incoming stream, discarding any local changes on the backup target. This ensures your backup remains an exact replica of the source - but be careful not to&amp;nbsp;use &lt;code&gt;-F&lt;/code&gt; against a dataset that contains data you care about independently, as any local modifications will be&amp;nbsp;destroyed.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;-o mountpoint=none -o canmount=off&lt;/code&gt; flags are equally important: without them, the backup server will try to mount the incoming datasets at their original mount points. If you&amp;#8217;re replicating a dataset&amp;nbsp;with &lt;code&gt;mountpoint=/etc&lt;/code&gt; or &lt;code&gt;mountpoint=/var/mail&lt;/code&gt;, the backup server would mount it right over its own live directories. Setting these overrides ensures received datasets are stored but never&amp;nbsp;auto-mounted.&lt;/p&gt;
&lt;h3 id="raw-sends-for-encrypted-datasets"&gt;Raw Sends for Encrypted&amp;nbsp;Datasets&lt;/h3&gt;
&lt;p&gt;When sending encrypted datasets, you can choose whether to send the data encrypted or&amp;nbsp;decrypted:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Raw send: data stays encrypted. Receiver doesn&amp;#39;t need the key.&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;--raw&lt;span class="w"&gt; &lt;/span&gt;tank/private@2026-03-13&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;backup-server&lt;span class="w"&gt; &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;recv&lt;span class="w"&gt; &lt;/span&gt;backup/private

&lt;span class="c1"&gt;# Normal send: data is decrypted, then re-encrypted (or stored plain) at the receiver.&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;tank/private@2026-03-13&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;backup-server&lt;span class="w"&gt; &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;recv&lt;span class="w"&gt; &lt;/span&gt;backup/private
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Raw sends&amp;nbsp;(&lt;code&gt;--raw&lt;/code&gt;) are essential for secure off-site backups. The backup server receives and stores encrypted data without ever having the decryption key. If the backup server is compromised, the attacker gets encrypted blocks they can&amp;#8217;t read. An important practical benefit: the receiving pool doesn&amp;#8217;t need to have encryption support enabled at all - it just stores opaque blocks. This makes raw sends ideal for replicating to older backup servers or cloud storage targets that may not run a current OpenZFS&amp;nbsp;version.&lt;/p&gt;
&lt;h3 id="sanoid-and-syncoid-automated-snapshot-management"&gt;Sanoid and Syncoid: Automated Snapshot&amp;nbsp;Management&lt;/h3&gt;
&lt;p&gt;Manually creating snapshots and&amp;nbsp;running &lt;code&gt;zfs send&lt;/code&gt; works, but it doesn&amp;#8217;t scale. &lt;a href="https://github.com/jimsalterjrs/sanoid"&gt;Sanoid&lt;/a&gt; automates snapshot creation and retention, and its companion tool &lt;strong&gt;syncoid&lt;/strong&gt;&amp;nbsp;automates &lt;code&gt;zfs send/recv&lt;/code&gt; replication.&lt;/p&gt;
&lt;p&gt;Sanoid is available in FreeBSD&amp;#8217;s package&amp;nbsp;repository:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;sanoid
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Sanoid uses a configuration file&amp;nbsp;at &lt;code&gt;/usr/local/etc/sanoid/sanoid.conf&lt;/code&gt; to define snapshot&amp;nbsp;policies:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[tank/jails]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;use_template&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;production&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;recursive&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;

&lt;span class="k"&gt;[tank/home]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;use_template&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;production&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;recursive&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;

&lt;span class="k"&gt;[tank/postgresql]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;use_template&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;production&lt;/span&gt;

&lt;span class="k"&gt;[template_production]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;hourly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;24&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;daily&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;30&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;monthly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;12&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;yearly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autosnap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autoprune&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This keeps 24 hourly, 30 daily, 12 monthly, and 2 yearly snapshots - all managed automatically. Old snapshots are pruned when they exceed the retention count.&amp;nbsp;Run &lt;code&gt;sanoid --cron&lt;/code&gt; from a cron job (typically every 15 minutes), and snapshot management becomes a solved&amp;nbsp;problem.&lt;/p&gt;
&lt;p&gt;Syncoid handles the replication&amp;nbsp;side:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Replicate a dataset to a remote server&lt;/span&gt;
syncoid&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver&lt;span class="w"&gt; &lt;/span&gt;backup-server:backup/jails/mailserver

&lt;span class="c1"&gt;# Replicate recursively (all child datasets)&lt;/span&gt;
syncoid&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;tank/jails&lt;span class="w"&gt; &lt;/span&gt;backup-server:backup/jails

&lt;span class="c1"&gt;# Use raw mode for encrypted datasets&lt;/span&gt;
syncoid&lt;span class="w"&gt; &lt;/span&gt;--sendoptions&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;w&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tank/private&lt;span class="w"&gt; &lt;/span&gt;backup-server:backup/private
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Syncoid automatically determines whether a full or incremental send is needed, creates temporary snapshots for consistency, handles&amp;nbsp;the &lt;code&gt;zfs send | ssh | zfs recv&lt;/code&gt; pipeline, and cleans up after itself. A single cron entry&amp;nbsp;running &lt;code&gt;syncoid&lt;/code&gt; gives you continuous off-site replication with minimal bandwidth&amp;nbsp;usage.&lt;/p&gt;
&lt;p&gt;A typical backup cron setup (shown here as root&amp;#8217;s personal crontab&amp;nbsp;via &lt;code&gt;crontab -e&lt;/code&gt;; if you add these&amp;nbsp;to &lt;code&gt;/etc/crontab&lt;/code&gt; instead,&amp;nbsp;add &lt;code&gt;root&lt;/code&gt; as the user field after the time&amp;nbsp;specification):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;#&lt;/span&gt; Sanoid: manage local snapshots every 15 minutes
*/15 &lt;span class="gs"&gt;* *&lt;/span&gt; * &lt;span class="gs"&gt;* /usr/local/bin/sanoid --cron&lt;/span&gt;

&lt;span class="gs"&gt;# Syncoid: replicate to backup server daily at 2 AM&lt;/span&gt;
&lt;span class="gs"&gt;0 2 *&lt;/span&gt; * * /usr/local/bin/syncoid -r tank backup-server:backup/tank
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="performance-tuning-arc-and-l2arc"&gt;Performance Tuning: &lt;span class="caps"&gt;ARC&lt;/span&gt; and &lt;span class="caps"&gt;L2ARC&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; uses a sophisticated caching system called the &lt;strong&gt;&lt;span class="caps"&gt;ARC&lt;/span&gt;&lt;/strong&gt; (Adaptive Replacement Cache). The &lt;span class="caps"&gt;ARC&lt;/span&gt; lives in &lt;span class="caps"&gt;RAM&lt;/span&gt; and caches recently and frequently accessed data. Unlike traditional filesystem caches, the &lt;span class="caps"&gt;ARC&lt;/span&gt; uses an algorithm that balances recency and frequency, keeping both &amp;#8220;recently used&amp;#8221; and &amp;#8220;frequently used&amp;#8221; data&amp;nbsp;cached.&lt;/p&gt;
&lt;p&gt;On FreeBSD, &lt;span class="caps"&gt;ZFS&lt;/span&gt; is greedy by default - it will automatically use most of your system&amp;#8217;s &lt;span class="caps"&gt;RAM&lt;/span&gt; for the &lt;span class="caps"&gt;ARC&lt;/span&gt;, leaving only a small reserve for the kernel. On a dedicated file server, this is exactly what you want and no tuning is needed. But if you&amp;#8217;re running memory-hungry jails on the same machine (PostgreSQL, application runtimes), you need to &lt;strong&gt;restrict&lt;/strong&gt; the &lt;span class="caps"&gt;ARC&lt;/span&gt; so it doesn&amp;#8217;t starve your&amp;nbsp;applications:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check current ARC size&lt;/span&gt;
sysctl&lt;span class="w"&gt; &lt;/span&gt;kstat.zfs.misc.arcstats.size

&lt;span class="c1"&gt;# Restrict ARC to 8GB on a jail host (in /boot/loader.conf)&lt;/span&gt;
&lt;span class="c1"&gt;# Leave the rest for jails and applications&lt;/span&gt;
vfs.zfs.arc_max&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;8589934592&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For systems with an available &lt;span class="caps"&gt;SSD&lt;/span&gt;, &lt;strong&gt;&lt;span class="caps"&gt;L2ARC&lt;/span&gt;&lt;/strong&gt; extends the cache to &lt;span class="caps"&gt;SSD&lt;/span&gt; storage. It&amp;#8217;s a second-level cache: data evicted from the &lt;span class="caps"&gt;RAM&lt;/span&gt;-based &lt;span class="caps"&gt;ARC&lt;/span&gt; gets written to the &lt;span class="caps"&gt;L2ARC&lt;/span&gt; device, giving you a much larger effective&amp;nbsp;cache:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Add an SSD as L2ARC&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;tank&lt;span class="w"&gt; &lt;/span&gt;cache&lt;span class="w"&gt; &lt;/span&gt;nvd1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;L2ARC&lt;/span&gt; is most effective when your working set is larger than available &lt;span class="caps"&gt;RAM&lt;/span&gt; but fits on an &lt;span class="caps"&gt;SSD&lt;/span&gt;. For read-heavy workloads on spinning disks (databases, mail servers), an &lt;span class="caps"&gt;L2ARC&lt;/span&gt; can provide dramatic performance&amp;nbsp;improvements.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Warning:&lt;/strong&gt; &lt;span class="caps"&gt;L2ARC&lt;/span&gt; isn&amp;#8217;t free. Every block cached in &lt;span class="caps"&gt;L2ARC&lt;/span&gt; requires an index pointer in your primary &lt;span class="caps"&gt;ARC&lt;/span&gt; (&lt;span class="caps"&gt;RAM&lt;/span&gt;). If you add a massive &lt;span class="caps"&gt;1TB&lt;/span&gt; NVMe as &lt;span class="caps"&gt;L2ARC&lt;/span&gt; to a system with limited &lt;span class="caps"&gt;RAM&lt;/span&gt;, the pointer overhead will starve the primary &lt;span class="caps"&gt;ARC&lt;/span&gt; and actually &lt;em&gt;degrade&lt;/em&gt; overall performance. Size your &lt;span class="caps"&gt;L2ARC&lt;/span&gt; proportionally to your available &lt;span class="caps"&gt;RAM&lt;/span&gt; - not to the size of the &lt;span class="caps"&gt;SSD&lt;/span&gt; you happen to have lying&amp;nbsp;around.&lt;/p&gt;
&lt;p&gt;Similarly, a &lt;strong&gt;&lt;span class="caps"&gt;SLOG&lt;/span&gt;&lt;/strong&gt; (Separate Log) device accelerates synchronous writes by providing a fast storage target for the &lt;span class="caps"&gt;ZFS&lt;/span&gt; Intent Log (&lt;span class="caps"&gt;ZIL&lt;/span&gt;):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Add an SSD as SLOG&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;tank&lt;span class="w"&gt; &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;nvd2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is primarily useful for &lt;span class="caps"&gt;NFS&lt;/span&gt; servers, databases&amp;nbsp;with &lt;code&gt;fsync&lt;/code&gt;-heavy workloads, and iSCSI targets where synchronous write performance&amp;nbsp;matters.&lt;/p&gt;
&lt;h2 id="zfs-and-jails-better-together"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; and Jails: Better&amp;nbsp;Together&lt;/h2&gt;
&lt;p&gt;If you read the &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;first article in this series&lt;/a&gt;, you know that Jails and &lt;span class="caps"&gt;ZFS&lt;/span&gt; are a natural pairing. Each jail gets its own dataset, which&amp;nbsp;means:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Independent snapshots:&lt;/strong&gt; Snapshot a jail before upgrading its packages. Roll back if something breaks. No impact on other&amp;nbsp;jails.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Instant cloning:&lt;/strong&gt; Want a test copy of your production mail&amp;nbsp;jail? &lt;code&gt;zfs clone&lt;/code&gt; creates one in seconds, sharing data with the original until&amp;nbsp;divergence.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Per-jail quotas:&lt;/strong&gt; Prevent one jail from consuming all available&amp;nbsp;storage.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Independent compression and recordsize:&lt;/strong&gt; A jail running PostgreSQL&amp;nbsp;gets &lt;code&gt;recordsize=8k&lt;/code&gt;. A jail serving static files&amp;nbsp;gets &lt;code&gt;recordsize=1M&lt;/code&gt;. Each workload gets the optimal&amp;nbsp;setting.&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Snapshot a jail before upgrade&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@pre-upgrade

&lt;span class="c1"&gt;# Clone a jail for testing&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;clone&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver@pre-upgrade&lt;span class="w"&gt; &lt;/span&gt;tank/jails/mailserver-test

&lt;span class="c1"&gt;# Set per-jail storage quota&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;quota&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;20G&lt;span class="w"&gt; &lt;/span&gt;tank/jails/webmail

&lt;span class="c1"&gt;# Tune recordsize per workload&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;recordsize&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;8k&lt;span class="w"&gt; &lt;/span&gt;tank/jails/database
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Cloning is particularly powerful. A clone is a writable snapshot - it starts sharing all data with the original and only consumes space as you make changes. Standing up a test environment from production data takes seconds, not&amp;nbsp;hours.&lt;/p&gt;
&lt;h2 id="common-pitfalls"&gt;Common&amp;nbsp;Pitfalls&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Not&amp;nbsp;setting &lt;code&gt;recordsize&lt;/code&gt; before importing data:&lt;/strong&gt;&amp;nbsp;Changing &lt;code&gt;recordsize&lt;/code&gt; only affects new writes. If you import a PostgreSQL database and &lt;em&gt;then&lt;/em&gt;&amp;nbsp;set &lt;code&gt;recordsize=8k&lt;/code&gt;, the existing data keeps the old block size. Set it&amp;nbsp;first.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Forgetting to scrub:&lt;/strong&gt; &lt;span class="caps"&gt;ZFS&lt;/span&gt; can only repair corruption it knows about. Without regular scrubs, corruption sits undetected until something tries to read the affected block - possibly months later when the redundant copy has also degraded. Scrub&amp;nbsp;weekly.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Letting the pool get too full:&lt;/strong&gt; &lt;span class="caps"&gt;ZFS&lt;/span&gt; performance degrades as the pool fills up, particularly above 80% capacity. This is because &lt;span class="caps"&gt;ZFS&lt;/span&gt; uses a copy-on-write allocation strategy that becomes increasingly constrained as free space fragments. Monitor pool usage and expand before you hit&amp;nbsp;80%.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Using &lt;code&gt;dedup&lt;/code&gt; without understanding the cost:&lt;/strong&gt; &lt;span class="caps"&gt;ZFS&lt;/span&gt; supports block-level deduplication, but it requires enormous amounts of &lt;span class="caps"&gt;RAM&lt;/span&gt; for the deduplication table (roughly &lt;span class="caps"&gt;5GB&lt;/span&gt; of &lt;span class="caps"&gt;RAM&lt;/span&gt; per &lt;span class="caps"&gt;TB&lt;/span&gt; of deduplicated storage). Unless you have a specific, measured use case and the &lt;span class="caps"&gt;RAM&lt;/span&gt; to support it, don&amp;#8217;t enable dedup. Use compression instead - it&amp;#8217;s nearly&amp;nbsp;free.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Not using raw send for encrypted backups:&lt;/strong&gt; If&amp;nbsp;you &lt;code&gt;zfs send&lt;/code&gt; an encrypted dataset&amp;nbsp;without &lt;code&gt;--raw&lt;/code&gt;, the data is decrypted for the send and arrives unencrypted at the receiver. This might be what you want, or it might defeat the entire purpose of encrypting it. Be intentional about which mode you&amp;nbsp;use.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; replaces an entire stack of tools - filesystem, volume manager, &lt;span class="caps"&gt;RAID&lt;/span&gt;, backup - with a single system that checksums everything, heals itself, compresses transparently, encrypts natively, and replicates efficiently. It&amp;#8217;s the foundation that makes FreeBSD&amp;#8217;s jail-based architecture practical at scale: each jail gets its own dataset with its own properties, snapshots, and&amp;nbsp;quotas.&lt;/p&gt;
&lt;p&gt;The features that matter most in practice aren&amp;#8217;t the flashy ones. It&amp;#8217;s the scrub that catches a degrading disk before data loss. It&amp;#8217;s&amp;nbsp;the &lt;code&gt;.zfs&lt;/code&gt; directory that lets a user recover a deleted file without filing a ticket. It&amp;#8217;s the incremental send that replicates &lt;span class="caps"&gt;500GB&lt;/span&gt; of jail data by transferring &lt;span class="caps"&gt;2GB&lt;/span&gt; of changes. These are the things that let you sleep at&amp;nbsp;night.&lt;/p&gt;
&lt;p&gt;If you take nothing else away from this article: enable compression, schedule scrubs, automate snapshots with sanoid, and replicate off-site with syncoid. That&amp;#8217;s a complete data management strategy in four&amp;nbsp;decisions.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/zfs/"&gt;FreeBSD Handbook - The Z File&amp;nbsp;System&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=zpool&amp;amp;sektion=8"&gt;zpool(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=zfs&amp;amp;sektion=8"&gt;zfs(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://openzfs.github.io/openzfs-docs/"&gt;OpenZFS&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/jimsalterjrs/sanoid"&gt;Sanoid/Syncoid - Policy-driven snapshot&amp;nbsp;management&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://arstechnica.com/information-technology/2020/05/zfs-101-understanding-zfs-storage-and-performance/"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; 101 - Understanding &lt;span class="caps"&gt;ZFS&lt;/span&gt; storage and performance (Ars&amp;nbsp;Technica)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://web.archive.org/web/20060428092023/http://www.opensolaris.org/os/community/zfs/docs/zfs_last.pdf"&gt;Jeff Bonwick - &lt;span class="caps"&gt;ZFS&lt;/span&gt;: The Last Word in Filesystems&lt;/a&gt; - the original Sun Microsystems&amp;nbsp;presentation&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="zfs"/><category term="filesystem"/><category term="storage"/><category term="encryption"/><category term="backup"/><category term="snapshots"/><category term="sanoid"/></entry><entry><title>FreeBSD Foundationals: Jails - From Chroot on Steroids to Full Virtual Networks</title><link href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/" rel="alternate"/><published>2026-03-02T00:00:00+01:00</published><updated>2026-03-02T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-03-02:/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/</id><summary type="html">&lt;p&gt;The first in a series on FreeBSD fundamentals. This one covers Jails from the ground up: why they exist, how classic Jails differ from &lt;span class="caps"&gt;VNET&lt;/span&gt; Jails, what epair interfaces actually are, how bridges tie it all together, and what devfs rules do for your isolation story. Practical configurations&amp;nbsp;included.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;FreeBSD Jails have been around since FreeBSD 4.0, released in the year 2000. That makes them older than Linux cgroups, older than &lt;span class="caps"&gt;LXC&lt;/span&gt;, older than Docker, and older than most people&amp;#8217;s understanding of what &amp;#8220;containers&amp;#8221; even means. Yet they remain one of the most elegant and underappreciated isolation mechanisms available on any operating&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;This article is the first in a series called &lt;strong&gt;FreeBSD Foundationals&lt;/strong&gt; - covering core FreeBSD concepts that deserve more than a man page skim. We start with Jails because they&amp;#8217;re central to how FreeBSD is deployed in practice: from hosting providers to Netflix&amp;#8217;s &lt;span class="caps"&gt;CDN&lt;/span&gt; to small mail servers on rented&amp;nbsp;hardware.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;ve used Docker or &lt;span class="caps"&gt;LXC&lt;/span&gt; on Linux, some concepts will feel familiar. But Jails are not Linux containers with a different name. The design philosophy, the networking model, and the security boundaries are fundamentally different. Understanding those differences is the&amp;nbsp;point.&lt;/p&gt;
&lt;h2 id="what-problem-do-jails-solve"&gt;What Problem Do Jails&amp;nbsp;Solve?&lt;/h2&gt;
&lt;p&gt;Unix has always&amp;nbsp;had &lt;code&gt;chroot&lt;/code&gt;, which changes the apparent root directory for a process. A process inside a chroot&amp;nbsp;sees &lt;code&gt;/&lt;/code&gt; as whatever directory you pointed it at. It can&amp;#8217;t traverse above that point in the filesystem. Useful, but limited: chroot only isolates the filesystem view. A chrooted process still shares the network stack, the process table, &lt;span class="caps"&gt;IPC&lt;/span&gt;, and - crucially - the ability to escape the chroot if it has root&amp;nbsp;privileges.&lt;/p&gt;
&lt;p&gt;Jails extend this concept into a proper isolation boundary. A jailed process&amp;nbsp;gets:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Its own filesystem root&lt;/strong&gt; (like chroot, but enforced at the kernel&amp;nbsp;level)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Its own process space&lt;/strong&gt; - processes inside a jail can only see other processes in the same&amp;nbsp;jail&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Its own hostname and network&amp;nbsp;identity&lt;/strong&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Restricted system calls&lt;/strong&gt; - no loading kernel modules, no mounting filesystems (unless explicitly allowed), no modifying the host&amp;#8217;s network&amp;nbsp;configuration&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The key insight: Jails are a kernel-enforced boundary, not a userspace trick. A root user inside a jail is &lt;em&gt;not&lt;/em&gt; equivalent to root on the host. The kernel itself refuses operations that would break the isolation. This is a fundamentally different security model from &amp;#8220;root in a Docker container&amp;#8221; on Linux, where escaping to the host has historically been a much shorter&amp;nbsp;path.&lt;/p&gt;
&lt;h2 id="classic-jails-vs-vnet-jails"&gt;Classic Jails vs &lt;span class="caps"&gt;VNET&lt;/span&gt;&amp;nbsp;Jails&lt;/h2&gt;
&lt;p&gt;There are two distinct approaches to jail networking, and understanding the difference matters for every design decision that&amp;nbsp;follows.&lt;/p&gt;
&lt;h3 id="classic-ip-based-jails"&gt;Classic (&lt;span class="caps"&gt;IP&lt;/span&gt;-Based)&amp;nbsp;Jails&lt;/h3&gt;
&lt;p&gt;In a classic jail, you assign one or more &lt;span class="caps"&gt;IP&lt;/span&gt; addresses to the jail. The jail shares the host&amp;#8217;s network stack - it uses the host&amp;#8217;s interfaces, the host&amp;#8217;s routing table, the host&amp;#8217;s firewall. The kernel simply restricts which addresses the jailed processes can bind to and communicate&amp;nbsp;from.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Classic Jail networking:

  ┌───────────────────────────────────────────────┐
  │                  Host Kernel                  │
  │                                               │
  │   Network Stack (shared)                      │
  │   ┌───────────┐                               │
  │   │  vtnet0   │  203.0.113.50                 │
  │   │           │  10.0.0.11 (bound to jail A)  │
  │   │           │  10.0.0.12 (bound to jail B)  │
  │   └───────────┘                               │
  │                                               │
  │   ┌─────────────┐    ┌─────────────┐          │
  │   │   Jail A    │    │   Jail B    │          │
  │   │ can only    │    │ can only    │          │
  │   │ bind to     │    │ bind to     │          │
  │   │ 10.0.0.11   │    │ 10.0.0.12   │          │
  │   └─────────────┘    └─────────────┘          │
  └───────────────────────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is simple and lightweight. No extra interfaces, no bridges, no routing between host and jail. The jail just sees &amp;#8220;its&amp;#8221; &lt;span class="caps"&gt;IP&lt;/span&gt; on the host&amp;#8217;s interface. Classic jails work well when you need basic process isolation and don&amp;#8217;t need the jail to run its own firewall, its own routing, or anything that requires full control over a network&amp;nbsp;stack.&lt;/p&gt;
&lt;p&gt;The limitation: because the jail shares the host&amp;#8217;s network stack, it cannot run services that need raw socket access, can&amp;#8217;t configure its own routes, can&amp;#8217;t run its own &lt;span class="caps"&gt;DHCP&lt;/span&gt; client, and can&amp;#8217;t have its own firewall rules. The jail has an &lt;span class="caps"&gt;IP&lt;/span&gt; address, but it doesn&amp;#8217;t have a&amp;nbsp;network.&lt;/p&gt;
&lt;h3 id="vnet-virtual-network-jails"&gt;&lt;span class="caps"&gt;VNET&lt;/span&gt; (Virtual Network)&amp;nbsp;Jails&lt;/h3&gt;
&lt;p&gt;&lt;span class="caps"&gt;VNET&lt;/span&gt; jails get their own complete network stack. Not just an &lt;span class="caps"&gt;IP&lt;/span&gt; - a full, independent network stack with its own interfaces, its own routing table, its own &lt;span class="caps"&gt;ARP&lt;/span&gt; cache, and the ability to run network services that would be impossible in a classic&amp;nbsp;jail.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;VNET Jail networking:

  ┌────────────────────────────────────────────────────────────────┐
  │                        Host Kernel                             │
  │                                                                │
  │  Host Network Stack            Jail A Network Stack            │
  │  ┌──────────┐                  ┌──────────┐                    │
  │  │ vtnet0   │                  │  vnet0   │                    │
  │  │ bridge0  │                  │  lo0     │                    │
  │  │ e0a_jail │                  │ (own routing table)           │
  │  └──────────┘                  └──────────┘                    │
  │       │                              │                         │
  │       └──────── epair ───────────────┘                         │
  │            (virtual ethernet cable)                            │
  └────────────────────────────────────────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is what you want for production workloads. A &lt;span class="caps"&gt;VNET&lt;/span&gt; jail can run its own &lt;span class="caps"&gt;DNS&lt;/span&gt; resolver, its own firewall (yes, pf inside a jail), handle its own IPv6 router advertisements, and generally behave like a separate machine. From the network&amp;#8217;s perspective, it &lt;em&gt;is&lt;/em&gt; a separate machine - just one that happens to share a kernel with the&amp;nbsp;host.&lt;/p&gt;
&lt;p&gt;The trade-off is complexity. &lt;span class="caps"&gt;VNET&lt;/span&gt; jails need virtual interfaces, bridges, and explicit routing between the host and jail networks. That complexity is what the rest of this article&amp;nbsp;explains.&lt;/p&gt;
&lt;h3 id="when-to-use-which"&gt;When to Use&amp;nbsp;Which&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Requirement&lt;/th&gt;
&lt;th&gt;Classic Jail&lt;/th&gt;
&lt;th&gt;&lt;span class="caps"&gt;VNET&lt;/span&gt; Jail&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Simple process isolation&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Overkill&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service needs to bind to a specific &lt;span class="caps"&gt;IP&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service needs raw sockets&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Jail needs its own routing table&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Jail needs its own firewall&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Jail runs &lt;span class="caps"&gt;DNS&lt;/span&gt; resolver (unbound, etc.)&lt;/td&gt;
&lt;td&gt;Tricky&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Jail needs &lt;span class="caps"&gt;DHCP&lt;/span&gt; client&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Performance overhead&lt;/td&gt;
&lt;td&gt;Negligible&lt;/td&gt;
&lt;td&gt;Minimal&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;For anything resembling a real server workload - a mail server, a web server with its own &lt;span class="caps"&gt;TLS&lt;/span&gt; stack, a database - &lt;span class="caps"&gt;VNET&lt;/span&gt; is the right choice. Classic jails still make sense for batch jobs, build environments, or situations where the jail is purely a filesystem and process&amp;nbsp;boundary.&lt;/p&gt;
&lt;h2 id="the-vnet-networking-model"&gt;The &lt;span class="caps"&gt;VNET&lt;/span&gt; Networking&amp;nbsp;Model&lt;/h2&gt;
&lt;p&gt;This is where most people get lost, so let&amp;#8217;s build the mental model piece by&amp;nbsp;piece.&lt;/p&gt;
&lt;h3 id="epair-interfaces-virtual-ethernet-cables"&gt;Epair Interfaces: Virtual Ethernet&amp;nbsp;Cables&lt;/h3&gt;
&lt;p&gt;An &lt;code&gt;epair&lt;/code&gt; is a pair of virtual Ethernet interfaces connected back-to-back. Think of it as a virtual Ethernet cable with a plug on each end. Creating one gives you two&amp;nbsp;interfaces: &lt;code&gt;epair0a&lt;/code&gt; and &lt;code&gt;epair0b&lt;/code&gt;. Whatever goes into one end comes out the&amp;nbsp;other.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create an epair - this gives you epair0a and epair0b&lt;/span&gt;
ifconfig&lt;span class="w"&gt; &lt;/span&gt;epair&lt;span class="w"&gt; &lt;/span&gt;create
&lt;span class="c1"&gt;# Output: epair0a&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;a&lt;/code&gt; side stays on the host.&amp;nbsp;The &lt;code&gt;b&lt;/code&gt; side gets moved into the jail. They&amp;#8217;re connected at the kernel level - traffic sent&amp;nbsp;into &lt;code&gt;epair0a&lt;/code&gt; appears&amp;nbsp;on &lt;code&gt;epair0b&lt;/code&gt;, and vice versa. No routing, no forwarding decision - it&amp;#8217;s a direct Layer 2 link, like plugging an Ethernet cable between two physical&amp;nbsp;machines.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;  Host                          Jail
  ┌──────────┐                  ┌──────────┐
  │          │    epair &amp;quot;wire&amp;quot;  │          │
  │  e0a_web ├──────────────────┤ e0b_web  │
  │          │                  │ (→vnet0) │
  └──────────┘                  └──────────┘
     stays                        moves
     on host                      into jail
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In practice, you rename both sides to something meaningful&amp;nbsp;(like &lt;code&gt;e0a_mailserver&lt;/code&gt; and &lt;code&gt;e0b_mailserver&lt;/code&gt;) so you can tell which pair belongs to which jail when you have a dozen of&amp;nbsp;them.&lt;/p&gt;
&lt;h3 id="the-bridge-connecting-jails-together"&gt;The Bridge: Connecting Jails&amp;nbsp;Together&lt;/h3&gt;
&lt;p&gt;Each epair connects exactly one jail to the host. But jails usually need to talk to each other and to the outside world. That&amp;#8217;s&amp;nbsp;where &lt;code&gt;bridge0&lt;/code&gt; comes&amp;nbsp;in.&lt;/p&gt;
&lt;p&gt;A bridge is a virtual network switch. You add the host-side&amp;nbsp;(&lt;code&gt;a&lt;/code&gt;) end of each epair to the bridge, assign the bridge an &lt;span class="caps"&gt;IP&lt;/span&gt; address (which becomes the jails&amp;#8217; default gateway), and now all jails are on the same Layer 2 network&amp;nbsp;segment:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                        ┌─────────────────────────────────────────┐
                        │              bridge0                    │
                        │          10.0.0.1/24                    │
                        │    2001:db8:1234:5678::1/64             │
  Internet              │                                         │
     │                  │  ┌──────────┐  ┌──────────┐  ┌────────┐ │
     │   NAT (pf)       │  │e0a_mail  │  │e0a_web   │  │e0a_mon │ │
  ┌──┴──────┐           │  └────┬─────┘  └────┬─────┘  └───┬────┘ │
  │ vtnet0  │           └───────┼─────────────┼────────────┼──────┘
  │ (host)  │                   │             │            │
  └─────────┘             ┌─────┴─────┐ ┌─────┴─────┐ ┌────┴──────┐
                          │ mailserver│ │  webmail  │ │ monitor   │
                          │ 10.0.0.11 │ │ 10.0.0.12 │ │ 10.0.0.13 │
                          └───────────┘ └───────────┘ └───────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This topology is the same as plugging physical servers into a physical switch. The bridge does &lt;span class="caps"&gt;MAC&lt;/span&gt; learning, forwards frames between ports, and the jails communicate at Layer 2. The host is the gateway - it has an &lt;span class="caps"&gt;IP&lt;/span&gt; on the bridge and runs pf to &lt;span class="caps"&gt;NAT&lt;/span&gt; the jail traffic out to the internet (for IPv4) and to filter what comes&amp;nbsp;in.&lt;/p&gt;
&lt;p&gt;For IPv6, if you have a large enough allocation, you can route a subnet directly to the bridge and give each jail a public IPv6 address. No &lt;span class="caps"&gt;NAT&lt;/span&gt; needed. This is exactly how dual-stack jail hosting should&amp;nbsp;work.&lt;/p&gt;
&lt;h3 id="putting-it-together-the-lifecycle-of-a-vnet-jail"&gt;Putting It Together: The Lifecycle of a &lt;span class="caps"&gt;VNET&lt;/span&gt;&amp;nbsp;Jail&lt;/h3&gt;
&lt;p&gt;When a &lt;span class="caps"&gt;VNET&lt;/span&gt; jail starts, several things happen in sequence.&amp;nbsp;The &lt;code&gt;exec.prestart&lt;/code&gt; commands&amp;nbsp;in &lt;code&gt;jail.conf&lt;/code&gt; run &lt;em&gt;before&lt;/em&gt; the jail is created, on the&amp;nbsp;host:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Create the&amp;nbsp;epair:&lt;/strong&gt; &lt;code&gt;ifconfig epair create&lt;/code&gt; produces &lt;code&gt;epair0a&lt;/code&gt; and &lt;code&gt;epair0b&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rename and bring up both&amp;nbsp;ends:&lt;/strong&gt; &lt;code&gt;ifconfig epair0a up name e0a_mailserver&lt;/code&gt;, same for&amp;nbsp;the &lt;code&gt;b&lt;/code&gt; side&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Add the host side to the&amp;nbsp;bridge:&lt;/strong&gt; &lt;code&gt;ifconfig bridge0 addm e0a_mailserver&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Then the jail starts.&amp;nbsp;The &lt;code&gt;vnet.interface&lt;/code&gt; directive moves&amp;nbsp;the &lt;code&gt;b&lt;/code&gt;-side interface into the jail&amp;#8217;s network&amp;nbsp;stack:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Move &lt;code&gt;e0b_mailserver&lt;/code&gt; into the jail&lt;/strong&gt; - it disappears from the host&amp;#8217;s interface&amp;nbsp;list&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Inside the&amp;nbsp;jail&amp;#8217;s &lt;code&gt;rc.conf&lt;/code&gt;:&lt;/strong&gt;&amp;nbsp;rename &lt;code&gt;e0b_mailserver&lt;/code&gt; to &lt;code&gt;vnet0&lt;/code&gt;, assign &lt;span class="caps"&gt;IP&lt;/span&gt; addresses, set the default route to the bridge &lt;span class="caps"&gt;IP&lt;/span&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;When the jail&amp;nbsp;stops, &lt;code&gt;exec.poststop&lt;/code&gt; cleans&amp;nbsp;up:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Destroy &lt;code&gt;e0a_mailserver&lt;/code&gt;&lt;/strong&gt; - this automatically destroys the&amp;nbsp;paired &lt;code&gt;e0b_mailserver&lt;/code&gt; too&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;An important detail about&amp;nbsp;the &lt;code&gt;exec.prestart&lt;/code&gt; lines: each line runs in its own shell invocation.&amp;nbsp;The &lt;code&gt;$epair0&lt;/code&gt; variable from&amp;nbsp;the &lt;code&gt;ifconfig epair create&lt;/code&gt; call only exists within the same line, which is why the create, rename and bring-up are chained&amp;nbsp;with &lt;code&gt;&amp;amp;&amp;amp;&lt;/code&gt; into a single line in the config&amp;nbsp;below.&lt;/p&gt;
&lt;p&gt;Here&amp;#8217;s what this looks like&amp;nbsp;in &lt;code&gt;jail.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;mailserver&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Filesystem and process isolation&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/jails/mailserver&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hostname&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mailserver&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;enforce_statfs&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;devfs&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;devfs_ruleset&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="c1"&gt;# reset environment before executing jail commands&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Standard start/stop&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/bin/sh /etc/rc&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/bin/sh /etc/rc.shutdown&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;consolelog&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;jails&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mailserver_console&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# VNET networking&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;vnet&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;vnet&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;e0b_mailserver&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Create epair, rename, bridge - runs on the host before jail starts&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;epair0=\$(ifconfig epair create) &amp;amp;&amp;amp; ifconfig \${epair0} up name e0a_mailserver &amp;amp;&amp;amp; ifconfig \${epair0%a}b up name e0b_mailserver&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig bridge0 addm e0a_mailserver&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig e0a_mailserver description &lt;/span&gt;&lt;span class="se"&gt;\&amp;quot;&lt;/span&gt;&lt;span class="s2"&gt;vnet0 host interface for Jail mailserver&lt;/span&gt;&lt;span class="se"&gt;\&amp;quot;&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Clean up - runs on the host after jail stops&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;poststop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig e0a_mailserver destroy&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And the&amp;nbsp;corresponding &lt;code&gt;rc.conf&lt;/code&gt; inside the jail&amp;nbsp;at &lt;code&gt;/jails/mailserver/etc/rc.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Rename the interface that was moved in by vnet.interface&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_e0b_mailserver_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vnet0&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# IPv4&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.0.0.11 netmask 255.255.255.0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.0.0.1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# IPv6 (public address, no NAT needed)&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:1234:5678::11/64&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1234:5678::1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Description for ifconfig output&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_descr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;jail interface for bridge0&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Standard jail housekeeping&lt;/span&gt;
&lt;span class="nv"&gt;syslogd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-ss&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_submit_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_outbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_msp_queue_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;cron_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-J 60&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;moused_nondefault_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;dumpdev&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Actual services&lt;/span&gt;
&lt;span class="nv"&gt;postfix_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;dovecot_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rspamd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;local_unbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few things to note about&amp;nbsp;the &lt;code&gt;rc.conf&lt;/code&gt; pattern:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;syslogd_flags="-ss"&lt;/code&gt;&lt;/strong&gt; disables syslogd&amp;#8217;s network socket. Without this, syslogd inside the jail tries to open a socket it shouldn&amp;#8217;t&amp;nbsp;need.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;sendmail_*="NO"&lt;/code&gt;&lt;/strong&gt; disables all sendmail components. Even if you&amp;#8217;re running Postfix, the base system&amp;#8217;s sendmail cron jobs will try to start&amp;nbsp;otherwise.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;cron_flags="-J 60"&lt;/code&gt;&lt;/strong&gt; tells cron to stagger job execution with a random delay of up to 60 seconds. Prevents all jails from firing cron jobs at the exact same&amp;nbsp;second.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="host-configuration-enabling-the-infrastructure"&gt;Host Configuration: Enabling the&amp;nbsp;Infrastructure&lt;/h2&gt;
&lt;p&gt;The host needs a few things configured&amp;nbsp;in &lt;code&gt;/etc/rc.conf&lt;/code&gt; before jails can use &lt;span class="caps"&gt;VNET&lt;/span&gt;&amp;nbsp;networking:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create the bridge at boot&lt;/span&gt;
&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bridge0&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Bridge gets the gateway IP for the jail network&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.0.0.1 netmask 255.255.255.0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:1234:5678::1/64&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Enable IP forwarding (the host is the jails&amp;#39; router)&lt;/span&gt;
&lt;span class="nv"&gt;gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Enable the jail subsystem and packet filter&lt;/span&gt;
&lt;span class="nv"&gt;jail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;gateway_enable&lt;/code&gt; lines are critical. Without them, the kernel won&amp;#8217;t forward packets between the bridge (jail network) and the external interface. Your jails would have &lt;span class="caps"&gt;IP&lt;/span&gt; addresses but no connectivity beyond the&amp;nbsp;bridge.&lt;/p&gt;
&lt;h2 id="firewalling-with-pf"&gt;Firewalling with&amp;nbsp;pf&lt;/h2&gt;
&lt;p&gt;With &lt;span class="caps"&gt;VNET&lt;/span&gt; jails, the host is the router. All traffic between jails and the internet passes through the host&amp;#8217;s network stack, which means pf on the host controls&amp;nbsp;everything.&lt;/p&gt;
&lt;h3 id="nat-for-ipv4"&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt; for&amp;nbsp;IPv4&lt;/h3&gt;
&lt;p&gt;Unless you have a large IPv4 allocation (unlikely and expensive), your jails will use private addresses on the bridge and &lt;span class="caps"&gt;NAT&lt;/span&gt; through the host&amp;#8217;s public &lt;span class="caps"&gt;IP&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;10.0.0.0/24&amp;quot;&lt;/span&gt;

&lt;span class="cp"&gt;# NAT jail traffic going to the internet&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The parentheses&amp;nbsp;around &lt;code&gt;($ext_if)&lt;/code&gt; tell pf to dynamically resolve the interface&amp;#8217;s current &lt;span class="caps"&gt;IP&lt;/span&gt;. If the host&amp;#8217;s &lt;span class="caps"&gt;IP&lt;/span&gt; changes (&lt;span class="caps"&gt;DHCP&lt;/span&gt;), the &lt;span class="caps"&gt;NAT&lt;/span&gt; rule adapts&amp;nbsp;automatically.&lt;/p&gt;
&lt;h3 id="redirecting-inbound-traffic"&gt;Redirecting Inbound&amp;nbsp;Traffic&lt;/h3&gt;
&lt;p&gt;For services that need to be reachable from the internet over IPv4,&amp;nbsp;use &lt;code&gt;rdr&lt;/code&gt; to forward incoming connections to the jail&amp;#8217;s private&amp;nbsp;address:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;mail_ipv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;10.0.0.11&amp;quot;&lt;/span&gt;

&lt;span class="cp"&gt;# SMTP from anywhere&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;mail_ipv4&lt;/span&gt;

&lt;span class="cp"&gt;# IMAP and submission&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;143&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;587&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4190&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;mail_ipv4&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="ipv6-no-nat-needed"&gt;IPv6: No &lt;span class="caps"&gt;NAT&lt;/span&gt;&amp;nbsp;Needed&lt;/h3&gt;
&lt;p&gt;With IPv6, each jail has a globally routable address. No &lt;span class="caps"&gt;NAT&lt;/span&gt;,&amp;nbsp;no &lt;code&gt;rdr&lt;/code&gt; - just pass&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mail_ipv6 = &amp;quot;2001:db8:1234:5678::11&amp;quot;

pass in quick on $ext_if inet6 proto tcp from any to $mail_ipv6 \
    port 25 flags S/SA keep state
pass in quick on $ext_if inet6 proto tcp from any to $mail_ipv6 \
    port {143, 587, 4190} flags S/SA keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is one of the practical advantages of deploying jails on a host with IPv6: the networking model becomes dramatically simpler. Each jail is a first-class citizen on the internet, directly addressable, with filtering handled by pass/block rules rather than &lt;span class="caps"&gt;NAT&lt;/span&gt; and redirection&amp;nbsp;gymnastics.&lt;/p&gt;
&lt;h3 id="jail-egress"&gt;Jail&amp;nbsp;Egress&lt;/h3&gt;
&lt;p&gt;Jails also need to initiate outbound connections (&lt;span class="caps"&gt;DNS&lt;/span&gt; queries, package downloads, sending mail). Allow this with a broad egress rule on the&amp;nbsp;bridge:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;jail_net = &amp;quot;10.0.0.0/24&amp;quot;
jail_net6 = &amp;quot;2001:db8:1234:5678::/64&amp;quot;

&lt;span class="gh"&gt;#&lt;/span&gt; Allow jails to reach the internet, but not each other&amp;#39;s private range
pass in quick on bridge0 from $jail_net to ! $jail_net keep state
pass in quick on bridge0 inet6 from $jail_net6 to ! $jail_net6 keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;! $jail_net&lt;/code&gt; exclusion means this rule only matches traffic leaving the jail network towards the internet. Traffic between jails on the same bridge is switched at Layer 2 and never passes through pf in the default FreeBSD configuration (controlled&amp;nbsp;by &lt;code&gt;net.link.bridge.pfil_member&lt;/code&gt; and &lt;code&gt;net.link.bridge.pfil_bridge&lt;/code&gt; sysctls, both off by default). If you need to restrict inter-jail communication, that requires either separate bridges per jail or&amp;nbsp;enabling &lt;code&gt;pfil_member&lt;/code&gt; and adding explicit block rules on the bridge member&amp;nbsp;interfaces.&lt;/p&gt;
&lt;h2 id="devfs-rules-controlling-device-access"&gt;devfs Rules: Controlling Device&amp;nbsp;Access&lt;/h2&gt;
&lt;p&gt;When a jail&amp;nbsp;has &lt;code&gt;mount.devfs&lt;/code&gt; enabled, it gets&amp;nbsp;a &lt;code&gt;/dev&lt;/code&gt; filesystem. But you don&amp;#8217;t want a jail to see the host&amp;#8217;s disk devices, &lt;span class="caps"&gt;USB&lt;/span&gt; devices, or hardware random number generators it shouldn&amp;#8217;t touch. devfs rulesets control which device nodes appear inside the&amp;nbsp;jail.&lt;/p&gt;
&lt;p&gt;The relevant rulesets are defined&amp;nbsp;in &lt;code&gt;/etc/devfs.rules&lt;/code&gt; on the host. FreeBSD ships with a default set, and ruleset 4&amp;nbsp;(&lt;code&gt;devfsrules_jail&lt;/code&gt;) is specifically designed for&amp;nbsp;jails:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# View the built-in rulesets&lt;/span&gt;
devfs&lt;span class="w"&gt; &lt;/span&gt;rule&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;show
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Ruleset 4 typically hides everything and then unhides only what a jail&amp;nbsp;needs:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;/dev/null&lt;/code&gt;, &lt;code&gt;/dev/zero&lt;/code&gt;, &lt;code&gt;/dev/random&lt;/code&gt;, &lt;code&gt;/dev/urandom&lt;/code&gt; - standard Unix&amp;nbsp;devices&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/dev/fd/*&lt;/code&gt; - file&amp;nbsp;descriptors&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/dev/stdin&lt;/code&gt;, &lt;code&gt;/dev/stdout&lt;/code&gt;, &lt;code&gt;/dev/stderr&lt;/code&gt; - standard&amp;nbsp;streams&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;What it &lt;em&gt;doesn&amp;#8217;t&lt;/em&gt;&amp;nbsp;include: &lt;code&gt;/dev/ad*&lt;/code&gt;, &lt;code&gt;/dev/da*&lt;/code&gt; (disk&amp;nbsp;devices), &lt;code&gt;/dev/mem&lt;/code&gt; (physical&amp;nbsp;memory), &lt;code&gt;/dev/kmem&lt;/code&gt; (kernel&amp;nbsp;memory), &lt;code&gt;/dev/io&lt;/code&gt; (I/O port access). A jailed process cannot read raw disk data or probe kernel memory, even as&amp;nbsp;root.&lt;/p&gt;
&lt;p&gt;Notably, ruleset 4 also does &lt;em&gt;not&lt;/em&gt;&amp;nbsp;unhide &lt;code&gt;/dev/bpf*&lt;/code&gt; (Berkeley Packet Filter devices). This matters for &lt;span class="caps"&gt;VNET&lt;/span&gt; jails: without bpf, tools&amp;nbsp;like &lt;code&gt;tcpdump&lt;/code&gt; and &lt;code&gt;dhclient&lt;/code&gt; inside the jail won&amp;#8217;t work. If you need either, you&amp;#8217;ll want a custom ruleset that adds bpf access - shown&amp;nbsp;below.&lt;/p&gt;
&lt;p&gt;You reference the ruleset&amp;nbsp;in &lt;code&gt;jail.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;devfs_ruleset = 4;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you need to customize device access (for example, giving a jail access&amp;nbsp;to &lt;code&gt;/dev/pf&lt;/code&gt; for its own packet filter,&amp;nbsp;or &lt;code&gt;/dev/zvol/*&lt;/code&gt; for &lt;span class="caps"&gt;ZFS&lt;/span&gt; volume access), you create a custom ruleset&amp;nbsp;in &lt;code&gt;/etc/devfs.rules&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[devfsrules_jail_with_pf=100]&lt;/span&gt;
&lt;span class="na"&gt;add include $devfsrules_hide_all&lt;/span&gt;
&lt;span class="na"&gt;add include $devfsrules_unhide_basic&lt;/span&gt;
&lt;span class="na"&gt;add include $devfsrules_unhide_login&lt;/span&gt;
&lt;span class="na"&gt;add path &amp;#39;bpf*&amp;#39; unhide&lt;/span&gt;
&lt;span class="na"&gt;add path pf unhide&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then reference it&amp;nbsp;as &lt;code&gt;devfs_ruleset = 100&lt;/code&gt; in the jail&amp;#8217;s&amp;nbsp;configuration.&lt;/p&gt;
&lt;h2 id="security-what-jails-actually-prevent"&gt;Security: What Jails Actually&amp;nbsp;Prevent&lt;/h2&gt;
&lt;p&gt;Jails enforce security at the kernel level. Here&amp;#8217;s what a root user inside a jail &lt;em&gt;cannot&lt;/em&gt; do, regardless of their privilege&amp;nbsp;level:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Load or unload kernel modules&lt;/strong&gt; -&amp;nbsp;no &lt;code&gt;kldload&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;kldunload&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Mount or unmount filesystems&lt;/strong&gt; -&amp;nbsp;unless &lt;code&gt;enforce_statfs&lt;/code&gt; and &lt;code&gt;allow.mount*&lt;/code&gt; are explicitly&amp;nbsp;relaxed&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Modify the host&amp;#8217;s network&lt;/strong&gt; - can&amp;#8217;t touch the host&amp;#8217;s interfaces, routes, or&amp;nbsp;firewall&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;See processes outside the jail&lt;/strong&gt;&amp;nbsp;- &lt;code&gt;ps aux&lt;/code&gt; shows only jail-local&amp;nbsp;processes&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Access raw disk devices&lt;/strong&gt; - blocked by devfs&amp;nbsp;rules&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Modify the running kernel&lt;/strong&gt; - no access&amp;nbsp;to &lt;code&gt;/dev/mem&lt;/code&gt;, &lt;code&gt;/dev/kmem&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Create device nodes&lt;/strong&gt; -&amp;nbsp;no &lt;code&gt;mknod&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Modify the system clock&lt;/strong&gt; -&amp;nbsp;no &lt;code&gt;adjtime&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;settimeofday&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="enforce_statfs"&gt;enforce_statfs&lt;/h3&gt;
&lt;p&gt;The &lt;code&gt;enforce_statfs&lt;/code&gt; directive controls what the jail can see about mounted&amp;nbsp;filesystems:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;enforce_statfs = 0&lt;/code&gt; - jail sees all mount points (don&amp;#8217;t use&amp;nbsp;this)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;enforce_statfs = 1&lt;/code&gt; - jail sees mount points beneath its&amp;nbsp;root&lt;/li&gt;
&lt;li&gt;&lt;code&gt;enforce_statfs = 2&lt;/code&gt; - jail sees only its own mount points&amp;nbsp;(recommended)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;At level&amp;nbsp;2, &lt;code&gt;df&lt;/code&gt; inside the jail shows only the jail&amp;#8217;s own filesystems. The jail can&amp;#8217;t discover the host&amp;#8217;s &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool layout, other jails&amp;#8217; mount points, or the overall disk&amp;nbsp;topology.&lt;/p&gt;
&lt;h2 id="managing-jails"&gt;Managing&amp;nbsp;Jails&lt;/h2&gt;
&lt;h3 id="starting-and-stopping"&gt;Starting and&amp;nbsp;Stopping&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Start all jails&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;jail&lt;span class="w"&gt; &lt;/span&gt;start

&lt;span class="c1"&gt;# Start a specific jail&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;jail&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;mailserver

&lt;span class="c1"&gt;# Stop a specific jail&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;jail&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt; &lt;/span&gt;webmail

&lt;span class="c1"&gt;# Restart (stop + start) a specific jail&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;jail&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;mailserver
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="entering-a-running-jail"&gt;Entering a Running&amp;nbsp;Jail&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Get a shell inside a jail&lt;/span&gt;
jexec&lt;span class="w"&gt; &lt;/span&gt;mailserver&lt;span class="w"&gt; &lt;/span&gt;/bin/sh

&lt;span class="c1"&gt;# Run a specific command&lt;/span&gt;
jexec&lt;span class="w"&gt; &lt;/span&gt;mailserver&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;update

&lt;span class="c1"&gt;# See running jails&lt;/span&gt;
jls
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;jls&lt;/code&gt; shows all active jails with their &lt;span class="caps"&gt;JID&lt;/span&gt; (Jail &lt;span class="caps"&gt;ID&lt;/span&gt;), &lt;span class="caps"&gt;IP&lt;/span&gt; addresses, hostname, and path. It&amp;#8217;s the FreeBSD equivalent&amp;nbsp;of &lt;code&gt;docker ps&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="filesystem-zfs-datasets"&gt;Filesystem: &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;nbsp;Datasets&lt;/h3&gt;
&lt;p&gt;While you can use any directory as a jail root, &lt;span class="caps"&gt;ZFS&lt;/span&gt; datasets are the standard approach for&amp;nbsp;production:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a dataset for your jail infrastructure&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;zroot/jails

&lt;span class="c1"&gt;# Create a dataset for a specific jail&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;zroot/jails/mailserver

&lt;span class="c1"&gt;# Install a fresh userland into it&lt;/span&gt;
bsdinstall&lt;span class="w"&gt; &lt;/span&gt;jail&lt;span class="w"&gt; &lt;/span&gt;/jails/mailserver
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; gives you snapshots&amp;nbsp;(&lt;code&gt;zfs snapshot zroot/jails/mailserver@before-upgrade&lt;/code&gt;), cloning (create a new jail from an existing snapshot in seconds), quotas, compression, and send/receive for backup and migration. This is comparable to Docker&amp;#8217;s layered images, but without the overlay filesystem&amp;nbsp;complexity.&lt;/p&gt;
&lt;h3 id="nullfs-mounts-sharing-data-between-host-and-jail"&gt;Nullfs Mounts: Sharing Data Between Host and&amp;nbsp;Jail&lt;/h3&gt;
&lt;p&gt;Sometimes a jail needs access to data that lives outside its filesystem root.&amp;nbsp;The &lt;code&gt;nullfs&lt;/code&gt; mount (a loopback mount, similar&amp;nbsp;to &lt;code&gt;mount --bind&lt;/code&gt; on Linux) makes a host directory visible inside the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# In jail.conf&lt;/span&gt;
&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/vmail /jails/mailserver/var/vmail nullfs rw 0 0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This mounts the&amp;nbsp;host&amp;#8217;s &lt;code&gt;/var/vmail&lt;/code&gt; at &lt;code&gt;/jails/mailserver/var/vmail&lt;/code&gt;. The mail jail sees it as a local directory. This is useful for shared mail spools, database directories on fast storage, or any data that should persist independently of the jail&amp;#8217;s&amp;nbsp;lifecycle.&lt;/p&gt;
&lt;h2 id="a-complete-example-three-jail-architecture"&gt;A Complete Example: Three-Jail&amp;nbsp;Architecture&lt;/h2&gt;
&lt;p&gt;Here&amp;#8217;s a complete topology for a generic server running three &lt;span class="caps"&gt;VNET&lt;/span&gt;&amp;nbsp;jails:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                                  Internet
                                     │
                            ┌────────┴────────┐
                            │     vtnet0      │
                            │   203.0.113.50  │
                            │  (public IPv4)  │
                            └────────┬────────┘
                                     │
                                 pf (NAT/RDR)
                                     │
     ┌───────────────────────────────┴──────────────────────────────┐
     │                          bridge0                             │
     │                      10.0.0.1/24                             │
     │               2001:db8:1234:5678::1/64                       │
     │                                                              │
     │  ┌───────────────┐  ┌──────────────┐  ┌───────────────────┐  │
     │  │e0a_mailserver │  │ e0a_webmail  │  │  e0a_monitor      │  │
     │  └───────┬───────┘  └──────┬───────┘  └─────────┬─────────┘  │
     └──────────┼─────────────────┼────────────────────┼────────────┘
                │ epair           │ epair              │ epair
     ┌──────────┴──────┐  ┌───────┴───────┐  ┌─────────┴──────────┐
     │   mailserver    │  │    webmail    │  │      monitor       │
     │   10.0.0.11     │  │   10.0.0.12   │  │     10.0.0.13      │
     │                 │  │               │  │                    │
     │   Postfix       │  │    nginx      │  │   Prometheus       │
     │   Dovecot       │  │    php-fpm    │  │   Grafana          │
     │   Rspamd        │  │               │  │                    │
     │   Unbound       │  │               │  │                    │
     └─────────────────┘  └───────────────┘  └────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Each jail gets its own epair, its own &lt;span class="caps"&gt;IP&lt;/span&gt;, and runs only the services it needs. The mailserver jail handles &lt;span class="caps"&gt;SMTP&lt;/span&gt;, &lt;span class="caps"&gt;IMAP&lt;/span&gt;, and spam filtering. The webmail jail runs a web interface. The monitor jail runs Prometheus and Grafana for metrics&amp;nbsp;collection.&lt;/p&gt;
&lt;p&gt;This separation means a vulnerability in the webmail application (&lt;span class="caps"&gt;PHP&lt;/span&gt;, web framework) doesn&amp;#8217;t automatically grant access to the mail spool, the &lt;span class="caps"&gt;SMTP&lt;/span&gt; relay, or the host system. Each blast radius is&amp;nbsp;contained.&lt;/p&gt;
&lt;h2 id="common-pitfalls"&gt;Common&amp;nbsp;Pitfalls&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Forgetting &lt;code&gt;gateway_enable&lt;/code&gt;:&lt;/strong&gt; Your jails will get IPs, the bridge will be up, but nothing can reach the internet. The host kernel silently drops forwarded&amp;nbsp;packets.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Not&amp;nbsp;creating &lt;code&gt;/var/log/jails/&lt;/code&gt;:&lt;/strong&gt; If&amp;nbsp;the &lt;code&gt;exec.consolelog&lt;/code&gt; directory doesn&amp;#8217;t exist, jail startup fails silently. Create it ahead of&amp;nbsp;time:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/var/log/jails
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Interface naming conflicts:&lt;/strong&gt; If two jails try to create epairs simultaneously without unique names, you get collisions. The naming convention&amp;nbsp;in &lt;code&gt;jail.conf&lt;/code&gt; (&lt;code&gt;e0a_&amp;lt;jailname&amp;gt;&lt;/code&gt; / &lt;code&gt;e0b_&amp;lt;jailname&amp;gt;&lt;/code&gt;) avoids&amp;nbsp;this.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;DNS&lt;/span&gt; resolution inside jails:&lt;/strong&gt; Jails don&amp;#8217;t automatically inherit the&amp;nbsp;host&amp;#8217;s &lt;code&gt;/etc/resolv.conf&lt;/code&gt;. Either copy it into the jail&amp;#8217;s root, run a local resolver like unbound inside the jail, or&amp;nbsp;point &lt;code&gt;resolv.conf&lt;/code&gt; at the bridge &lt;span class="caps"&gt;IP&lt;/span&gt; if the host runs a resolver&amp;nbsp;there.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Forgetting to enable jail services:&lt;/strong&gt;&amp;nbsp;After &lt;code&gt;bsdinstall jail&lt;/code&gt;, the base system is minimal. Services need to be installed&amp;nbsp;(&lt;code&gt;pkg -j mailserver install postfix&lt;/code&gt;) and enabled in the&amp;nbsp;jail&amp;#8217;s &lt;code&gt;rc.conf&lt;/code&gt;. Don&amp;#8217;t&amp;nbsp;forget &lt;code&gt;sshd_enable="YES"&lt;/code&gt; if you want to &lt;span class="caps"&gt;SSH&lt;/span&gt; into the jail directly&amp;nbsp;(though &lt;code&gt;jexec&lt;/code&gt; from the host is often&amp;nbsp;sufficient).&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Jails are not a historical curiosity. They&amp;#8217;re a production-grade isolation mechanism backed by over 25 years of kernel-level enforcement. &lt;span class="caps"&gt;VNET&lt;/span&gt; gives each jail a real network identity, epairs provide the plumbing, bridges create the fabric, and pf ties it all together with filtering and &lt;span class="caps"&gt;NAT&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The mental model to carry away: a &lt;span class="caps"&gt;VNET&lt;/span&gt; jail is a separate machine that shares a kernel with the host. It has its own network stack, its own filesystem view, and its own process space. What it doesn&amp;#8217;t have is the ability to affect the host - and that&amp;#8217;s enforced by the kernel, not by&amp;nbsp;convention.&lt;/p&gt;
&lt;p&gt;The next article in this series will cover &lt;span class="caps"&gt;ZFS&lt;/span&gt; - the filesystem that makes jail management, snapshots, and backups actually practical at&amp;nbsp;scale.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook -&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=jail&amp;amp;sektion=8"&gt;jail(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=jail.conf&amp;amp;sektion=5"&gt;jail.conf(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=devfs.rules&amp;amp;sektion=5"&gt;devfs.rules(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=epair&amp;amp;sektion=4"&gt;epair(4) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=if_bridge&amp;amp;sektion=4"&gt;if_bridge(4) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pf.conf&amp;amp;sektion=5"&gt;pf.conf(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://papers.freebsd.org/2000/phk-jails/"&gt;Poul-Henning Kamp, Robert &lt;span class="caps"&gt;N. M.&lt;/span&gt; Watson - Jails: Confining the Omnipotent Root&lt;/a&gt; - the original 2000 paper introducing&amp;nbsp;Jails&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="jails"/><category term="vnet"/><category term="networking"/><category term="containers"/><category term="isolation"/><category term="security"/><category term="pf"/></entry><entry><title>HTTP/3 on FreeBSD: Getting QUIC Working with nginx in a Bastille Jail</title><link href="https://hofstede.it/http3-on-freebsd-getting-quic-working-with-nginx-in-a-bastille-jail/" rel="alternate"/><published>2026-02-21T00:00:00+01:00</published><updated>2026-02-21T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-02-21:/http3-on-freebsd-getting-quic-working-with-nginx-in-a-bastille-jail/</id><summary type="html">&lt;p&gt;What looked like a simple nginx config change turned into a tour through &lt;span class="caps"&gt;SSL&lt;/span&gt; library incompatibilities, pf firewall rules for a new protocol, and a multi-worker affinity problem that only shows up under real traffic. A complete guide to getting &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 working with nginx 1.28 on FreeBSD 15.0 inside a Bastille&amp;nbsp;jail.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Nginx" src="https://hofstede.it/images/2026-02-21-http3-quic-nginx-freebsd.png" title="Nginx Logo"&gt;&lt;/p&gt;
&lt;p&gt;What looks like a simple nginx config change turns out to involve &lt;span class="caps"&gt;SSL&lt;/span&gt; library compatibility, firewall rules for a new protocol, and a subtle multi-process routing problem that only surfaces under real traffic. This documents what it actually took to get &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 (&lt;span class="caps"&gt;QUIC&lt;/span&gt;) working on nginx 1.28 inside a FreeBSD 15.0 Bastille jail - serving the Mastodon instance at &lt;a href="https://burningboard.net"&gt;burningboard.net&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re running nginx on FreeBSD and want &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3, this should save you several hours of&amp;nbsp;troubleshooting.&lt;/p&gt;
&lt;h2 id="prerequisites"&gt;Prerequisites&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;FreeBSD 12+ (this guide uses FreeBSD 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;)&lt;/li&gt;
&lt;li&gt;nginx 1.25+&amp;nbsp;with &lt;code&gt;HTTPV3=on&lt;/code&gt; and &lt;code&gt;HTTPV3_BORING=on&lt;/code&gt; (built from&amp;nbsp;ports)&lt;/li&gt;
&lt;li&gt;A valid &lt;span class="caps"&gt;TLS&lt;/span&gt; certificate (Let&amp;#8217;s Encrypt or&amp;nbsp;similar)&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;UDP&lt;/span&gt; port 443 open in your&amp;nbsp;firewall&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;DNS&lt;/span&gt; A and &lt;span class="caps"&gt;AAAA&lt;/span&gt; records pointing to your&amp;nbsp;server&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="step-1-the-listen-directives"&gt;Step 1: The Listen&amp;nbsp;Directives&lt;/h2&gt;
&lt;p&gt;The most common first mistake:&amp;nbsp;replacing &lt;code&gt;listen 443 ssl&lt;/code&gt; with &lt;code&gt;listen 443 quic&lt;/code&gt;. &lt;span class="caps"&gt;QUIC&lt;/span&gt; doesn&amp;#8217;t replace &lt;span class="caps"&gt;TCP&lt;/span&gt; - it runs alongside it. Browsers always connect via regular &lt;span class="caps"&gt;HTTPS&lt;/span&gt; (&lt;span class="caps"&gt;TCP&lt;/span&gt;) first, then upgrade to &lt;span class="caps"&gt;QUIC&lt;/span&gt;/&lt;span class="caps"&gt;HTTP3&lt;/span&gt; only after seeing&amp;nbsp;an &lt;code&gt;Alt-Svc&lt;/code&gt; header announcing that &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 is&amp;nbsp;available.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Wrong:&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;http2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This makes nginx listen only on &lt;span class="caps"&gt;UDP&lt;/span&gt;. No browser can connect at all because the initial connection is always &lt;span class="caps"&gt;TCP&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Correct:&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;http2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Both &lt;span class="caps"&gt;TCP&lt;/span&gt;&amp;nbsp;(&lt;code&gt;ssl&lt;/code&gt;) and &lt;span class="caps"&gt;UDP&lt;/span&gt;&amp;nbsp;(&lt;code&gt;quic&lt;/code&gt;) listeners must be present in the same server&amp;nbsp;block.&lt;/p&gt;
&lt;p&gt;The reason &lt;span class="caps"&gt;TCP&lt;/span&gt; can never be skipped is that the &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 upgrade is self-announcing: the server tells the browser about &lt;span class="caps"&gt;QUIC&lt;/span&gt; support &lt;em&gt;inside&lt;/em&gt; an &lt;span class="caps"&gt;HTTP&lt;/span&gt; response, which first requires a working &lt;span class="caps"&gt;TCP&lt;/span&gt; connection. Here is the full&amp;nbsp;sequence:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt; Client                              nginx (:443)
   │                                     │
   │   ① TCP + TLS handshake             │
   ├────────────────────────────────────&amp;gt;┤  \
   │&amp;lt;────────────────────────────────────┤   TCP (always required for first contact)
   │                                     │  /
   │   ② First request - still over TCP  │
   ├─── GET / ──────────────────────────&amp;gt;┤  \
   │&amp;lt;── 200 OK ──────────────────────────┤   TCP
   │    Alt-Svc: h3=&amp;quot;:443&amp;quot;; ma=86400     │  /  ← browser learns HTTP/3 is available
   │                                     │
   │   (browser caches Alt-Svc)          │
   │                                     │
   │   ③ Subsequent requests via QUIC    │
   ├═══ QUIC handshake (UDP) ═══════════&amp;gt;┤  \
   │&amp;lt;════════════════════════════════════┤   UDP (HTTP/3)
   ├═══ GET /page ═══════════════════════&amp;gt;┤   lower latency, multiplexed
   │&amp;lt;═══ 200 OK ══════════════════════════┤  /
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is why both&amp;nbsp;the &lt;code&gt;ssl&lt;/code&gt; and &lt;code&gt;quic&lt;/code&gt; listen directives are always required together: one to serve the first response that announces &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3, the other to handle all subsequent &lt;span class="caps"&gt;QUIC&lt;/span&gt;&amp;nbsp;connections.&lt;/p&gt;
&lt;h2 id="step-2-the-alt-svc-header"&gt;Step 2: The Alt-Svc&amp;nbsp;Header&lt;/h2&gt;
&lt;p&gt;Browsers discover &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 support through&amp;nbsp;the &lt;code&gt;Alt-Svc&lt;/code&gt; response header. Without it, no browser will ever attempt a &lt;span class="caps"&gt;QUIC&lt;/span&gt;&amp;nbsp;connection:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Alt-Svc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;h3=&amp;quot;:443&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;ma=86400&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;always&lt;/code&gt; keyword ensures the header is sent even on error responses and redirects, not just successful&amp;nbsp;200s.&lt;/p&gt;
&lt;h3 id="the-add_header-inheritance-trap"&gt;The &lt;code&gt;add_header&lt;/code&gt; Inheritance&amp;nbsp;Trap&lt;/h3&gt;
&lt;p&gt;This is a subtle but critical nginx&amp;nbsp;behavior: &lt;code&gt;add_header&lt;/code&gt; directives inside&amp;nbsp;a &lt;code&gt;location&lt;/code&gt; block &lt;strong&gt;completely override&lt;/strong&gt;&amp;nbsp;all &lt;code&gt;add_header&lt;/code&gt; directives from the&amp;nbsp;parent &lt;code&gt;server&lt;/code&gt; block. If you&amp;nbsp;set &lt;code&gt;Alt-Svc&lt;/code&gt; at the server level but have&amp;nbsp;any &lt;code&gt;location&lt;/code&gt; block with its&amp;nbsp;own &lt;code&gt;add_header&lt;/code&gt; (for &lt;code&gt;Cache-Control&lt;/code&gt;, &lt;code&gt;Access-Control-Allow-Origin&lt;/code&gt;, etc.),&amp;nbsp;the &lt;code&gt;Alt-Svc&lt;/code&gt; header silently disappears for those&amp;nbsp;locations.&lt;/p&gt;
&lt;p&gt;The fix:&amp;nbsp;add &lt;code&gt;Alt-Svc&lt;/code&gt; to&amp;nbsp;every &lt;code&gt;location&lt;/code&gt; block that already has&amp;nbsp;any &lt;code&gt;add_header&lt;/code&gt; directive.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/assets&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Cache-Control&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;public&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;Access-Control-Allow-Origin&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;*&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;X-Cache-Status&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$upstream_cache_status&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Alt-Svc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;h3=&amp;quot;:443&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;ma=86400&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# Must be here too!&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Verify coverage with&amp;nbsp;curl:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check the root path&lt;/span&gt;
curl&lt;span class="w"&gt; &lt;/span&gt;-sI&lt;span class="w"&gt; &lt;/span&gt;https://yourdomain.com/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;alt-svc

&lt;span class="c1"&gt;# Check a subpath - if this is empty, you have the inheritance problem&lt;/span&gt;
curl&lt;span class="w"&gt; &lt;/span&gt;-sI&lt;span class="w"&gt; &lt;/span&gt;https://yourdomain.com/some/path&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;alt-svc
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="step-3-the-ssl-library-problem"&gt;Step 3: The &lt;span class="caps"&gt;SSL&lt;/span&gt; Library&amp;nbsp;Problem&lt;/h2&gt;
&lt;p&gt;With a correct config and firewall in place, browsers may still refuse &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3. Capturing &lt;span class="caps"&gt;UDP&lt;/span&gt; traffic on port 443 inside the nginx jail reveals the&amp;nbsp;problem:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;tcpdump&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;vnet0&lt;span class="w"&gt; &lt;/span&gt;udp&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;20&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With stock OpenSSL, the server responds with tiny 51-byte packets to every &lt;span class="caps"&gt;QUIC&lt;/span&gt; connection attempt. These are &lt;span class="caps"&gt;QUIC&lt;/span&gt; version negotiation failures - the handshake is breaking at the &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 framing layer, not the &lt;span class="caps"&gt;TLS&lt;/span&gt;&amp;nbsp;layer.&lt;/p&gt;
&lt;h3 id="the-misleading-openssl-s_client-test"&gt;The&amp;nbsp;Misleading &lt;code&gt;openssl s_client&lt;/code&gt; Test&lt;/h3&gt;
&lt;p&gt;A common false positive when diagnosing&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;openssl&lt;span class="w"&gt; &lt;/span&gt;s_client&lt;span class="w"&gt; &lt;/span&gt;-connect&lt;span class="w"&gt; &lt;/span&gt;yourdomain.com:443&lt;span class="w"&gt; &lt;/span&gt;-quic&lt;span class="w"&gt; &lt;/span&gt;-alpn&lt;span class="w"&gt; &lt;/span&gt;h3
&lt;span class="c1"&gt;# Shows: CONNECTED&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This only validates &lt;span class="caps"&gt;TLS&lt;/span&gt;-level &lt;span class="caps"&gt;QUIC&lt;/span&gt; support. The actual &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 protocol layer on top can still be broken - and with stock OpenSSL on FreeBSD, it&amp;nbsp;is.&lt;/p&gt;
&lt;h3 id="why-stock-openssl-doesnt-work"&gt;Why Stock OpenSSL Doesn&amp;#8217;t&amp;nbsp;Work&lt;/h3&gt;
&lt;p&gt;nginx&amp;#8217;s &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 implementation was originally built against BoringSSL. While OpenSSL 3.2+ added &lt;span class="caps"&gt;QUIC&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt; support, there are known compatibility issues at the &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 framing layer. The &lt;span class="caps"&gt;TLS&lt;/span&gt; handshake may succeed, but actual &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 data transfer&amp;nbsp;fails.&lt;/p&gt;
&lt;h3 id="rebuilding-nginx-with-boringssl"&gt;Rebuilding nginx with&amp;nbsp;BoringSSL&lt;/h3&gt;
&lt;p&gt;The FreeBSD nginx port offers several &lt;span class="caps"&gt;SSL&lt;/span&gt; backends for &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3. Check the available&amp;nbsp;options:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;make&lt;span class="w"&gt; &lt;/span&gt;-C&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/www/nginx&lt;span class="w"&gt; &lt;/span&gt;showconfig&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;HTTPV3
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In nginx 1.28.2 the relevant options&amp;nbsp;are:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;HTTPV3=on: Enable HTTP/3 protocol support
HTTPV3_BORING=on: Use security/boringssl
HTTPV3_LSSL=off: Use security/libressl-devel
HTTPV3_QTLS=off: Use security/openssl33-quictls
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Enable &lt;code&gt;HTTPV3&lt;/code&gt; and &lt;code&gt;HTTPV3_BORING&lt;/code&gt;, then&amp;nbsp;rebuild:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/ports/www/nginx
make&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="c1"&gt;# Enable HTTPV3 and HTTPV3_BORING&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;unlock&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="c1"&gt;# If the package is locked&lt;/span&gt;
make&lt;span class="w"&gt; &lt;/span&gt;clean
make&lt;span class="w"&gt; &lt;/span&gt;deinstall
make&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;make&lt;span class="w"&gt; &lt;/span&gt;install
pkg&lt;span class="w"&gt; &lt;/span&gt;lock&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="c1"&gt;# Re-lock if desired&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;restart
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Verify the new binary links against&amp;nbsp;BoringSSL:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;-V&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;head&lt;span class="w"&gt; &lt;/span&gt;-3
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;nginx&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;nginx&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nt"&gt;1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;28&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;2&lt;/span&gt;
&lt;span class="nt"&gt;built&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;OpenSSL&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;1&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nt"&gt;compatible&lt;/span&gt;&lt;span class="o"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;BoringSSL&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nt"&gt;running&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;BoringSSL&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="nt"&gt;TLS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;SNI&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;support&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;enabled&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After rebuilding, tcpdump shows proper-sized &lt;span class="caps"&gt;QUIC&lt;/span&gt; handshake packets (1200, 892, 223 bytes) instead of the 51-byte&amp;nbsp;rejections.&lt;/p&gt;
&lt;h2 id="step-4-the-worker-process-affinity-problem"&gt;Step 4: The Worker Process Affinity&amp;nbsp;Problem&lt;/h2&gt;
&lt;p&gt;With BoringSSL in place, the first &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 request succeeds - but subsequent requests on the same connection fail&amp;nbsp;with &lt;code&gt;ERR_QUIC_PROTOCOL_ERROR_QUIC_PUBLIC_RESET&lt;/code&gt; in&amp;nbsp;Chrome.&lt;/p&gt;
&lt;p&gt;Setting &lt;code&gt;worker_processes 1&lt;/code&gt; in &lt;code&gt;nginx.conf&lt;/code&gt; makes &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 work perfectly. This confirms the diagnosis: &lt;span class="caps"&gt;QUIC&lt;/span&gt; packets from the same connection are being routed to different worker processes. A worker that didn&amp;#8217;t establish the connection responds with a &amp;#8220;public reset&amp;#8221; packet, killing&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The contrast between the broken and fixed&amp;nbsp;behaviour:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;WITHOUT reuseport - one shared UDP socket, OS picks any worker per packet:

  pkt 1 (conn A) ──┐
  pkt 2 (conn A) ──┼──&amp;gt; [ :443 UDP ] ──&amp;gt; OS ──&amp;gt; Worker 1  ✓ (owns conn A)
  pkt 3 (conn A) ──┘                       └──&amp;gt; Worker 2  ✗ (no context)
                                                     │
                                               sends RST → connection killed


WITH reuseport + quic_retry - one socket per worker, OS hashes by connection:

  pkt 1 (conn A) ──┐                    ┌──&amp;gt; Worker 1 socket ──&amp;gt; Worker 1  ✓
  pkt 2 (conn A) ──┼──&amp;gt; SO_REUSEPORT_LB ┤
  pkt 3 (conn A) ──┘  (hashes 4-tuple)  ├──&amp;gt; Worker 2 socket ──&amp;gt; Worker 2
                                         └──&amp;gt; Worker 3 socket ──&amp;gt; Worker 3

  All packets for the same connection always reach the same worker.
  quic_retry ensures the correct worker is pinned from the very first packet.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="the-linux-solution-not-available-on-freebsd"&gt;The Linux Solution (Not Available on&amp;nbsp;FreeBSD)&lt;/h3&gt;
&lt;p&gt;On Linux, nginx solves this&amp;nbsp;with &lt;code&gt;quic_bpf on&lt;/code&gt;, which uses eBPF to route &lt;span class="caps"&gt;QUIC&lt;/span&gt; packets to the correct worker based on the connection &lt;span class="caps"&gt;ID&lt;/span&gt;. FreeBSD doesn&amp;#8217;t have eBPF - this option doesn&amp;#8217;t&amp;nbsp;exist.&lt;/p&gt;
&lt;h3 id="the-freebsd-solution-reuseport-quic_retry"&gt;The FreeBSD&amp;nbsp;Solution: &lt;code&gt;reuseport&lt;/code&gt; + &lt;code&gt;quic_retry&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;Two directives work together to solve this on&amp;nbsp;FreeBSD:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;reuseport&lt;/code&gt;&lt;/strong&gt; on the &lt;span class="caps"&gt;QUIC&lt;/span&gt; listeners creates per-worker listening sockets. FreeBSD 12+&amp;nbsp;supports &lt;code&gt;SO_REUSEPORT_LB&lt;/code&gt;, which load-balances incoming &lt;span class="caps"&gt;UDP&lt;/span&gt; packets across workers while keeping each connection pinned to one worker socket. Add it to &lt;strong&gt;one&lt;/strong&gt; server block only - typically the primary vhost - to avoid duplicate binding&amp;nbsp;errors:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Primary server block only:&lt;/span&gt;
&lt;span class="k"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;reuseport&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;reuseport&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="c1"&gt;# All other server blocks:&lt;/span&gt;
&lt;span class="k"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;quic_retry on&lt;/code&gt;&lt;/strong&gt; forces a &lt;span class="caps"&gt;QUIC&lt;/span&gt; retry handshake (address validation) before establishing the connection. This extra round-trip ensures the correct worker process handles the entire &lt;span class="caps"&gt;QUIC&lt;/span&gt; session. Add it to&amp;nbsp;the &lt;code&gt;http&lt;/code&gt; block&amp;nbsp;in &lt;code&gt;nginx.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;http&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;quic_retry&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# ...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Verify &lt;code&gt;reuseport&lt;/code&gt; is active by checking for multiple &lt;span class="caps"&gt;UDP&lt;/span&gt;&amp;nbsp;sockets:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sockstat&lt;span class="w"&gt; &lt;/span&gt;-4&lt;span class="w"&gt; &lt;/span&gt;-6&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;443&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You should see multiple &lt;span class="caps"&gt;UDP&lt;/span&gt; entries on port 443 - one per worker&amp;nbsp;process.&lt;/p&gt;
&lt;h2 id="step-5-firewall-configuration"&gt;Step 5: Firewall&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;QUIC&lt;/span&gt; runs over &lt;span class="caps"&gt;UDP&lt;/span&gt;, so you need explicit &lt;span class="caps"&gt;UDP&lt;/span&gt; rules for port 443 alongside the existing &lt;span class="caps"&gt;TCP&lt;/span&gt; rules. The exact configuration depends on your network&amp;nbsp;topology.&lt;/p&gt;
&lt;p&gt;In this setup, nginx runs inside a Bastille jail. IPv6 is directly routed - the jail holds a public IPv6 address, so no &lt;span class="caps"&gt;RDR&lt;/span&gt; or &lt;span class="caps"&gt;NAT&lt;/span&gt; is needed for IPv6 &lt;span class="caps"&gt;QUIC&lt;/span&gt;. For IPv4, the host&amp;#8217;s pf forwards traffic from the public &lt;span class="caps"&gt;IP&lt;/span&gt; to the jail&amp;#8217;s private address&amp;nbsp;using &lt;code&gt;rdr&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The important detail: &lt;span class="caps"&gt;QUIC&lt;/span&gt; (&lt;span class="caps"&gt;UDP&lt;/span&gt;) needs its&amp;nbsp;own &lt;code&gt;rdr&lt;/code&gt; rule on the host, separate from the &lt;span class="caps"&gt;TCP&lt;/span&gt; one.&amp;nbsp;A &lt;code&gt;pass&lt;/code&gt; rule alone is not enough -&amp;nbsp;without &lt;code&gt;rdr&lt;/code&gt;, &lt;span class="caps"&gt;UDP&lt;/span&gt; packets arriving at the public &lt;span class="caps"&gt;IP&lt;/span&gt; are still addressed to the public &lt;span class="caps"&gt;IP&lt;/span&gt; when the filter rules run, so&amp;nbsp;a &lt;code&gt;pass ... to $frontend_v4&lt;/code&gt; (jail &lt;span class="caps"&gt;IP&lt;/span&gt;) rule will never match&amp;nbsp;them.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;# --- RDR ---&lt;/span&gt;
&lt;span class="cp"&gt;# Redirect TCP 80/443 to the nginx jail&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v4&lt;/span&gt;

&lt;span class="cp"&gt;# Redirect UDP 443 (QUIC) to the nginx jail - separate rule required&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v4&lt;/span&gt;

&lt;span class="cp"&gt;# --- Pass rules ---&lt;/span&gt;
&lt;span class="cp"&gt;# HTTP/HTTPS (TCP)&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="cp"&gt;# QUIC (UDP) - IPv4 via NAT/rdr above, IPv6 direct&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Don&amp;#8217;t forget both IPv4 and IPv6 if your server is dual-stack. For a server without a Bastille jail - nginx running directly on the host -&amp;nbsp;the &lt;code&gt;rdr&lt;/code&gt; rules are not needed; only&amp;nbsp;the &lt;code&gt;pass&lt;/code&gt; rules for both &lt;span class="caps"&gt;TCP&lt;/span&gt; and &lt;span class="caps"&gt;UDP&lt;/span&gt;.&lt;/p&gt;
&lt;h2 id="step-6-performance-tuning"&gt;Step 6: Performance&amp;nbsp;Tuning&lt;/h2&gt;
&lt;h3 id="tls-13-early-data-0-rtt"&gt;&lt;span class="caps"&gt;TLS&lt;/span&gt; 1.3 Early Data (0-&lt;span class="caps"&gt;RTT&lt;/span&gt;)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;ssl_early_data&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Speeds up reconnections by allowing data in the first &lt;span class="caps"&gt;TLS&lt;/span&gt; flight. Early data is vulnerable to replay attacks, so pass the status to your&amp;nbsp;backend:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Early-Data&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$ssl_early_data&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Add this to&amp;nbsp;all &lt;code&gt;proxy_pass&lt;/code&gt; locations so your application can reject replayed requests on sensitive endpoints (&lt;span class="caps"&gt;POST&lt;/span&gt;, &lt;span class="caps"&gt;DELETE&lt;/span&gt;,&amp;nbsp;etc.).&lt;/p&gt;
&lt;h3 id="session-tickets"&gt;Session&amp;nbsp;Tickets&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;ssl_session_tickets&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Required for &lt;span class="caps"&gt;QUIC&lt;/span&gt; 0-&lt;span class="caps"&gt;RTT&lt;/span&gt; resumption. Unlike the common advice to disable session tickets for &lt;span class="caps"&gt;TLS&lt;/span&gt; 1.2 security, they&amp;#8217;re important for &lt;span class="caps"&gt;QUIC&lt;/span&gt;&amp;nbsp;performance.&lt;/p&gt;
&lt;h2 id="complete-example-configuration"&gt;Complete Example&amp;nbsp;Configuration&lt;/h2&gt;
&lt;h3 id="nginxconf"&gt;nginx.conf&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;worker_processes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;auto&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;error_log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/var/log/nginx/error.log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="k"&gt;events&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;worker_connections&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1024&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;http&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;include&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="s"&gt;mime.types&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;default_type&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;application/octet-stream&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;sendfile&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;tcp_nopush&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;tcp_nodelay&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;keepalive_timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# QUIC settings&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;quic_retry&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_early_data&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# SSL global settings&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_protocols&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;TLSv1.2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;TLSv1.3&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_prefer_server_ciphers&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_session_cache&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;shared:SSL:10m&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_session_tickets&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# gzip compression&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;gzip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;gzip_vary&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;gzip_proxied&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;any&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;gzip_comp_level&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;gzip_types&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;text/plain&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;text/css&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;application/json&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;application/javascript&lt;/span&gt;
&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="s"&gt;text/xml&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;application/xml&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;application/xml+rss&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;text/javascript&lt;/span&gt;
&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="s"&gt;image/svg+xml&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;image/x-icon&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/usr/local/etc/nginx/vhosts/*.conf&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="primary-vhost-with-reuseport"&gt;Primary vhost&amp;nbsp;(with &lt;code&gt;reuseport&lt;/code&gt;)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;reuseport&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;reuseport&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;http2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yourdomain.com&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_certificate&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="s"&gt;/path/to/fullchain.pem&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_certificate_key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/path/to/privkey.pem&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Alt-Svc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;h3=&amp;quot;:443&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;ma=86400&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Alt-Svc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;h3=&amp;quot;:443&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;ma=86400&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# ... your config ...&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;@proxy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Alt-Svc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;h3=&amp;quot;:443&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;ma=86400&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Early-Data&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$ssl_early_data&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# ... your proxy config ...&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="additional-vhosts-without-reuseport"&gt;Additional vhosts&amp;nbsp;(without &lt;code&gt;reuseport&lt;/code&gt;)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;[::]:443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quic&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;http2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="no"&gt;on&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;other.yourdomain.com&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_certificate&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="s"&gt;/path/to/fullchain.pem&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;ssl_certificate_key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/path/to/privkey.pem&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Alt-Svc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;h3=&amp;quot;:443&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;ma=86400&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;add_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Alt-Svc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;h3=&amp;quot;:443&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;ma=86400&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;# ... your config ...&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="testing-and-verification"&gt;Testing and&amp;nbsp;Verification&lt;/h2&gt;
&lt;h3 id="server-side"&gt;Server-side&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Verify QUIC TLS handshake (note: only tests TLS layer, not full HTTP/3 framing)&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;openssl&lt;span class="w"&gt; &lt;/span&gt;s_client&lt;span class="w"&gt; &lt;/span&gt;-connect&lt;span class="w"&gt; &lt;/span&gt;yourdomain.com:443&lt;span class="w"&gt; &lt;/span&gt;-quic&lt;span class="w"&gt; &lt;/span&gt;-alpn&lt;span class="w"&gt; &lt;/span&gt;h3

&lt;span class="c1"&gt;# Check Alt-Svc is present on all paths&lt;/span&gt;
curl&lt;span class="w"&gt; &lt;/span&gt;-sI&lt;span class="w"&gt; &lt;/span&gt;https://yourdomain.com/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;alt-svc
curl&lt;span class="w"&gt; &lt;/span&gt;-sI&lt;span class="w"&gt; &lt;/span&gt;https://yourdomain.com/some/path&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;alt-svc

&lt;span class="c1"&gt;# Monitor live QUIC traffic (inside the nginx jail)&lt;/span&gt;
tcpdump&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;vnet0&lt;span class="w"&gt; &lt;/span&gt;udp&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;20&lt;/span&gt;

&lt;span class="c1"&gt;# Verify reuseport sockets - expect multiple UDP entries on 443&lt;/span&gt;
sockstat&lt;span class="w"&gt; &lt;/span&gt;-4&lt;span class="w"&gt; &lt;/span&gt;-6&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;443&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="client-side"&gt;Client-side&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Online checker:&lt;/strong&gt; &lt;a href="https://http3check.net"&gt;http3check.net&lt;/a&gt; - verifies both Alt-Svc and &lt;span class="caps"&gt;QUIC&lt;/span&gt;&amp;nbsp;connectivity&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Browser DevTools:&lt;/strong&gt; Network tab → right-click column headers → enable &amp;#8220;Protocol&amp;#8221;. Look&amp;nbsp;for &lt;code&gt;h3&lt;/code&gt; (may require a second page load after Alt-Svc is&amp;nbsp;cached)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Chrome &lt;span class="caps"&gt;QUIC&lt;/span&gt;&amp;nbsp;internals:&lt;/strong&gt; &lt;code&gt;chrome://net-export/&lt;/code&gt; captures a netlog, analyzable at &lt;a href="https://netlog-viewer.appspot.com"&gt;netlog-viewer.appspot.com&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="debugging-tips"&gt;Debugging&amp;nbsp;Tips&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;If Chrome marks &lt;span class="caps"&gt;QUIC&lt;/span&gt;&amp;nbsp;as &lt;code&gt;is_broken: true&lt;/code&gt;, clear cached data via Settings → Clear browsing data → Cached images and&amp;nbsp;files&lt;/li&gt;
&lt;li&gt;Use an incognito window for clean tests without cached Alt-Svc&amp;nbsp;state&lt;/li&gt;
&lt;li&gt;Check nginx error&amp;nbsp;logs: &lt;code&gt;grep -i quic /var/log/nginx/error.log&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;For deep debugging, temporarily&amp;nbsp;set &lt;code&gt;error_log /var/log/nginx/quic-debug.log debug;&lt;/code&gt; - remove it again promptly, it fills up&amp;nbsp;fast&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="summary-of-pitfalls"&gt;Summary of&amp;nbsp;Pitfalls&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Problem&lt;/th&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Solution&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Missing &lt;code&gt;listen 443 ssl&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Site completely down&lt;/td&gt;
&lt;td&gt;Add&amp;nbsp;both &lt;code&gt;ssl&lt;/code&gt; and &lt;code&gt;quic&lt;/code&gt; listeners&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Missing &lt;code&gt;Alt-Svc&lt;/code&gt; header&lt;/td&gt;
&lt;td&gt;Browsers never try &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3&lt;/td&gt;
&lt;td&gt;Add &lt;code&gt;Alt-Svc&lt;/code&gt; with &lt;code&gt;always&lt;/code&gt; flag&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;add_header&lt;/code&gt; inheritance&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Alt-Svc&lt;/code&gt; missing on some paths&lt;/td&gt;
&lt;td&gt;Repeat &lt;code&gt;Alt-Svc&lt;/code&gt; in&amp;nbsp;every &lt;code&gt;location&lt;/code&gt; with &lt;code&gt;add_header&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stock OpenSSL&lt;/td&gt;
&lt;td&gt;51-byte &lt;span class="caps"&gt;QUIC&lt;/span&gt;&amp;nbsp;rejections, &lt;code&gt;openssl s_client&lt;/code&gt; lies&lt;/td&gt;
&lt;td&gt;Rebuild nginx&amp;nbsp;with &lt;code&gt;HTTPV3_BORING=on&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Worker affinity&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ERR_QUIC_PROTOCOL_ERROR_QUIC_PUBLIC_RESET&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Use &lt;code&gt;reuseport&lt;/code&gt; + &lt;code&gt;quic_retry on&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Missing &lt;span class="caps"&gt;UDP&lt;/span&gt; &lt;code&gt;rdr&lt;/code&gt; in pf&lt;/td&gt;
&lt;td&gt;IPv4 &lt;span class="caps"&gt;QUIC&lt;/span&gt; silently unreachable despite pass rules&lt;/td&gt;
&lt;td&gt;Add &lt;code&gt;rdr&lt;/code&gt; for &lt;span class="caps"&gt;UDP&lt;/span&gt; 443 alongside the &lt;span class="caps"&gt;TCP&lt;/span&gt; one&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Firewall blocks &lt;span class="caps"&gt;UDP&lt;/span&gt; 443&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;QUIC&lt;/span&gt; unreachable from clients&lt;/td&gt;
&lt;td&gt;Add &lt;span class="caps"&gt;UDP&lt;/span&gt;&amp;nbsp;443 &lt;code&gt;pass&lt;/code&gt; rules for IPv4 and IPv6&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Getting &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 working on FreeBSD with nginx requires more legwork than on Linux, mostly because of the &lt;span class="caps"&gt;SSL&lt;/span&gt; library situation and the absence&amp;nbsp;of &lt;code&gt;quic_bpf&lt;/code&gt;. The four things that actually&amp;nbsp;matter:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;BoringSSL&lt;/strong&gt; - rebuild from ports&amp;nbsp;with &lt;code&gt;HTTPV3=on&lt;/code&gt; and &lt;code&gt;HTTPV3_BORING=on&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;quic_retry on&lt;/code&gt;&lt;/strong&gt; - ensures correct worker routing during session&amp;nbsp;establishment&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;reuseport&lt;/code&gt;&lt;/strong&gt; on &lt;span class="caps"&gt;QUIC&lt;/span&gt; listeners - per-worker &lt;span class="caps"&gt;UDP&lt;/span&gt; sockets for reliable packet&amp;nbsp;distribution&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;Alt-Svc&lt;/code&gt; in every location block&lt;/strong&gt; - the&amp;nbsp;silent &lt;code&gt;add_header&lt;/code&gt; inheritance gotcha will bite&amp;nbsp;you&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;In a jail setup with IPv4 &lt;span class="caps"&gt;NAT&lt;/span&gt;, add a&amp;nbsp;dedicated &lt;code&gt;rdr&lt;/code&gt; rule for &lt;span class="caps"&gt;UDP&lt;/span&gt; 443 -&amp;nbsp;a &lt;code&gt;pass&lt;/code&gt; rule alone isn&amp;#8217;t&amp;nbsp;enough.&lt;/p&gt;
&lt;p&gt;Once all of this is in place, &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 works reliably and delivers noticeably lower latency, especially for connection-heavy workloads like a Mastodon&amp;nbsp;instance.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://nginx.org/en/docs/http/ngx_http_v3_module.html"&gt;nginx &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 module&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://boringssl.googlesource.com/boringssl/"&gt;BoringSSL&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://http3check.net"&gt;http3check.net&lt;/a&gt; - online &lt;span class="caps"&gt;HTTP&lt;/span&gt;/3 availability&amp;nbsp;checker&lt;/li&gt;
&lt;li&gt;&lt;a href="https://netlog-viewer.appspot.com"&gt;Chrome netlog&amp;nbsp;viewer&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook –&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="nginx"/><category term="http3"/><category term="quic"/><category term="networking"/><category term="jails"/><category term="bastille"/></entry><entry><title>Integrating LaCrosse Sensors into Home Assistant via JeeLink on FreeBSD</title><link href="https://hofstede.it/integrating-lacrosse-sensors-into-home-assistant-via-jeelink-on-freebsd/" rel="alternate"/><published>2026-02-19T00:00:00+01:00</published><updated>2026-02-19T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-02-19:/integrating-lacrosse-sensors-into-home-assistant-via-jeelink-on-freebsd/</id><summary type="html">&lt;p&gt;Using a JeeLink &lt;span class="caps"&gt;USB&lt;/span&gt; stick inside a FreeBSD jail to bridge legacy LaCrosse temperature and humidity sensors to Home Assistant over &lt;span class="caps"&gt;MQTT&lt;/span&gt;&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="LaCrosse Sensor" src="https://hofstede.it/images/2026-02-19-lacrosse-jeelink-homeassistant-freebsd.png" title="LaCrosse Temperature Sensor"&gt;&lt;/p&gt;
&lt;p&gt;Home Assistant runs great inside a bhyve &lt;span class="caps"&gt;VM&lt;/span&gt; on FreeBSD, but &lt;span class="caps"&gt;USB&lt;/span&gt; passthrough into bhyve is not straightforward. When I needed to integrate a ConBee &lt;span class="caps"&gt;II&lt;/span&gt; Zigbee dongle, the solution was to run zigbee2mqtt inside a FreeBSD jail - where &lt;span class="caps"&gt;USB&lt;/span&gt; devices can be exposed via devfs rules - and forward everything over &lt;span class="caps"&gt;MQTT&lt;/span&gt;. This same pattern turns out to work perfectly for other &lt;span class="caps"&gt;USB&lt;/span&gt; receivers, like the JeeLink with its legacy LaCrosse temperature and humidity&amp;nbsp;sensors.&lt;/p&gt;
&lt;p&gt;The JeeLink is a small &lt;span class="caps"&gt;USB&lt;/span&gt; stick with an &lt;span class="caps"&gt;RFM12B&lt;/span&gt; radio module running the LaCrosseITPlusReader sketch. It picks up transmissions from LaCrosse &lt;span class="caps"&gt;TX&lt;/span&gt;-series sensors on 868 MHz, decodes them, and outputs the raw data over a serial port. All we need is a small bridge script that reads the serial data and publishes it to &lt;span class="caps"&gt;MQTT&lt;/span&gt;.&lt;/p&gt;
&lt;h2 id="the-stack"&gt;The&amp;nbsp;Stack&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[ LaCrosse TX sensors ]
        |  868 MHz
        v
[ JeeLink USB (FT232R) ] --&amp;gt; /dev/cuaU1
        |
        v
[ FreeBSD Jail ]
  lacrosse2mqtt.py
        |  MQTT
        v
[ Mosquitto Broker ]
        |
        v
[ Home Assistant (bhyve VM) ]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The jail already exists for zigbee2mqtt, so adding a second serial device and a Python script is minimal additional&amp;nbsp;overhead.&lt;/p&gt;
&lt;h2 id="identifying-the-device"&gt;Identifying the&amp;nbsp;Device&lt;/h2&gt;
&lt;p&gt;FreeBSD recognizes the JeeLink&amp;#8217;s &lt;span class="caps"&gt;FTDI&lt;/span&gt; chip out of the&amp;nbsp;box:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;usbconfig&lt;span class="w"&gt; &lt;/span&gt;list
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ugen0.4: &amp;lt;FT232 Serial (UART) IC Future Technology Devices International, Ltd&amp;gt; at usbus0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The kernel attaches it via&amp;nbsp;the &lt;code&gt;uftdi&lt;/code&gt; driver, which maps to&amp;nbsp;a &lt;code&gt;/dev/cuaU*&lt;/code&gt; serial device. If you already have another &lt;span class="caps"&gt;USB&lt;/span&gt; serial device (like a Zigbee dongle&amp;nbsp;on &lt;code&gt;/dev/cuaU0&lt;/code&gt;), the JeeLink will appear&amp;nbsp;as &lt;code&gt;/dev/cuaU1&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="passing-the-device-into-the-jail"&gt;Passing the Device into the&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;In &lt;code&gt;/etc/devfs.rules&lt;/code&gt;, extend the jail&amp;#8217;s ruleset to include the new serial&amp;nbsp;port:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[devfsrules_jail_z2m=10]
add include $devfsrules_hide_all
add include $devfsrules_unhide_login
add path &amp;#39;cuaU0&amp;#39; unhide
add path &amp;#39;cuaU1&amp;#39; unhide
add path &amp;#39;ugen*&amp;#39; unhide
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The jail configuration&amp;nbsp;in &lt;code&gt;/etc/jail.conf&lt;/code&gt; references this&amp;nbsp;ruleset:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;z2m {
    # ... other settings ...
    devfs_ruleset = 10;
    enforce_statfs = 1;
    allow.mount.devfs;
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Restart the jail for the new device to appear inside&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="verifying-the-serial-connection"&gt;Verifying the Serial&amp;nbsp;Connection&lt;/h2&gt;
&lt;p&gt;Inside the jail,&amp;nbsp;use &lt;code&gt;cu&lt;/code&gt; to confirm the JeeLink is transmitting and the baud rate is correct. The LaCrosseITPlusReader sketch runs at 57600&amp;nbsp;baud:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cu&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;/dev/cuaU1&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;57600&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If everything is working, you&amp;#8217;ll see a header line followed by sensor&amp;nbsp;readings:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[LaCrosseITPlusReader.10.1s (RFM12B f:868300 r:17241)]
OK 9 8 1 4 22 106
OK 9 44 129 4 182 52
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Exit &lt;code&gt;cu&lt;/code&gt; with &lt;code&gt;~.&lt;/code&gt; and move on. If you see garbage, double-check the baud rate - this was by far the most common cause of confusion during my&amp;nbsp;setup.&lt;/p&gt;
&lt;h2 id="decoding-the-data-format"&gt;Decoding the Data&amp;nbsp;Format&lt;/h2&gt;
&lt;p&gt;Each &lt;code&gt;OK 9&lt;/code&gt; line from the JeeLink represents a sensor&amp;nbsp;reading:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;OK 9 &amp;lt;ID&amp;gt; &amp;lt;flags&amp;gt; &amp;lt;temp_high&amp;gt; &amp;lt;temp_low&amp;gt; &amp;lt;humidity&amp;gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Temperature&lt;/strong&gt; is a 16-bit value divided by 10, offset by&amp;nbsp;100:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;temp = (temp_high * 256 + temp_low) / 10.0 - 100.0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For example,&amp;nbsp;bytes &lt;code&gt;4 182&lt;/code&gt; decode&amp;nbsp;to &lt;code&gt;(4 * 256 + 182) / 10.0 - 100.0 = 20.6°C&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Humidity&lt;/strong&gt; is the raw byte value in percent. A value&amp;nbsp;of &lt;code&gt;52&lt;/code&gt; means 52% relative&amp;nbsp;humidity.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Flags&lt;/strong&gt; encode battery status: bit 7&amp;nbsp;(&lt;code&gt;0x80&lt;/code&gt;) indicates a new battery was just inserted, bit 0&amp;nbsp;(&lt;code&gt;0x01&lt;/code&gt;) signals a weak&amp;nbsp;battery.&lt;/p&gt;
&lt;h2 id="the-bridge-script"&gt;The Bridge&amp;nbsp;Script&lt;/h2&gt;
&lt;p&gt;Install the dependencies inside the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;py311-serial&lt;span class="w"&gt; &lt;/span&gt;py311-paho-mqtt
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then&amp;nbsp;create &lt;code&gt;/usr/local/bin/lacrosse2mqtt.py&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/usr/bin/env python3&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;serial&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;paho.mqtt.client&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="nn"&gt;mqtt&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;time&lt;/span&gt;
&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="nn"&gt;logging&lt;/span&gt;

&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;basicConfig&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;level&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;INFO&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;format&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="si"&gt;%(asctime)s&lt;/span&gt;&lt;span class="s1"&gt; &lt;/span&gt;&lt;span class="si"&gt;%(message)s&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;SERIAL_PORT&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;/dev/cuaU1&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;BAUD&lt;/span&gt;        &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;57600&lt;/span&gt;
&lt;span class="n"&gt;MQTT_HOST&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;127.0.0.1&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;MQTT_PORT&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;1883&lt;/span&gt;
&lt;span class="n"&gt;MQTT_USER&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;your_mqtt_user&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;MQTT_PASS&lt;/span&gt;   &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;your_mqtt_password&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;SENSOR_IDS&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;44&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;               &lt;span class="c1"&gt;# whitelist: only your own sensor IDs&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;mqtt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Client&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;mqtt&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;CallbackAPIVersion&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;VERSION2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;username_pw_set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MQTT_USER&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;MQTT_PASS&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;connect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MQTT_HOST&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;MQTT_PORT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;loop_start&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;ser&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;serial&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;Serial&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SERIAL_PORT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;BAUD&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Connected to JeeLink on &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;SERIAL_PORT&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;while&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="k"&gt;try&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;ser&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;readline&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;decode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;utf-8&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;errors&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ignore&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;strip&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;OK 9&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;

        &lt;span class="n"&gt;parts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;split&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&lt;/span&gt; &lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;

        &lt;span class="n"&gt;sensor_id&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;

        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;sensor_id&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;SENSOR_IDS&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;continue&lt;/span&gt;

        &lt;span class="n"&gt;flags&lt;/span&gt;     &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="n"&gt;temp&lt;/span&gt;      &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt; &lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="mi"&gt;256&lt;/span&gt; &lt;span class="o"&gt;+&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;]))&lt;/span&gt; &lt;span class="o"&gt;/&lt;/span&gt; &lt;span class="mf"&gt;10.0&lt;/span&gt; &lt;span class="o"&gt;-&lt;/span&gt; &lt;span class="mf"&gt;100.0&lt;/span&gt;
        &lt;span class="n"&gt;humidity&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;parts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
        &lt;span class="n"&gt;new_batt&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="mh"&gt;0x80&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;weak_batt&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nb"&gt;bool&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt; &lt;span class="o"&gt;&amp;amp;&lt;/span&gt; &lt;span class="mh"&gt;0x01&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;info&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Sensor &lt;/span&gt;&lt;span class="si"&gt;%d&lt;/span&gt;&lt;span class="s2"&gt;: &lt;/span&gt;&lt;span class="si"&gt;%.1f&lt;/span&gt;&lt;span class="s2"&gt;°C, &lt;/span&gt;&lt;span class="si"&gt;%d%%&lt;/span&gt;&lt;span class="s2"&gt; rH (new_batt=&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt; weak_batt=&lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;)&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                     &lt;span class="n"&gt;sensor_id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;temp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;humidity&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;new_batt&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;weak_batt&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

        &lt;span class="n"&gt;topic&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;lacrosse/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;sensor_id&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
        &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;publish&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;topic&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/temperature&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;round&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;temp&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;retain&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;publish&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;topic&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/humidity&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;humidity&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;retain&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;publish&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;topic&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/battery_low&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="nb"&gt;int&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;weak_batt&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt; &lt;span class="n"&gt;retain&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

    &lt;span class="k"&gt;except&lt;/span&gt; &lt;span class="ne"&gt;Exception&lt;/span&gt; &lt;span class="k"&gt;as&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="n"&gt;logging&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;error&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Error: &lt;/span&gt;&lt;span class="si"&gt;%s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;e&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;time&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sleep&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;SENSOR_IDS&lt;/code&gt; whitelist is important. The 868 MHz band is shared, and you will pick up sensors from neighboring households. Temporarily remove the filter to discover your sensor&amp;#8217;s &lt;span class="caps"&gt;ID&lt;/span&gt; - cross-reference it against the physical display or move the sensor around to identify which readings are&amp;nbsp;yours.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;retain=True&lt;/code&gt; flag on every publish call is equally important. Without it, Home Assistant will show the entity as unavailable after a restart, because the broker has no stored value to replay to new&amp;nbsp;subscribers.&lt;/p&gt;
&lt;h2 id="running-as-a-service"&gt;Running as a&amp;nbsp;Service&lt;/h2&gt;
&lt;p&gt;Create an rc script&amp;nbsp;at &lt;code&gt;/usr/local/etc/rc.d/lacrosse2mqtt&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# PROVIDE: lacrosse2mqtt&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: NETWORKING&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;

.&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.subr

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;lacrosse2mqtt&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rcvar&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;lacrosse2mqtt_enable&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/local/bin/python3&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;command_args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/local/bin/lacrosse2mqtt.py&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;logfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/log/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.log&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;start_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_start&amp;quot;&lt;/span&gt;

lacrosse2mqtt_start&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Starting &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;...&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/usr/sbin/daemon&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;logfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;command&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;command_args&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;lacrosse2mqtt_enable&lt;/span&gt;&lt;span class="p"&gt;:=NO&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;

run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Enable and start&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;+x&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/rc.d/lacrosse2mqtt
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;lacrosse2mqtt_enable=&amp;quot;YES&amp;quot;&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.conf
service&lt;span class="w"&gt; &lt;/span&gt;lacrosse2mqtt&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Verify&amp;nbsp;with &lt;code&gt;service lacrosse2mqtt status&lt;/code&gt; and &lt;code&gt;tail -f /var/log/lacrosse2mqtt.log&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="verifying-mqtt"&gt;Verifying &lt;span class="caps"&gt;MQTT&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Before touching Home Assistant, confirm messages are arriving at the&amp;nbsp;broker:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mosquitto_sub&lt;span class="w"&gt; &lt;/span&gt;-h&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;127&lt;/span&gt;.0.0.1&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;your_mqtt_user&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;your_mqtt_password&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;lacrosse/#&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-v
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;lacrosse/44/temperature 20.7
lacrosse/44/humidity 51
lacrosse/44/battery_low 0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If nothing shows up, check the log file and make sure the &lt;span class="caps"&gt;MQTT&lt;/span&gt; credentials are&amp;nbsp;correct.&lt;/p&gt;
&lt;h2 id="home-assistant-configuration"&gt;Home Assistant&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Add the sensors&amp;nbsp;to &lt;code&gt;configuration.yaml&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;mqtt&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;sensor&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Living&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Room&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Temperature&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;unique_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;lacrosse_44_temperature&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;state_topic&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;lacrosse/44/temperature&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;unit_of_measurement&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;°C&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;device_class&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;temperature&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;state_class&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;measurement&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Living&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Room&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Humidity&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;unique_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;lacrosse_44_humidity&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;state_topic&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;lacrosse/44/humidity&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;unit_of_measurement&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;%&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;device_class&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;humidity&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;state_class&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;measurement&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Living&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Room&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Sensor&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;Battery&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;unique_id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;lacrosse_44_battery&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;state_topic&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;lacrosse/44/battery_low&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;device_class&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;battery&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Check the configuration under &lt;strong&gt;Developer Tools&lt;/strong&gt;, restart Home Assistant, and the entities will appear under &lt;strong&gt;Settings &amp;gt; Devices &lt;span class="amp"&gt;&amp;amp;&lt;/span&gt; Services &amp;gt; Entities&lt;/strong&gt;. You can also verify &lt;span class="caps"&gt;MQTT&lt;/span&gt; messages are arriving inside &lt;span class="caps"&gt;HA&lt;/span&gt; at &lt;strong&gt;Settings &amp;gt; Devices &lt;span class="amp"&gt;&amp;amp;&lt;/span&gt; Services &amp;gt; &lt;span class="caps"&gt;MQTT&lt;/span&gt; &amp;gt; Configure&lt;/strong&gt; using the &amp;#8220;Listen to a topic&amp;#8221; field&amp;nbsp;with &lt;code&gt;lacrosse/#&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="adding-more-sensors"&gt;Adding More&amp;nbsp;Sensors&lt;/h2&gt;
&lt;p&gt;When you add a new LaCrosse&amp;nbsp;sensor:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Insert fresh batteries and let it transmit for a&amp;nbsp;minute&lt;/li&gt;
&lt;li&gt;Watch the log&amp;nbsp;(&lt;code&gt;tail -f /var/log/lacrosse2mqtt.log&lt;/code&gt;) for the new sensor &lt;span class="caps"&gt;ID&lt;/span&gt;&lt;/li&gt;
&lt;li&gt;Add the &lt;span class="caps"&gt;ID&lt;/span&gt;&amp;nbsp;to &lt;code&gt;SENSOR_IDS&lt;/code&gt; in the Python&amp;nbsp;script&lt;/li&gt;
&lt;li&gt;Restart the service&amp;nbsp;(&lt;code&gt;service lacrosse2mqtt restart&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;Add the corresponding &lt;span class="caps"&gt;MQTT&lt;/span&gt; sensors&amp;nbsp;to &lt;code&gt;configuration.yaml&lt;/code&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="troubleshooting"&gt;Troubleshooting&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Garbage output from the serial port:&lt;/strong&gt; Wrong baud rate. The JeeLink with LaCrosseITPlusReader uses 57600. Verify interactively&amp;nbsp;with &lt;code&gt;cu&lt;/code&gt; before anything&amp;nbsp;else.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Sensor IDs from neighbors:&lt;/strong&gt; Expected on the shared 868 MHz band. Use&amp;nbsp;the &lt;code&gt;SENSOR_IDS&lt;/code&gt; whitelist.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;new_batt=True&lt;/code&gt; after battery change:&lt;/strong&gt; Normal. The sensor sets this flag temporarily after a battery insertion. It clears on its&amp;nbsp;own.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Entity shows &amp;#8220;—&amp;#8221; in Home Assistant:&lt;/strong&gt; &lt;span class="caps"&gt;MQTT&lt;/span&gt; messages are not being retained. Check&amp;nbsp;that &lt;code&gt;retain=True&lt;/code&gt; is set in&amp;nbsp;all &lt;code&gt;client.publish()&lt;/code&gt; calls.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;MQTT&lt;/span&gt; connection refused:&lt;/strong&gt; Your broker likely requires authentication. Add credentials&amp;nbsp;via &lt;code&gt;client.username_pw_set()&lt;/code&gt; in the&amp;nbsp;script.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Serial device&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/dev/cuaU1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Baud rate&lt;/td&gt;
&lt;td&gt;57600&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;JeeLink sketch&lt;/td&gt;
&lt;td&gt;LaCrosseITPlusReader 10.1s&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;MQTT&lt;/span&gt; topic structure&lt;/td&gt;
&lt;td&gt;&lt;code&gt;lacrosse/&amp;lt;sensor_id&amp;gt;/temperature&lt;/code&gt; etc.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Temperature formula&lt;/td&gt;
&lt;td&gt;&lt;code&gt;(high * 256 + low) / 10.0 - 100.0&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Humidity&lt;/td&gt;
&lt;td&gt;Direct byte value&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The pattern here - &lt;span class="caps"&gt;USB&lt;/span&gt; device in a FreeBSD jail, data bridged over &lt;span class="caps"&gt;MQTT&lt;/span&gt; - generalizes well beyond LaCrosse sensors. Any &lt;span class="caps"&gt;USB&lt;/span&gt;-based &lt;span class="caps"&gt;RF&lt;/span&gt; receiver that can&amp;#8217;t be passed directly into a bhyve &lt;span class="caps"&gt;VM&lt;/span&gt; can be handled this way. If you&amp;#8217;re already running zigbee2mqtt in a jail, adding another serial device is just a few lines in your devfs rules and a small Python script&amp;nbsp;away.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/rinie/LaCrosseITPlusReader"&gt;LaCrosseITPlusReader&amp;nbsp;firmware&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://jeelabs.net/projects/hardware/wiki/JeeLink"&gt;JeeLink&amp;nbsp;hardware&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.home-assistant.io/integrations/mqtt/"&gt;Home Assistant &lt;span class="caps"&gt;MQTT&lt;/span&gt;&amp;nbsp;integration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook:&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://eclipse.dev/paho/files/paho.mqtt.python/html/client.html"&gt;paho-mqtt Python&amp;nbsp;client&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="jails"/><category term="home-assistant"/><category term="mqtt"/><category term="iot"/><category term="sensors"/></entry><entry><title>PF Firewall on FreeBSD: A Practical Guide</title><link href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/" rel="alternate"/><published>2026-02-06T00:00:00+01:00</published><updated>2026-02-06T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-02-06:/pf-firewall-on-freebsd-a-practical-guide/</id><summary type="html">&lt;p&gt;A comprehensive guide to configuring &lt;span class="caps"&gt;PF&lt;/span&gt; on FreeBSD, covering core concepts, practical configurations for jails and dual-stack networking, and advanced techniques including brute-force protection and bastion host setups with&amp;nbsp;authpf.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Logo" src="https://hofstede.it/images/2026-02-06-pf-firewall-freebsd-guide.png" title="PF Firewall Tutorial: Header image"&gt;&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; (Packet Filter) is one of the most elegant firewall systems available on any operating system. Originally developed for OpenBSD and ported to FreeBSD, it combines a clean configuration syntax with powerful capabilities for filtering, &lt;span class="caps"&gt;NAT&lt;/span&gt;, traffic shaping, and logging. After running &lt;span class="caps"&gt;PF&lt;/span&gt; across multiple FreeBSD servers for years, I&amp;#8217;ve developed a consistent configuration pattern that balances security with&amp;nbsp;practicality.&lt;/p&gt;
&lt;p&gt;This guide covers everything from basic concepts to production configurations, with tips and patterns I&amp;#8217;ve refined through real-world deployment. Whether you&amp;#8217;re protecting a single server or a complex jail infrastructure, the principles here should give you a solid&amp;nbsp;foundation.&lt;/p&gt;
&lt;h2 id="core-concepts"&gt;Core&amp;nbsp;Concepts&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; processes rules in order, but with an important twist: the &lt;strong&gt;last matching rule wins&lt;/strong&gt; (unless you&amp;nbsp;use &lt;code&gt;quick&lt;/code&gt;). This means you typically structure rules from general to specific, with explicit blocks at the top and specific allows below.&amp;nbsp;The &lt;code&gt;quick&lt;/code&gt; keyword short-circuits evaluation - when&amp;nbsp;a &lt;code&gt;quick&lt;/code&gt; rule matches, processing stops&amp;nbsp;immediately.&lt;/p&gt;
&lt;p&gt;The configuration file lives&amp;nbsp;at &lt;code&gt;/etc/pf.conf&lt;/code&gt;. After editing, validate syntax&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-nf&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If validation passes, load the&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;To enable &lt;span class="caps"&gt;PF&lt;/span&gt; at boot, add&amp;nbsp;to &lt;code&gt;/etc/rc.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pflog_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Since this guide covers &lt;span class="caps"&gt;NAT&lt;/span&gt; and routing traffic between jails and the internet, you must also enable packet forwarding. Without this, the firewall rules will load but traffic will die at the&amp;nbsp;interface:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="anatomy-of-a-pf-configuration"&gt;Anatomy of a &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;A&amp;nbsp;well-organized &lt;code&gt;pf.conf&lt;/code&gt; follows a predictable structure. Here&amp;#8217;s the skeleton I use across all&amp;nbsp;servers:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# --- Macros ---&lt;/span&gt;
&lt;span class="c1"&gt;# Named variables for interfaces and addresses&lt;/span&gt;

&lt;span class="c1"&gt;# --- Tables ---&lt;/span&gt;
&lt;span class="c1"&gt;# Dynamic lists of addresses&lt;/span&gt;

&lt;span class="c1"&gt;# --- Options ---&lt;/span&gt;
&lt;span class="c1"&gt;# Global behavior settings&lt;/span&gt;

&lt;span class="c1"&gt;# --- Scrub ---&lt;/span&gt;
&lt;span class="c1"&gt;# Packet normalization&lt;/span&gt;

&lt;span class="c1"&gt;# --- NAT ---&lt;/span&gt;
&lt;span class="c1"&gt;# Network address translation&lt;/span&gt;

&lt;span class="c1"&gt;# --- RDR ---&lt;/span&gt;
&lt;span class="c1"&gt;# Port redirections&lt;/span&gt;

&lt;span class="c1"&gt;# --- Filtering ---&lt;/span&gt;
&lt;span class="c1"&gt;# The actual firewall rules&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Each section has a distinct purpose. Let&amp;#8217;s walk through them with practical&amp;nbsp;examples.&lt;/p&gt;
&lt;h2 id="macros-naming-your-network"&gt;Macros: Naming Your&amp;nbsp;Network&lt;/h2&gt;
&lt;p&gt;Macros make configurations readable and maintainable. Define them at the&amp;nbsp;top:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Macros&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;bridge&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kd"&gt;interface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;created&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;by&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Bastille&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;framework&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;If&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;using&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;iocage&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;or&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;raw&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;jails&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;this&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;might&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;be&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;lo1&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;or&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;bridge0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="nx"&gt;int_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;

&lt;span class="nx"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;10.100.0.0/24&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2001:db8:1000:8000::/65&amp;quot;&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;For&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;NAT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;needed&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;single&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;address&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;CIDR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;block&lt;/span&gt;
&lt;span class="nx"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2001:db8:1000::1&amp;quot;&lt;/span&gt;

&lt;span class="nx"&gt;web_jail_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;10.100.0.10&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;web_jail_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2001:db8:1000:8000::10&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;db_jail_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;10.100.0.20&amp;quot;&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Lists&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;braces&lt;/span&gt;
&lt;span class="nx"&gt;trusted_ipv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{ 198.51.100.22, 203.0.113.50 }&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;trusted_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{ 2001:db8:ffff::/48 }&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Using macros means changing an &lt;span class="caps"&gt;IP&lt;/span&gt; address requires editing one line, not hunting through the entire ruleset.&amp;nbsp;The &lt;code&gt;$macro&lt;/code&gt; syntax references them in&amp;nbsp;rules.&lt;/p&gt;
&lt;h2 id="tables-dynamic-address-lists"&gt;Tables: Dynamic Address&amp;nbsp;Lists&lt;/h2&gt;
&lt;p&gt;Tables are &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s mechanism for large or changing address sets. Unlike macros, tables can be modified at runtime without reloading the entire&amp;nbsp;ruleset:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;#&lt;/span&gt; --- Tables ---
table &amp;lt;bruteforce&amp;gt; persist
table &amp;lt;jails_v4&amp;gt; { $jail_net }
table &amp;lt;jails_v6&amp;gt; { $jail_net6 }
table &amp;lt;trusted_v4&amp;gt; persist { $trusted_ipv4 }
table &amp;lt;trusted_v6&amp;gt; persist { $trusted_ipv6 }
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;persist&lt;/code&gt; keyword keeps the table in memory even when empty. This is essential for tables populated dynamically&amp;nbsp;(like &lt;code&gt;&amp;lt;bruteforce&amp;gt;&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;Manage tables at&amp;nbsp;runtime:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Show table contents&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;show

&lt;span class="c1"&gt;# Add an address&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.100

&lt;span class="c1"&gt;# Remove an address&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;delete&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.100

&lt;span class="c1"&gt;# Flush entire table&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;flush

&lt;span class="c1"&gt;# Load from file&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;trusted_v4&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.trusted.txt
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Tables scale efficiently - &lt;span class="caps"&gt;PF&lt;/span&gt; uses radix trees internally, so even tables with hundreds of thousands of entries (like GeoIP databases) perform&amp;nbsp;well.&lt;/p&gt;
&lt;h2 id="options-global-behavior"&gt;Options: Global&amp;nbsp;Behavior&lt;/h2&gt;
&lt;p&gt;The options section configures &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s global&amp;nbsp;behavior:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;#&lt;/span&gt; --- Options ---
set skip on lo0
set block-policy drop
set loginterface $ext_if
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Key options&amp;nbsp;explained:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;set skip on lo0&lt;/code&gt;&lt;/strong&gt;: Never filter loopback traffic. Critical for local&amp;nbsp;services.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;set block-policy drop&lt;/code&gt;&lt;/strong&gt;: Silently drop blocked packets&amp;nbsp;(vs. &lt;code&gt;return&lt;/code&gt; which sends &lt;span class="caps"&gt;RST&lt;/span&gt;/&lt;span class="caps"&gt;ICMP&lt;/span&gt;). Drop is better for security - it gives attackers no&amp;nbsp;information.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;set loginterface&lt;/code&gt;&lt;/strong&gt;: Enable per-interface packet/byte counters. View&amp;nbsp;with &lt;code&gt;pfctl -si&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For large tables (GeoIP filtering, for example), increase the table entry&amp;nbsp;limit:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;set limit table-entries 1000000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="scrub-packet-normalization"&gt;Scrub: Packet&amp;nbsp;Normalization&lt;/h2&gt;
&lt;p&gt;Scrubbing normalizes packets to prevent various attacks and fix fragmentation&amp;nbsp;issues:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;#&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;Scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="nv"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;fragment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;reassemble&lt;/span&gt;
&lt;span class="nv"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;random&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;mss&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1500&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;fragment reassemble&lt;/code&gt; directive reassembles fragmented packets before filtering - essential because fragments can be used to evade stateless filters.&amp;nbsp;The &lt;code&gt;random-id&lt;/code&gt; directive randomizes &lt;span class="caps"&gt;IP&lt;/span&gt; IDs on outbound packets, making traffic analysis harder.&amp;nbsp;The &lt;code&gt;max-mss&lt;/code&gt; clamps &lt;span class="caps"&gt;TCP&lt;/span&gt; &lt;span class="caps"&gt;MSS&lt;/span&gt; to prevent fragmentation issues, particularly important for tunneled traffic or &lt;span class="caps"&gt;VPN&lt;/span&gt;&amp;nbsp;scenarios.&lt;/p&gt;
&lt;p&gt;For specific interfaces with lower MTUs (tunnels, VSwitch connections), add targeted scrub&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;scrub out on gif0 max-mss 1240
scrub out on $int_if max-mss 1410
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="nat-address-translation"&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt;: Address&amp;nbsp;Translation&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt; translates private addresses to public ones for outbound traffic. With FreeBSD jails on private networks, &lt;span class="caps"&gt;NAT&lt;/span&gt; is essential for&amp;nbsp;IPv4:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;# --- NAT ---&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The parentheses&amp;nbsp;around &lt;code&gt;($ext_if)&lt;/code&gt; make the &lt;span class="caps"&gt;NAT&lt;/span&gt; rule dynamic - if the external interface&amp;#8217;s &lt;span class="caps"&gt;IP&lt;/span&gt; changes (&lt;span class="caps"&gt;DHCP&lt;/span&gt;), &lt;span class="caps"&gt;PF&lt;/span&gt; automatically uses the new&amp;nbsp;address.&lt;/p&gt;
&lt;p&gt;For IPv6, if you have a routed prefix, skip &lt;span class="caps"&gt;NAT&lt;/span&gt; entirely and route natively - this is the recommended approach. Jails get real global addresses and IPv6 works end-to-end as&amp;nbsp;designed.&lt;/p&gt;
&lt;p&gt;If you must &lt;span class="caps"&gt;NAT&lt;/span&gt; IPv6 (not recommended, but sometimes necessary), the syntax is similar. Ensure the target is a single address, not a &lt;span class="caps"&gt;CIDR&lt;/span&gt;&amp;nbsp;block:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;must&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;be&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;single&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;assigned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kd"&gt;external&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kd"&gt;interface&lt;/span&gt;
&lt;span class="nx"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;jails_v6&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;host_ipv6&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="rdr-port-forwarding"&gt;&lt;span class="caps"&gt;RDR&lt;/span&gt;: Port&amp;nbsp;Forwarding&lt;/h2&gt;
&lt;p&gt;Redirections forward incoming traffic to internal hosts. For a web server&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;# --- RDR ---&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;web_jail_v4&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;web_jail_v6&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;rdr pass&lt;/code&gt; syntax combines redirection with an implicit pass rule.&amp;nbsp;Without &lt;code&gt;pass&lt;/code&gt;, you&amp;#8217;d need a separate filter rule for the translated&amp;nbsp;traffic.&lt;/p&gt;
&lt;p&gt;For port translation (external port differs from&amp;nbsp;internal):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;# External 2222 -&amp;gt; internal 22&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;trusted_runner&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2222&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;deploy_jail&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Restricting the source in &lt;span class="caps"&gt;RDR&lt;/span&gt; rules is a powerful pattern - the service is invisible to everyone except specified&amp;nbsp;sources.&lt;/p&gt;
&lt;h2 id="filtering-the-ruleset-core"&gt;Filtering: The Ruleset&amp;nbsp;Core&lt;/h2&gt;
&lt;p&gt;The filtering section is where access control happens. Start with default&amp;nbsp;deny:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;#&lt;/span&gt; --- Filtering ---
&lt;span class="gh"&gt;#&lt;/span&gt; Block known bad actors immediately
block quick from &amp;lt;bruteforce&amp;gt;

&lt;span class="gh"&gt;#&lt;/span&gt; Default deny everything
block drop in log all
block drop out log all

&lt;span class="gh"&gt;#&lt;/span&gt; Allow all established connections out
pass out quick all keep state

&lt;span class="gh"&gt;#&lt;/span&gt; Anti-spoofing
antispoof quick for { $ext_if, bastille0 }
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="understanding-quick-and-state"&gt;Understanding Quick and&amp;nbsp;State&lt;/h3&gt;
&lt;p&gt;The &lt;code&gt;quick&lt;/code&gt; keyword stops rule processing on match. Use it for definitive&amp;nbsp;decisions:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Immediately drop known attackers - no further processing
block quick from &amp;lt;bruteforce&amp;gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;keep state&lt;/code&gt; directive enables stateful filtering. When an outbound connection is allowed, &lt;span class="caps"&gt;PF&lt;/span&gt; creates a state entry that automatically permits return traffic. This is why a&amp;nbsp;single &lt;code&gt;pass out&lt;/code&gt; rule handles both the outbound packet and inbound&amp;nbsp;replies.&lt;/p&gt;
&lt;p&gt;Note: While newer OpenBSD versions of &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;nbsp;make &lt;code&gt;keep state&lt;/code&gt; implicit, FreeBSD&amp;#8217;s &lt;span class="caps"&gt;PF&lt;/span&gt; is based on an older codebase. Being explicit&amp;nbsp;with &lt;code&gt;keep state&lt;/code&gt; ensures connection tracking works exactly as intended across FreeBSD&amp;nbsp;versions.&lt;/p&gt;
&lt;h3 id="ssh-protection-pattern"&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; Protection&amp;nbsp;Pattern&lt;/h3&gt;
&lt;p&gt;Protecting &lt;span class="caps"&gt;SSH&lt;/span&gt; deserves special attention. This pattern restricts access to trusted sources, and automatically blocks brute-force&amp;nbsp;attempts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# SSH only from trusted sources&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Block and log all other SSH attempts (for monitoring)&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh_blocked&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The state options deserve&amp;nbsp;explanation:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;max-src-conn 5&lt;/code&gt;&lt;/strong&gt;: Maximum 5 simultaneous connections per source &lt;span class="caps"&gt;IP&lt;/span&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;max-src-conn-rate 3/30&lt;/code&gt;&lt;/strong&gt;: Maximum 3 new connections per 30 seconds per&amp;nbsp;source&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;overload &amp;lt;bruteforce&amp;gt;&lt;/code&gt;&lt;/strong&gt;: Add violators to the bruteforce&amp;nbsp;table&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;flush global&lt;/code&gt;&lt;/strong&gt;: Kill all existing connections from the&amp;nbsp;offender&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The &lt;code&gt;label&lt;/code&gt; on the block rule tags logged packets for easy&amp;nbsp;filtering.&lt;/p&gt;
&lt;h3 id="essential-icmp"&gt;Essential &lt;span class="caps"&gt;ICMP&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Never block all &lt;span class="caps"&gt;ICMP&lt;/span&gt; - it breaks essential network&amp;nbsp;functionality:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Essential&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ICMPv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;required&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;function&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;echorep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;neighbrsol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;neighbradv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;toobig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;timex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;paramprob&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Useful&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ICMPv4&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;unreach&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The IPv6 types are critical:
- &lt;strong&gt;neighbrsol/neighbradv&lt;/strong&gt;: Neighbor Discovery (IPv6&amp;#8217;s &lt;span class="caps"&gt;ARP&lt;/span&gt; equivalent)
- &lt;strong&gt;toobig&lt;/strong&gt;: Path &lt;span class="caps"&gt;MTU&lt;/span&gt; Discovery (breaks connections if blocked)
- &lt;strong&gt;echoreq/echorep&lt;/strong&gt;: Ping (useful for&amp;nbsp;diagnostics)&lt;/p&gt;
&lt;h3 id="jail-egress-rules"&gt;Jail Egress&amp;nbsp;Rules&lt;/h3&gt;
&lt;p&gt;Allow jails to reach the internet while preventing them from directly accessing other internal&amp;nbsp;networks:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;#&lt;/span&gt; Jails can reach anywhere except internal networks
pass in quick on bastille0 from &amp;lt;jails_v4&amp;gt; to ! 10.100.0.0/24 keep state
pass in quick on bastille0 inet6 from &amp;lt;jails_v6&amp;gt; to ! 2001:db8:1000:8000::/65 keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;!&lt;/code&gt; negation ensures jails can reach the internet but can&amp;#8217;t directly contact other jails (unless explicitly permitted&amp;nbsp;elsewhere).&lt;/p&gt;
&lt;h3 id="inter-jail-communication"&gt;Inter-Jail&amp;nbsp;Communication&lt;/h3&gt;
&lt;p&gt;When jails need to communicate (database connections, for example), add explicit&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;# Web jail -&amp;gt; Database jail (PostgreSQL + Redis)&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;web_jail_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;db_jail_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5432&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6379&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This explicit allowlisting is preferable to permitting all inter-jail traffic - it documents and enforces your service&amp;nbsp;architecture.&lt;/p&gt;
&lt;h2 id="complete-example-configuration"&gt;Complete Example&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Here&amp;#8217;s a production-ready configuration for a server running multiple&amp;nbsp;jails:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# --- Macros ---&lt;/span&gt;
&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Jail bridge interface (bastille0 for Bastille, bridge0 or lo1 for others)&lt;/span&gt;
&lt;span class="n"&gt;int_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.100.0.0/24&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1000:8000::/65&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1000::1&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;web_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.100.0.10&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;web_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1000:8000::10&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;db_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.100.0.20&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;deploy_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.100.0.25&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;trusted_ipv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;{ 198.51.100.22, 203.0.113.50 }&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;trusted_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;{ 2001:db8:ffff::/48 }&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;deploy_runner&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;192.0.2.100&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# --- Tables ---&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;trusted_ipv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;trusted_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# --- Options ---&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;skip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lo0&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loginterface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;

&lt;span class="c1"&gt;# --- Scrub ---&lt;/span&gt;
&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fragment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;reassemble&lt;/span&gt;
&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;mss&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1500&lt;/span&gt;

&lt;span class="c1"&gt;# --- NAT ---&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- RDR ---&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;web_v4&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;web_v6&lt;/span&gt;

&lt;span class="c1"&gt;# Deployment SSH from CI runner only&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;deploy_runner&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2222&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;deploy_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;

&lt;span class="c1"&gt;# --- Filtering ---&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="n"&gt;antispoof&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Management SSH&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh_blocked&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Essential ICMP&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echorep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbrsol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbradv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;toobig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;timex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paramprob&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unreach&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Web jail public access&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;web_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;web_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Inter-jail: web -&amp;gt; database&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;web_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;db_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5432&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6379&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Jail egress&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;10.100&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="tips-and-tricks"&gt;Tips and&amp;nbsp;Tricks&lt;/h2&gt;
&lt;h3 id="viewing-loaded-rules"&gt;Viewing Loaded&amp;nbsp;Rules&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Show all rules with rule numbers&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-sr

&lt;span class="c1"&gt;# Verbose output with statistics&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-sr&lt;span class="w"&gt; &lt;/span&gt;-v

&lt;span class="c1"&gt;# Show NAT rules&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-sn

&lt;span class="c1"&gt;# Show state table&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-ss

&lt;span class="c1"&gt;# Show interface statistics&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-si
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="logging-and-debugging"&gt;Logging and&amp;nbsp;Debugging&lt;/h3&gt;
&lt;p&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; logs&amp;nbsp;to &lt;code&gt;pflog0&lt;/code&gt;, viewable with&amp;nbsp;tcpdump:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Live log viewing&lt;/span&gt;
tcpdump&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;-ttt&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;pflog0

&lt;span class="c1"&gt;# Read saved log file&lt;/span&gt;
tcpdump&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;-ttt&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;/var/log/pflog

&lt;span class="c1"&gt;# Filter by label&lt;/span&gt;
tcpdump&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;-ttt&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;pflog0&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;label ssh_blocked&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Enable pflog in rc.conf and optionally configure rotation&amp;nbsp;in &lt;code&gt;/etc/newsyslog.conf&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="clearing-brute-force-entries"&gt;Clearing Brute-Force&amp;nbsp;Entries&lt;/h3&gt;
&lt;p&gt;Sometimes legitimate users get caught in rate&amp;nbsp;limiting:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Show who&amp;#39;s blocked&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;show

&lt;span class="c1"&gt;# Unblock specific IP&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;delete&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;198&lt;/span&gt;.51.100.50

&lt;span class="c1"&gt;# Flush all blocked IPs&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;flush
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Consider a cron job to expire old&amp;nbsp;entries:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Clear entries older than 24 hours&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;bruteforce&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;expire&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;86400&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="testing-changes-safely"&gt;Testing Changes&amp;nbsp;Safely&lt;/h3&gt;
&lt;p&gt;Before applying potentially locking changes, use the classic sysadmin safety&amp;nbsp;net:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Validate syntax first&lt;/span&gt;
pfctl&lt;span class="w"&gt; &lt;/span&gt;-nf&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.conf

&lt;span class="c1"&gt;# Apply with automatic rollback&lt;/span&gt;
&lt;span class="c1"&gt;# (If you get locked out, rules revert after timeout)&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pfctl -f /etc/pf.conf.backup&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;at&lt;span class="w"&gt; &lt;/span&gt;now&lt;span class="w"&gt; &lt;/span&gt;+&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;minutes
pfctl&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.conf
&lt;span class="c1"&gt;# If everything works, cancel the at job&lt;/span&gt;
atq
atrm&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;job_number&amp;gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note:&amp;nbsp;The &lt;code&gt;at&lt;/code&gt; command requires&amp;nbsp;the &lt;code&gt;atd&lt;/code&gt; daemon. On minimal installs, you may need to enable it&amp;nbsp;first:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# In /etc/rc.conf&lt;/span&gt;
&lt;span class="nv"&gt;atd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Then start it&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;atd&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="monitoring-specific-traffic"&gt;Monitoring Specific&amp;nbsp;Traffic&lt;/h3&gt;
&lt;p&gt;Add temporary rules with counters for&amp;nbsp;debugging:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pass in log on $ext_if proto tcp to port 443 label &amp;quot;https_debug&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then&amp;nbsp;watch:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-sr&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;https_debug
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="sidebar-authpf-for-bastion-hosts"&gt;Sidebar: authpf for Bastion&amp;nbsp;Hosts&lt;/h2&gt;
&lt;p&gt;For bastion hosts or jump servers, authpf provides per-user firewall rules that activate upon &lt;span class="caps"&gt;SSH&lt;/span&gt; login. Instead of static rules, the firewall dynamically permits access based on who&amp;#8217;s&amp;nbsp;connected.&lt;/p&gt;
&lt;p&gt;authpf works by loading a user-specific ruleset&amp;nbsp;(from &lt;code&gt;/etc/authpf/users/$USER/&lt;/code&gt; or &lt;code&gt;/etc/authpf/authpf.rules&lt;/code&gt;) when the user logs in via &lt;span class="caps"&gt;SSH&lt;/span&gt;. When they log out, the rules are removed. This creates a &amp;#8220;knock first&amp;#8221; security model where services are invisible until&amp;nbsp;authenticated.&lt;/p&gt;
&lt;p&gt;Basic&amp;nbsp;setup:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;Set the user&amp;#8217;s shell&amp;nbsp;to &lt;code&gt;/usr/sbin/authpf&lt;/code&gt;:
   &lt;code&gt;bash
   pw usermod bastion_user -s /usr/sbin/authpf&lt;/code&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;Create &lt;code&gt;/etc/authpf/authpf.rules&lt;/code&gt; with rules to load on&amp;nbsp;login:
   &lt;code&gt;pf
   pass in quick on $ext_if from $user_ip to $internal_net&lt;/code&gt;&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;The&amp;nbsp;macro &lt;code&gt;$user_ip&lt;/code&gt; is automatically set to the connecting client&amp;#8217;s&amp;nbsp;address.&lt;/p&gt;
&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;This approach is powerful for restricting access to internal networks: users must &lt;span class="caps"&gt;SSH&lt;/span&gt; to the bastion first, then their &lt;span class="caps"&gt;IP&lt;/span&gt; gets temporary firewall access. When the &lt;span class="caps"&gt;SSH&lt;/span&gt; session ends, access is revoked. Combined with short-lived certificates or hardware tokens, authpf creates a robust zero-trust perimeter without &lt;span class="caps"&gt;VPN&lt;/span&gt;&amp;nbsp;complexity.&lt;/p&gt;
&lt;p&gt;For production deployments, consider authpf as an alternative to VPNs for administrative access - it&amp;#8217;s simpler to audit, integrates with existing &lt;span class="caps"&gt;SSH&lt;/span&gt; infrastructure, and provides per-session access&amp;nbsp;control.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; rewards careful configuration with robust, maintainable security. The patterns here - default deny, explicit allowlists, brute-force protection, and clean macro organization - scale from single servers to complex multi-jail&amp;nbsp;deployments.&lt;/p&gt;
&lt;p&gt;The key principles worth&amp;nbsp;remembering:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Default deny with explicit allows&lt;/strong&gt;: Never assume traffic is&amp;nbsp;safe&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;State tracking&lt;/strong&gt;: Let &lt;span class="caps"&gt;PF&lt;/span&gt; manage connection state rather than permitting return traffic&amp;nbsp;explicitly&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Tables for dynamic data&lt;/strong&gt;: Use tables for address lists that change or grow&amp;nbsp;large&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Log selectively&lt;/strong&gt;: Log blocked traffic for analysis, but avoid logging high-volume allowed&amp;nbsp;traffic&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Test before applying&lt;/strong&gt;: Always validate syntax and have a rollback&amp;nbsp;plan&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s syntax takes time to internalize, but once it clicks, writing firewall rules becomes almost intuitive. The investment pays dividends in security, clarity, and the confidence that comes from truly understanding what your firewall&amp;nbsp;permits.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.openbsd.org/faq/pf/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; - The OpenBSD Packet Filter&lt;/a&gt; (&lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s canonical&amp;nbsp;documentation)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/firewalls/"&gt;FreeBSD Handbook:&amp;nbsp;Firewalls&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pf.conf"&gt;FreeBSD pf.conf(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=authpf"&gt;authpf(8) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;A good firewall is invisible when everything works and invaluable when something goes wrong. &lt;span class="caps"&gt;PF&lt;/span&gt; sits quietly in the kernel, inspecting every packet, blocking the noise, and letting through exactly what you&amp;#8217;ve specified. It&amp;#8217;s one of those tools that makes you appreciate the Unix philosophy: do one thing well, make it composable, keep it&amp;nbsp;simple.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="firewall"/><category term="pf"/><category term="security"/><category term="networking"/></entry><entry><title>Integrating FreeBSD 15 with FreeIPA: Native Kerberos and LDAP Authentication</title><link href="https://hofstede.it/integrating-freebsd-15-with-freeipa-native-kerberos-and-ldap-authentication/" rel="alternate"/><published>2026-01-25T00:00:00+01:00</published><updated>2026-01-25T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-01-25:/integrating-freebsd-15-with-freeipa-native-kerberos-and-ldap-authentication/</id><summary type="html">&lt;p&gt;A clean approach to integrating FreeBSD 15 into a FreeIPA realm using native components - Kerberos for authentication, &lt;span class="caps"&gt;LDAP&lt;/span&gt; for identity, and no local user&amp;nbsp;management.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Logo" src="https://hofstede.it/images/2026-01-25-freebsd-freeipa-kerberos-ldap.jpg" title="FreeBSD 15 with FreeIPA: Header image"&gt;&lt;/p&gt;
&lt;p&gt;Most FreeIPA documentation assumes you&amp;#8217;re running Linux and will&amp;nbsp;use &lt;code&gt;ipa-client-install&lt;/code&gt; to join hosts to the realm. FreeBSD doesn&amp;#8217;t have this luxury. There&amp;#8217;s no official &lt;span class="caps"&gt;IPA&lt;/span&gt; client, and the enrollment scripts expect systemd and other Linux-specific components. But that&amp;#8217;s not necessarily a&amp;nbsp;disadvantage.&lt;/p&gt;
&lt;p&gt;FreeBSD&amp;#8217;s native Kerberos implementation and the&amp;nbsp;lightweight &lt;code&gt;nslcd&lt;/code&gt; daemon provide everything needed to integrate with FreeIPA cleanly. The result is arguably more elegant than the Linux approach: pure &lt;span class="caps"&gt;GSSAPI&lt;/span&gt; authentication via &lt;span class="caps"&gt;SSH&lt;/span&gt;, &lt;span class="caps"&gt;LDAP&lt;/span&gt;-backed identity lookups, and zero local user management. No &lt;span class="caps"&gt;SSSD&lt;/span&gt;, no realm daemon, no&amp;nbsp;complexity.&lt;/p&gt;
&lt;p&gt;This guide details integrating a FreeBSD 15.0 host into a Red Hat Identity Management (IdM) or FreeIPA realm using only native FreeBSD&amp;nbsp;components.&lt;/p&gt;
&lt;h2 id="design-philosophy"&gt;Design&amp;nbsp;Philosophy&lt;/h2&gt;
&lt;p&gt;The approach here follows what I call &amp;#8220;Clean and Sane&amp;nbsp;Engineering&amp;#8221;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Authentication&lt;/strong&gt;: Pure Kerberos via &lt;span class="caps"&gt;GSSAPI&lt;/span&gt;. Users authenticate with their Kerberos tickets - no passwords transmitted over the&amp;nbsp;network&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Identity&lt;/strong&gt;: &lt;span class="caps"&gt;LDAP&lt;/span&gt;&amp;nbsp;via &lt;code&gt;nslcd&lt;/code&gt; for user and group lookups. The &lt;span class="caps"&gt;NSS&lt;/span&gt; layer queries FreeIPA&amp;#8217;s &lt;span class="caps"&gt;LDAP&lt;/span&gt; directory&amp;nbsp;transparently&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Management&lt;/strong&gt;: Stateless. No local users created manually. The host is a pure consumer of central identity&amp;nbsp;services&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This separation of concerns mirrors how enterprise authentication should work: the &lt;span class="caps"&gt;KDC&lt;/span&gt; handles cryptographic verification, the directory provides attributes, and the local system just needs to know where to&amp;nbsp;ask.&lt;/p&gt;
&lt;h3 id="why-not-sssd"&gt;Why Not &lt;span class="caps"&gt;SSSD&lt;/span&gt;?&lt;/h3&gt;
&lt;p&gt;The Linux world has largely converged on &lt;span class="caps"&gt;SSSD&lt;/span&gt; (System Security Services Daemon) for this kind of integration. It&amp;#8217;s a powerful tool, but it&amp;#8217;s also a monolith with a sprawling dependency tree, D-Bus requirements, and complexity that feels foreign on a FreeBSD&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;Following Unix philosophy and &lt;span class="caps"&gt;KISS&lt;/span&gt; principles, we actively avoid such heavyweight solutions. Each component in this setup does one thing&amp;nbsp;well: &lt;code&gt;nslcd&lt;/code&gt; handles &lt;span class="caps"&gt;LDAP&lt;/span&gt; lookups, the&amp;nbsp;native &lt;code&gt;krb5&lt;/code&gt; library handles Kerberos, &lt;span class="caps"&gt;PAM&lt;/span&gt; modules handle session setup. These are small, focused tools that compose&amp;nbsp;together.&lt;/p&gt;
&lt;p&gt;Yes, the initial setup requires understanding how the pieces fit together. But the result is a solution that&amp;#8217;s lighter, more transparent, and equally robust. When something breaks, you can debug each component independently. There&amp;#8217;s no opaque daemon trying to be clever on your behalf. The configuration files are human-readable, the logs are straightforward, and the failure modes are&amp;nbsp;predictable.&lt;/p&gt;
&lt;p&gt;This trade-off, slightly more upfront complexity for long-term simplicity and maintainability, is at the heart of the FreeBSD philosophy. We&amp;#8217;re not fighting the system. We&amp;#8217;re working with its native&amp;nbsp;primitives.&lt;/p&gt;
&lt;h2 id="prerequisites-and-packages"&gt;Prerequisites and&amp;nbsp;Packages&lt;/h2&gt;
&lt;p&gt;The package requirements are minimal. Install the &lt;span class="caps"&gt;LDAP&lt;/span&gt; connector and the &lt;span class="caps"&gt;PAM&lt;/span&gt; module for automatic home directory&amp;nbsp;creation:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;nss-pam-ldapd&lt;span class="w"&gt; &lt;/span&gt;pam_mkhomedir&lt;span class="w"&gt; &lt;/span&gt;sudo
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;nss-pam-ldapd&lt;/code&gt; package provides&amp;nbsp;the &lt;code&gt;nslcd&lt;/code&gt; daemon, which bridges FreeBSD&amp;#8217;s &lt;span class="caps"&gt;NSS&lt;/span&gt; (Name Service Switch) to &lt;span class="caps"&gt;LDAP&lt;/span&gt;. Unlike heavier solutions like &lt;span class="caps"&gt;SSSD&lt;/span&gt;, it does one thing well: translate &lt;span class="caps"&gt;NSS&lt;/span&gt; queries into &lt;span class="caps"&gt;LDAP&lt;/span&gt;&amp;nbsp;lookups.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;pam_mkhomedir&lt;/code&gt; module handles a common pain point: Creating home directories on first login. Without it, users would authenticate successfully but fail to get a shell&amp;nbsp;because &lt;code&gt;/home/username&lt;/code&gt; doesn&amp;#8217;t&amp;nbsp;exist.&lt;/p&gt;
&lt;h2 id="host-identity-and-keytab-provisioning"&gt;Host Identity and Keytab&amp;nbsp;Provisioning&lt;/h2&gt;
&lt;p&gt;Since there&amp;#8217;s&amp;nbsp;no &lt;code&gt;ipa-client-install&lt;/code&gt;, we provision the host identity manually. This is actually straightforward: Create the host entry on the &lt;span class="caps"&gt;IPA&lt;/span&gt; server and export a&amp;nbsp;keytab.&lt;/p&gt;
&lt;h3 id="on-the-ipa-server"&gt;On the &lt;span class="caps"&gt;IPA&lt;/span&gt;&amp;nbsp;Server&lt;/h3&gt;
&lt;p&gt;First, create the host entry. If &lt;span class="caps"&gt;DNS&lt;/span&gt; is integrated with &lt;span class="caps"&gt;IPA&lt;/span&gt;,&amp;nbsp;the &lt;code&gt;--ip-address&lt;/code&gt; flag will create the A record&amp;nbsp;automatically:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;host-add&lt;span class="w"&gt; &lt;/span&gt;bsdhost.example.com&lt;span class="w"&gt; &lt;/span&gt;--ip-address&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.168.1.25
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then generate and export the keytab.&amp;nbsp;The &lt;code&gt;ipa-getkeytab&lt;/code&gt; command randomizes the host&amp;#8217;s password (just like machine account passwords in Active Directory) and saves the cryptographic key&amp;nbsp;material:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa-getkeytab&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;idm01.example.com&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;host/bsdhost.example.com@EXAMPLE.COM&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;/tmp/bsdhost.keytab
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="on-the-freebsd-host"&gt;On the FreeBSD&amp;nbsp;Host&lt;/h3&gt;
&lt;p&gt;Transfer the keytab securely&amp;nbsp;(via &lt;code&gt;scp&lt;/code&gt;)&amp;nbsp;to &lt;code&gt;/etc/krb5.keytab&lt;/code&gt;. The permissions here&amp;nbsp;matter:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chown&lt;span class="w"&gt; &lt;/span&gt;root:nslcd&lt;span class="w"&gt; &lt;/span&gt;/etc/krb5.keytab
chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;640&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/etc/krb5.keytab
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;nslcd&lt;/code&gt; daemon needs read access to this keytab to authenticate its &lt;span class="caps"&gt;LDAP&lt;/span&gt; queries via &lt;span class="caps"&gt;GSSAPI&lt;/span&gt;. Without this, directory lookups will fail when the host&amp;#8217;s cached credentials&amp;nbsp;expire.&lt;/p&gt;
&lt;p&gt;You also need to ensure, sshd can read the file (or &lt;span class="caps"&gt;SSH&lt;/span&gt; authentication will fail). That can be done either with ACLs or group membership&amp;nbsp;for &lt;code&gt;sshd&lt;/code&gt; in&amp;nbsp;the &lt;code&gt;nslcd&lt;/code&gt; group.&lt;/p&gt;
&lt;h2 id="kerberos-configuration"&gt;Kerberos&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;The Kerberos configuration is minimal. FreeBSD&amp;#8217;s&amp;nbsp;native &lt;code&gt;krb5&lt;/code&gt; implementation needs to know where the &lt;span class="caps"&gt;KDC&lt;/span&gt; is and which realm to&amp;nbsp;use.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/krb5.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[libdefaults]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;default_realm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;EXAMPLE.COM&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;dns_lookup_kdc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;dns_lookup_realm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;false&lt;/span&gt;

&lt;span class="k"&gt;[realms]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;EXAMPLE.COM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;kdc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;idm01.example.com&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;admin_server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;idm01.example.com&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;[domain_realm]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;.example.com&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;EXAMPLE.COM&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;example.com&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;EXAMPLE.COM&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I explicitly disable &lt;span class="caps"&gt;DNS&lt;/span&gt; lookups for the &lt;span class="caps"&gt;KDC&lt;/span&gt;. While FreeIPA publishes &lt;span class="caps"&gt;SRV&lt;/span&gt; records, hardcoding the &lt;span class="caps"&gt;KDC&lt;/span&gt; hostname makes the configuration deterministic and easier to debug. In environments with multiple KDCs, you can list them all in&amp;nbsp;the &lt;code&gt;kdc&lt;/code&gt; directive.&lt;/p&gt;
&lt;p&gt;Verify the configuration&amp;nbsp;works:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;klist&lt;span class="w"&gt; &lt;/span&gt;-k
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This should display the host principal from the&amp;nbsp;keytab:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Keytab name: FILE:/etc/krb5.keytab
KVNO Principal
---- --------------------------------------------------------------------------
   1 host/bsdhost.example.com@EXAMPLE.COM
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="ldap-configuration-with-nslcd"&gt;&lt;span class="caps"&gt;LDAP&lt;/span&gt; Configuration with&amp;nbsp;nslcd&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;nslcd&lt;/code&gt; daemon handles all &lt;span class="caps"&gt;LDAP&lt;/span&gt; communication. It runs as an unprivileged user, connects to FreeIPA&amp;#8217;s &lt;span class="caps"&gt;LDAP&lt;/span&gt; server, and responds to &lt;span class="caps"&gt;NSS&lt;/span&gt; queries over a Unix&amp;nbsp;socket.&lt;/p&gt;
&lt;p&gt;The key insight is using &lt;span class="caps"&gt;SASL&lt;/span&gt;/&lt;span class="caps"&gt;GSSAPI&lt;/span&gt; authentication&amp;nbsp;- &lt;code&gt;nslcd&lt;/code&gt; authenticates to &lt;span class="caps"&gt;LDAP&lt;/span&gt; using the host&amp;#8217;s Kerberos keytab rather than a bind password. This eliminates stored credentials and leverages the existing trust&amp;nbsp;relationship.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/usr/local/etc/nslcd.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Run as unprivileged user
uid nslcd
gid nslcd

# Connection details
uri ldap://idm01.example.com
base dc=example,dc=com

# Authentication: Use the system keytab (Host Principal)
sasl_mech GSSAPI
sasl_realm EXAMPLE.COM

# Mapping: Force all IPA users to use /bin/sh
# IPA typically defaults to /bin/bash, which lives in /usr/local/bin on FreeBSD
map passwd loginShell &amp;quot;/bin/sh&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;loginShell&lt;/code&gt; mapping deserves explanation. FreeIPA&amp;nbsp;stores &lt;code&gt;/bin/bash&lt;/code&gt; as the default shell, but on FreeBSD, bash (if installed) lives&amp;nbsp;at &lt;code&gt;/usr/local/bin/bash&lt;/code&gt;. Rather than requiring every user to update their shell in &lt;span class="caps"&gt;IPA&lt;/span&gt;, we override it locally&amp;nbsp;to &lt;code&gt;/bin/sh&lt;/code&gt;, which is guaranteed to exist. Users who need bash can set it explicitly in their&amp;nbsp;profile.&lt;/p&gt;
&lt;p&gt;Enable and start the&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;nslcd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;nslcd&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="nss-configuration"&gt;&lt;span class="caps"&gt;NSS&lt;/span&gt;&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Tell FreeBSD&amp;#8217;s name service switch to query &lt;span class="caps"&gt;LDAP&lt;/span&gt; after local files. The order matters - local accounts (root, system users) take precedence, with &lt;span class="caps"&gt;LDAP&lt;/span&gt; providing everything&amp;nbsp;else.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/nsswitch.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;group: files ldap
passwd: files ldap
hosts: files dns
networks: files
shells: files
services: compat
protocols: files
rpc: files
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;We avoid the&amp;nbsp;legacy &lt;code&gt;compat&lt;/code&gt; mode for passwd and group -&amp;nbsp;the &lt;code&gt;files ldap&lt;/code&gt; ordering is cleaner and more&amp;nbsp;predictable.&lt;/p&gt;
&lt;p&gt;Test the integration&amp;nbsp;immediately:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;id&lt;span class="w"&gt; &lt;/span&gt;someuser
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This should return &lt;span class="caps"&gt;UID&lt;/span&gt; and &lt;span class="caps"&gt;GID&lt;/span&gt; information from&amp;nbsp;FreeIPA:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;uid=1234(someuser) gid=1234(someuser) groups=1234(someuser),5000(admins)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If this fails, check&amp;nbsp;that &lt;code&gt;nslcd&lt;/code&gt; is running and can read the keytab. The logs&amp;nbsp;in &lt;code&gt;/var/log/messages&lt;/code&gt; usually reveal the&amp;nbsp;problem.&lt;/p&gt;
&lt;h2 id="ssh-configuration"&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Configure &lt;span class="caps"&gt;SSH&lt;/span&gt; to accept Kerberos tickets via &lt;span class="caps"&gt;GSSAPI&lt;/span&gt;. This enables true single sign-on - users with valid Kerberos tickets can &lt;span class="caps"&gt;SSH&lt;/span&gt; without any password&amp;nbsp;prompt.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/ssh/sshd_config&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Disable password authentication entirely
PasswordAuthentication no
KbdInteractiveAuthentication no

# Optional: Disable pubkey if relying purely on Kerberos
# PubkeyAuthentication no

# Kerberos / GSSAPI authentication
GSSAPIAuthentication yes
GSSAPICleanupCredentials yes
GSSAPIStrictAcceptorCheck no
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;GSSAPIStrictAcceptorCheck no&lt;/code&gt; setting relaxes hostname verification. This is sometimes necessary when the &lt;span class="caps"&gt;SSH&lt;/span&gt; server&amp;#8217;s hostname doesn&amp;#8217;t exactly match the principal in the keytab (e.g., connecting via &lt;span class="caps"&gt;IP&lt;/span&gt; or a &lt;span class="caps"&gt;CNAME&lt;/span&gt;).&lt;/p&gt;
&lt;p&gt;Restart &lt;span class="caps"&gt;SSH&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;sshd&lt;span class="w"&gt; &lt;/span&gt;restart
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="pam-configuration-for-home-directories"&gt;&lt;span class="caps"&gt;PAM&lt;/span&gt; Configuration for Home&amp;nbsp;Directories&lt;/h2&gt;
&lt;p&gt;When a user logs in for the first time, their home directory won&amp;#8217;t exist.&amp;nbsp;The &lt;code&gt;pam_mkhomedir&lt;/code&gt; module creates it automatically with appropriate&amp;nbsp;permissions.&lt;/p&gt;
&lt;p&gt;Edit the &lt;span class="caps"&gt;SSH&lt;/span&gt; &lt;span class="caps"&gt;PAM&lt;/span&gt; configuration to add the&amp;nbsp;module:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/pam.d/sshd&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;auth            required        pam_unix.so             no_warn try_first_pass

# account
account         required        pam_nologin.so
account         required        pam_login_access.so
account         required        pam_unix.so

# session
session         required        pam_mkhomedir.so        mode=0700
session         required        pam_permit.so

# password
password        required        pam_unix.so             no_warn try_first_pass
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;mode=0700&lt;/code&gt; ensures home directories are created with restrictive&amp;nbsp;permissions.&lt;/p&gt;
&lt;p&gt;Before this works, verify&amp;nbsp;that &lt;code&gt;/home&lt;/code&gt; exists as a symlink&amp;nbsp;to &lt;code&gt;/usr/home&lt;/code&gt; (the FreeBSD&amp;nbsp;convention):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ls&lt;span class="w"&gt; &lt;/span&gt;-la&lt;span class="w"&gt; &lt;/span&gt;/home
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If &lt;code&gt;/home&lt;/code&gt; doesn&amp;#8217;t exist or isn&amp;#8217;t a&amp;nbsp;symlink:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ln&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;/usr/home&lt;span class="w"&gt; &lt;/span&gt;/home
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="sudo-configuration"&gt;Sudo&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Instead of managing local group membership, leverage FreeIPA&amp;#8217;s groups directly. Members of the &lt;span class="caps"&gt;IPA&lt;/span&gt; &lt;code&gt;admins&lt;/code&gt; group get sudo access without any local configuration per&amp;nbsp;user.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/usr/local/etc/sudoers&lt;/strong&gt; (or&amp;nbsp;use &lt;code&gt;visudo&lt;/code&gt;)&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Allow members of the LDAP &amp;#39;admins&amp;#39; group full sudo access
%admins ALL=(ALL:ALL) ALL
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;%&lt;/code&gt; prefix tells sudo to&amp;nbsp;treat &lt;code&gt;admins&lt;/code&gt; as a group name.&amp;nbsp;Since &lt;code&gt;nslcd&lt;/code&gt; handles group lookups, this transparently queries&amp;nbsp;FreeIPA.&lt;/p&gt;
&lt;p&gt;For more granular control, you can define multiple group-based&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;%admins ALL=(ALL:ALL) ALL
%developers ALL=(ALL) /usr/local/bin/docker, /usr/local/bin/podman
%dba ALL=(postgres) ALL
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="verification"&gt;Verification&lt;/h2&gt;
&lt;p&gt;With everything configured, verify each&amp;nbsp;component:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;1. Identity&amp;nbsp;Lookups&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;id&lt;span class="w"&gt; &lt;/span&gt;admin
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Expected output shows UIDs and groups from &lt;span class="caps"&gt;IPA&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;uid=1000(admin) gid=1000(admin) groups=1000(admin),5000(admins)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;2. Keytab&amp;nbsp;Status&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;klist&lt;span class="w"&gt; &lt;/span&gt;-k
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Should list the host&amp;nbsp;principal.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;3. &lt;span class="caps"&gt;SSH&lt;/span&gt; Login&amp;nbsp;Test&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;From another machine with a valid Kerberos&amp;nbsp;ticket:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;kinit&lt;span class="w"&gt; &lt;/span&gt;someuser@EXAMPLE.COM
ssh&lt;span class="w"&gt; &lt;/span&gt;bsdhost.example.com
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The login should succeed without a password prompt. After&amp;nbsp;login:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;pwd&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Should&amp;nbsp;show &lt;code&gt;/home/someuser&lt;/code&gt; (created automatically&amp;nbsp;by &lt;code&gt;pam_mkhomedir&lt;/code&gt;).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;4. Sudo&amp;nbsp;Test&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;If the user is in&amp;nbsp;the &lt;code&gt;admins&lt;/code&gt; group:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;whoami
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Should&amp;nbsp;return &lt;code&gt;root&lt;/code&gt; without asking for a password&amp;nbsp;(assuming &lt;code&gt;NOPASSWD&lt;/code&gt; is configured) or after entering the user&amp;#8217;s Kerberos&amp;nbsp;password.&lt;/p&gt;
&lt;h2 id="troubleshooting"&gt;Troubleshooting&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;nslcd fails to start or lookups&amp;nbsp;fail&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Check the keytab&amp;nbsp;permissions:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ls&lt;span class="w"&gt; &lt;/span&gt;-la&lt;span class="w"&gt; &lt;/span&gt;/etc/krb5.keytab
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;nslcd&lt;/code&gt; user (or group) must have read access. Also verify the keytab contains valid&amp;nbsp;keys:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;klist&lt;span class="w"&gt; &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;/etc/krb5.keytab
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; rejects &lt;span class="caps"&gt;GSSAPI&lt;/span&gt;&amp;nbsp;authentication&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Ensure the hostname matches the keytab principal.&amp;nbsp;Check &lt;code&gt;/var/log/auth.log&lt;/code&gt; for details. Common&amp;nbsp;issues:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Connecting via &lt;span class="caps"&gt;IP&lt;/span&gt; when the keytab has only the &lt;span class="caps"&gt;FQDN&lt;/span&gt;&lt;/li&gt;
&lt;li&gt;Clock skew between the FreeBSD host and &lt;span class="caps"&gt;KDC&lt;/span&gt; (Kerberos is sensitive to time&amp;nbsp;differences)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Users can authenticate but have no home&amp;nbsp;directory&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;Verify &lt;code&gt;pam_mkhomedir&lt;/code&gt; is&amp;nbsp;in &lt;code&gt;/etc/pam.d/sshd&lt;/code&gt; and&amp;nbsp;that &lt;code&gt;/home&lt;/code&gt; is a symlink&amp;nbsp;to &lt;code&gt;/usr/home&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Group membership not&amp;nbsp;visible&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;This usually&amp;nbsp;indicates &lt;code&gt;nslcd&lt;/code&gt; isn&amp;#8217;t querying group information correctly. Verify&amp;nbsp;the &lt;code&gt;base&lt;/code&gt; &lt;span class="caps"&gt;DN&lt;/span&gt;&amp;nbsp;in &lt;code&gt;nslcd.conf&lt;/code&gt; includes the container where groups are stored. For &lt;span class="caps"&gt;IPA&lt;/span&gt;, this is&amp;nbsp;typically &lt;code&gt;cn=groups,cn=accounts,dc=example,dc=com&lt;/code&gt;, but using&amp;nbsp;just &lt;code&gt;dc=example,dc=com&lt;/code&gt; as the base should work with subtree&amp;nbsp;searches.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Integrating FreeBSD with FreeIPA doesn&amp;#8217;t require complex third-party tools or Linux-specific software. The combination of native&amp;nbsp;Kerberos, &lt;code&gt;nslcd&lt;/code&gt; for &lt;span class="caps"&gt;LDAP&lt;/span&gt;, and standard &lt;span class="caps"&gt;PAM&lt;/span&gt; modules provides a clean, maintainable&amp;nbsp;solution.&lt;/p&gt;
&lt;p&gt;The key advantages of this&amp;nbsp;approach:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;No local user management&lt;/strong&gt; - users exist only in &lt;span class="caps"&gt;IPA&lt;/span&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;True &lt;span class="caps"&gt;SSO&lt;/span&gt;&lt;/strong&gt; - Kerberos tickets authenticate &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;sessions&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Minimal attack surface&lt;/strong&gt; - no &lt;span class="caps"&gt;SSSD&lt;/span&gt;, no realm daemon, no Python&amp;nbsp;dependencies&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Stateless configuration&lt;/strong&gt; - rebuild the host without losing identity&amp;nbsp;integration&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This pattern scales well across a fleet of FreeBSD hosts. Once you&amp;#8217;ve configured one system, the configuration files can be templated and deployed with Ansible or similar tools. The only per-host customization is the keytab, which &lt;span class="caps"&gt;IPA&lt;/span&gt; generates&amp;nbsp;automatically.&lt;/p&gt;
&lt;p&gt;For environments already invested in FreeIPA for Linux hosts, adding FreeBSD becomes trivial - just another host entry in the directory, with the same users, groups, and policies applying uniformly across the&amp;nbsp;infrastructure.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://freeipa.readthedocs.io/"&gt;FreeIPA&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/security/#kerberos5"&gt;FreeBSD Handbook:&amp;nbsp;Kerberos&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://arthurdejong.org/nss-pam-ldapd/"&gt;nss-pam-ldapd&amp;nbsp;Project&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://web.mit.edu/kerberos/krb5-latest/doc/"&gt;&lt;span class="caps"&gt;MIT&lt;/span&gt; Kerberos&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pam.conf"&gt;&lt;span class="caps"&gt;PAM&lt;/span&gt; Configuration (FreeBSD Man&amp;nbsp;Page)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="freeipa"/><category term="kerberos"/><category term="ldap"/><category term="authentication"/><category term="security"/></entry><entry><title>Self-Hosted CryptPad on FreeBSD with VNET Jails and Caddy</title><link href="https://hofstede.it/self-hosted-cryptpad-on-freebsd-with-vnet-jails-and-caddy/" rel="alternate"/><published>2026-01-24T00:00:00+01:00</published><updated>2026-01-24T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-01-24:/self-hosted-cryptpad-on-freebsd-with-vnet-jails-and-caddy/</id><summary type="html">&lt;p&gt;Running CryptPad in a FreeBSD &lt;span class="caps"&gt;VNET&lt;/span&gt; jail with isolated networking, &lt;span class="caps"&gt;NAT&lt;/span&gt; via &lt;span class="caps"&gt;PF&lt;/span&gt;, and Caddy for &lt;span class="caps"&gt;TLS&lt;/span&gt;&amp;nbsp;termination.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;CryptPad is an end-to-end encrypted collaboration suite. Think Google Docs, but where the server never sees your content. It&amp;#8217;s a compelling option for privacy-conscious teams or anyone wanting to own their data. This post documents installing CryptPad in a FreeBSD &lt;span class="caps"&gt;VNET&lt;/span&gt; jail, served behind a Caddy reverse proxy, with network isolation enforced by &lt;span class="caps"&gt;PF&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;This guide assumes familiarity with FreeBSD jails, &lt;span class="caps"&gt;PF&lt;/span&gt;, and BastilleBSD. It focuses on architecture and pitfalls rather than introductory jail&amp;nbsp;setup.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Cryptpad" src="https://hofstede.it/images/2026-01-24-cryptpad-freebsd-jail-caddy.png" title="Screenshot of Document in Cryptpad"&gt;&lt;/p&gt;
&lt;h2 id="why-cryptpad"&gt;Why&amp;nbsp;CryptPad?&lt;/h2&gt;
&lt;p&gt;Most collaborative document platforms require trusting the provider with your data. CryptPad takes a different approach: all encryption happens client-side in the browser. The server stores only encrypted blobs and has no way to decrypt your documents, even if compelled&amp;nbsp;to.&lt;/p&gt;
&lt;p&gt;Self-hosting adds another layer: your data never leaves infrastructure you control. Combined with FreeBSD&amp;#8217;s &lt;span class="caps"&gt;VNET&lt;/span&gt; jails for network isolation, this creates a robust privacy-respecting setup. Unlike tools such as Nextcloud or OnlyOffice, CryptPad’s threat model assumes the server itself is&amp;nbsp;untrusted.&lt;/p&gt;
&lt;h2 id="architecture-overview"&gt;Architecture&amp;nbsp;Overview&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[ Internet ]
     |
     v
+-----------------------+
|    PF Firewall        |
|    NAT + RDR          |
+-----------+-----------+
            |
     +------+------+
     | bastille0   |  (bridge interface)
     | 10.254.254.1|
     +------+------+
            |
   +--------+--------+
   |                 |
   v                 v
+-------+      +-----------+
| Caddy |      | CryptPad  |
| Jail  |      |   Jail    |
| .10   |-----&amp;gt;|   .36     |
+-------+      +-----------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The key security principle: jails live on a private &lt;span class="caps"&gt;RFC1918&lt;/span&gt; network&amp;nbsp;(&lt;code&gt;10.254.254.0/24&lt;/code&gt;) that is completely invisible to the internet. &lt;span class="caps"&gt;PF&lt;/span&gt; on the host performs &lt;span class="caps"&gt;NAT&lt;/span&gt; for outbound traffic and only redirects specific ports to specific jails. The CryptPad jail has no direct internet exposure - all traffic flows through the Caddy&amp;nbsp;jail.&lt;/p&gt;
&lt;h2 id="vnet-jail-networking"&gt;&lt;span class="caps"&gt;VNET&lt;/span&gt; Jail&amp;nbsp;Networking&lt;/h2&gt;
&lt;p&gt;This setup uses &lt;span class="caps"&gt;VNET&lt;/span&gt; jails, which give each jail its own complete network stack including interfaces, routing tables, and firewall state. Unlike traditional &lt;span class="caps"&gt;IP&lt;/span&gt;-based jails that share the host&amp;#8217;s network stack, &lt;span class="caps"&gt;VNET&lt;/span&gt; jails behave like independent networked&amp;nbsp;machines.&lt;/p&gt;
&lt;p&gt;The jails attach to a bridge interface on the&amp;nbsp;host:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Host /etc/rc.conf (relevant networking section)&lt;/span&gt;
&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bridge0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.254.254.1/24&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The host acts as the default gateway for all jails. Each jail gets an &lt;span class="caps"&gt;IP&lt;/span&gt; from&amp;nbsp;the &lt;code&gt;10.254.254.0/24&lt;/code&gt; range:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ bastille list
 JID  Name           State  IP Address
 2    caddy          Up     10.254.254.10
 10   cryptpad       Up     10.254.254.36
 ...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Creating the CryptPad jail with&amp;nbsp;Bastille:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-B&lt;span class="w"&gt; &lt;/span&gt;cryptpad&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.0-RELEASE&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.254.254.36&lt;span class="w"&gt; &lt;/span&gt;bastille0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-B&lt;/code&gt; flag creates a &lt;span class="caps"&gt;VNET&lt;/span&gt; jail attached to&amp;nbsp;the &lt;code&gt;bastille0&lt;/code&gt; bridge.&lt;/p&gt;
&lt;h2 id="pf-firewall-nat-and-selective-exposure"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; Firewall: &lt;span class="caps"&gt;NAT&lt;/span&gt; and Selective&amp;nbsp;Exposure&lt;/h2&gt;
&lt;p&gt;The critical security layer is &lt;span class="caps"&gt;PF&lt;/span&gt; on the host. It&amp;nbsp;provides:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt; for outbound traffic&lt;/strong&gt; - jails can reach the internet (for package updates, etc.) but appear to come from the host&amp;#8217;s public &lt;span class="caps"&gt;IP&lt;/span&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Port redirection&lt;/strong&gt; - only ports 80/443 are exposed, and only to the Caddy&amp;nbsp;jail&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Default deny&lt;/strong&gt; - everything else is silently&amp;nbsp;dropped&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# /etc/pf.conf
ext_if = &amp;quot;vtnet0&amp;quot;
jail_net = &amp;quot;10.254.254.0/24&amp;quot;
frontend_v4 = &amp;quot;10.254.254.10&amp;quot;  # Caddy jail

table &amp;lt;jails_v4&amp;gt; { $jail_net }

set skip on lo0
set block-policy drop

# NAT: Jails -&amp;gt; Internet
nat on $ext_if inet from &amp;lt;jails_v4&amp;gt; to any -&amp;gt; ($ext_if)

# RDR: Internet -&amp;gt; Caddy jail only
rdr pass on $ext_if inet proto tcp to ($ext_if) port {80,443} -&amp;gt; $frontend_v4

# Default deny
block drop in log all
block drop out log all

# Allow established connections out
pass out quick all keep state

# Anti-spoofing
antispoof quick for { $ext_if, bastille0 }

# ICMP
pass in inet proto icmp icmp-type { echoreq, unreach }

# Jail egress (but not to other jails)
pass in quick on bastille0 from &amp;lt;jails_v4&amp;gt; to ! 10.254.254.0/24 keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Notice what&amp;#8217;s &lt;strong&gt;not&lt;/strong&gt; here: there&amp;#8217;s no rule allowing direct internet access to the CryptPad jail. Traffic to ports 80/443 is redirected&amp;nbsp;to &lt;code&gt;10.254.254.10&lt;/code&gt; (Caddy), which then proxies to CryptPad internally over the bridge network. The CryptPad jail is completely hidden from the&amp;nbsp;internet.&lt;/p&gt;
&lt;p&gt;This is defense in depth: even if CryptPad had a vulnerability, an attacker couldn&amp;#8217;t establish a reverse shell or exfiltrate data directly - they&amp;#8217;d have to go through Caddy&amp;nbsp;first.&lt;/p&gt;
&lt;h2 id="cryptpad-installation"&gt;CryptPad&amp;nbsp;Installation&lt;/h2&gt;
&lt;p&gt;Two domains are required: one for the main application and one for the sandbox (CryptPad&amp;#8217;s &lt;span class="caps"&gt;XSS&lt;/span&gt; protection mechanism). Both domains resolve to the host&amp;#8217;s public &lt;span class="caps"&gt;IP&lt;/span&gt;, where &lt;span class="caps"&gt;PF&lt;/span&gt; redirects them to&amp;nbsp;Caddy.&lt;/p&gt;
&lt;h2 id="prerequisites"&gt;Prerequisites&lt;/h2&gt;
&lt;p&gt;Inside the CryptPad jail, install the necessary packages. CryptPad requires Node.js 20+ and build tools for native&amp;nbsp;modules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;git&lt;span class="w"&gt; &lt;/span&gt;node24&lt;span class="w"&gt; &lt;/span&gt;npm-node24&lt;span class="w"&gt; &lt;/span&gt;gmake&lt;span class="w"&gt; &lt;/span&gt;python3&lt;span class="w"&gt; &lt;/span&gt;bash
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The full package list for a working installation looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;brotli-1.2.0,1          git-tiny-2.52.0         node24-24.12.0
c-ares-1.34.6           gmake-4.4.1             npm-node24-11.7.0
corepack-0.34.5         icu-76.1,1              python311-3.11.14
libuv-1.51.0            llhttp-9.3.0            simdjson-4.2.4
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="user-and-installation"&gt;User and&amp;nbsp;Installation&lt;/h2&gt;
&lt;p&gt;Never run CryptPad as root. Create a dedicated&amp;nbsp;user:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pw&lt;span class="w"&gt; &lt;/span&gt;useradd&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;cryptpad&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;CryptPad User&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;/usr/local/cryptpad&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;/bin/sh&lt;span class="w"&gt; &lt;/span&gt;-m
su&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;cryptpad
git&lt;span class="w"&gt; &lt;/span&gt;clone&lt;span class="w"&gt; &lt;/span&gt;https://github.com/cryptpad/cryptpad.git&lt;span class="w"&gt; &lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="the-critical-installation-step"&gt;The Critical Installation&amp;nbsp;Step&lt;/h3&gt;
&lt;p&gt;This is where many installations fail. Running&amp;nbsp;only &lt;code&gt;npm install&lt;/code&gt; will leave you with a site that hangs on &amp;#8220;Loading&amp;#8230;&amp;#8221; and throws 404 errors&amp;nbsp;for &lt;code&gt;require.js&lt;/code&gt;. CryptPad needs three distinct build&amp;nbsp;steps:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# 1. Install backend dependencies&lt;/span&gt;
npm&lt;span class="w"&gt; &lt;/span&gt;install

&lt;span class="c1"&gt;# 2. Install frontend components (bower, bootstrap, require.js, etc.)&lt;/span&gt;
npm&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;install:components

&lt;span class="c1"&gt;# 3. Build static assets (pages, CSS, etc.)&lt;/span&gt;
npm&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;build
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The second step is easily missed because it&amp;#8217;s not obvious from the documentation. If you skip it, the application will appear to start but the browser will never finish&amp;nbsp;loading.&lt;/p&gt;
&lt;h2 id="configuration"&gt;Configuration&lt;/h2&gt;
&lt;p&gt;Copy the example configuration and edit&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cp&lt;span class="w"&gt; &lt;/span&gt;config/config.example.js&lt;span class="w"&gt; &lt;/span&gt;config/config.js
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The critical settings&amp;nbsp;in &lt;code&gt;config/config.js&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;module&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;exports&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Listen on the jail&amp;#39;s interface (not localhost)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;httpAddress&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;10.254.254.36&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Main domain - where users access CryptPad&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;httpUnsafeOrigin&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;https://pad.example.com&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Sandbox domain - MUST be different from main domain&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;httpSafeOrigin&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;https://sandbox.example.com&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Admin keys (add after registering your admin account)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;adminKeys&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;// &amp;quot;[your-public-key-here]&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="p"&gt;};&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="understanding-the-two-domains"&gt;Understanding the Two&amp;nbsp;Domains&lt;/h3&gt;
&lt;p&gt;CryptPad&amp;#8217;s security model relies on browser same-origin policy. The main domain handles authentication and cryptographic operations. The sandbox domain loads the &lt;span class="caps"&gt;UI&lt;/span&gt; in an iframe with a different origin, preventing &lt;span class="caps"&gt;XSS&lt;/span&gt; attacks from accessing your encryption&amp;nbsp;keys.&lt;/p&gt;
&lt;p&gt;Both domains point to the same CryptPad instance, the application routes requests appropriately based on the Host header. This is why correctly passing the Host header through the reverse proxy is&amp;nbsp;essential.&lt;/p&gt;
&lt;h2 id="freebsd-service-setup"&gt;FreeBSD Service&amp;nbsp;Setup&lt;/h2&gt;
&lt;p&gt;CryptPad includes an rc.d script template. Copy it and enable the&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cp&lt;span class="w"&gt; &lt;/span&gt;/usr/local/cryptpad/docs/rc.d-cryptpad&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/rc.d/cryptpad
chmod&lt;span class="w"&gt; &lt;/span&gt;+x&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/rc.d/cryptpad
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;cryptpad_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Here&amp;#8217;s the actual rc.d script adapted for&amp;nbsp;FreeBSD:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# PROVIDE: cryptpad&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: DAEMON&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;

.&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.subr

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;cryptpad&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;start_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;start&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;stop_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;stop&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rcvar&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;cryptpad_enable

&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/cryptpad/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;desc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;CryptPad Service&amp;quot;&lt;/span&gt;

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;

start&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/bin/mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/var/run/cryptpad
&lt;span class="w"&gt;    &lt;/span&gt;/usr/sbin/chown&lt;span class="w"&gt; &lt;/span&gt;cryptpad:cryptpad&lt;span class="w"&gt; &lt;/span&gt;/var/run/cryptpad

&lt;span class="w"&gt;    &lt;/span&gt;/usr/bin/su&lt;span class="w"&gt; &lt;/span&gt;cryptpad&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;export PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/usr/local/bin:~/bin &amp;amp;&amp;amp; \&lt;/span&gt;
&lt;span class="s2"&gt;        cd /usr/local/cryptpad &amp;amp;&amp;amp; \&lt;/span&gt;
&lt;span class="s2"&gt;        /usr/sbin/daemon -T &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; \&lt;/span&gt;
&lt;span class="s2"&gt;            -P /var/run/cryptpad/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_supervisor.pid \&lt;/span&gt;
&lt;span class="s2"&gt;            -p /var/run/cryptpad/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid \&lt;/span&gt;
&lt;span class="s2"&gt;            -f -S -r /usr/local/bin/node server&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

stop&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/bin/kill&lt;span class="w"&gt; &lt;/span&gt;-9&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/var/run/cryptpad/&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;_supervisor.pid&lt;span class="sb"&gt;`&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/bin/kill&lt;span class="w"&gt; &lt;/span&gt;-15&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/var/run/cryptpad/&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;.pid&lt;span class="sb"&gt;`&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Start the&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;cryptpad&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A running CryptPad instance spawns multiple worker&amp;nbsp;processes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;USER       PID %CPU %MEM    VSZ   RSS TT  STAT COMMAND
cryptpad 89849  0.0  0.0  13068  2508  -  IsJ  daemon: /usr/local/bin/node[89850]
cryptpad 89850  0.0  1.1 839584 87964  -  SJ   /usr/local/bin/node server
cryptpad 89851  0.0  1.1 840864 88624  -  SJ   /usr/local/bin/node ./lib/http-worker.js
cryptpad 89852  0.0  1.1 842400 88000  -  SJ   /usr/local/bin/node ./lib/http-worker.js
...
cryptpad 89857  0.0  1.0 834088 79448  -  SJ   /usr/local/bin/node lib/workers/db-worker
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="caddy-reverse-proxy"&gt;Caddy Reverse&amp;nbsp;Proxy&lt;/h2&gt;
&lt;p&gt;Caddy runs in its own jail&amp;nbsp;(&lt;code&gt;10.254.254.10&lt;/code&gt;) and is the only jail exposed to the internet via &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s port redirection. Configure it to proxy both CryptPad domains to the application jail. The Host header passthrough is crucial! Without it, CryptPad&amp;#8217;s WebSocket security checks will&amp;nbsp;fail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pad.example.com {
    reverse_proxy 10.254.254.36:3000 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }

    header {
        Strict-Transport-Security &amp;quot;max-age=31536000; includeSubdomains&amp;quot;
        X-XSS-Protection &amp;quot;1; mode=block&amp;quot;
        X-Content-Type-Options &amp;quot;nosniff&amp;quot;
        X-Frame-Options &amp;quot;SAMEORIGIN&amp;quot;
        Referrer-Policy &amp;quot;same-origin&amp;quot;
    }
}

sandbox.example.com {
    reverse_proxy 10.254.254.36:3000 {
        header_up Host {host}
        header_up X-Real-IP {remote}
    }

    header {
        Strict-Transport-Security &amp;quot;max-age=31536000; includeSubdomains&amp;quot;
    }
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note that both domains proxy to the same backend&amp;nbsp;(&lt;code&gt;10.254.254.36:3000&lt;/code&gt;). Caddy handles &lt;span class="caps"&gt;TLS&lt;/span&gt; automatically via Let&amp;#8217;s Encrypt. Since the jails communicate over the private bridge network, this internal traffic doesn&amp;#8217;t need&amp;nbsp;encryption.&lt;/p&gt;
&lt;h2 id="troubleshooting"&gt;Troubleshooting&lt;/h2&gt;
&lt;h3 id="site-hangs-on-loading"&gt;Site Hangs on&amp;nbsp;&amp;#8220;Loading&amp;#8230;&amp;#8221;&lt;/h3&gt;
&lt;p&gt;This is almost always caused by missing frontend components. Check if require.js&amp;nbsp;exists:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ls&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;/usr/local/cryptpad/www/components/requirejs/require.js
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If missing, you&amp;nbsp;skipped &lt;code&gt;npm run install:components&lt;/code&gt;. Fix&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;su&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;cryptpad
&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/local/cryptpad
npm&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;install:components
npm&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;build
&lt;span class="nb"&gt;exit&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;cryptpad&lt;span class="w"&gt; &lt;/span&gt;restart
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="websocket-connection-failures"&gt;WebSocket Connection&amp;nbsp;Failures&lt;/h3&gt;
&lt;p&gt;If you see WebSocket errors in the browser console,&amp;nbsp;verify:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The Host header is being passed through your reverse&amp;nbsp;proxy&lt;/li&gt;
&lt;li&gt;Both domains are correctly configured&amp;nbsp;in &lt;code&gt;config.js&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;The &lt;code&gt;httpAddress&lt;/code&gt; in config.js matches the jail&amp;#8217;s &lt;span class="caps"&gt;IP&lt;/span&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id="white-screen"&gt;White&amp;nbsp;Screen&lt;/h3&gt;
&lt;p&gt;Usually a sandbox domain misconfiguration. The main and sandbox domains must be different but both must resolve to your CryptPad instance. Check browser console for &lt;span class="caps"&gt;CORS&lt;/span&gt; or &lt;span class="caps"&gt;CSP&lt;/span&gt;&amp;nbsp;errors.&lt;/p&gt;
&lt;h2 id="adding-an-admin-account"&gt;Adding an Admin&amp;nbsp;Account&lt;/h2&gt;
&lt;p&gt;After the first successful&amp;nbsp;login:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Register a normal account through the web&amp;nbsp;interface&lt;/li&gt;
&lt;li&gt;Go to Settings and find your public signing&amp;nbsp;key&lt;/li&gt;
&lt;li&gt;Add it to&amp;nbsp;the &lt;code&gt;adminKeys&lt;/code&gt; array&amp;nbsp;in &lt;code&gt;config/config.js&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Restart&amp;nbsp;CryptPad&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;adminKeys&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;[cryptpad-admin@pad.example.com/YZgXQxKR0Rcb6r6CmxHPdAGLVludrAF2lEnkbx1vVOo=]&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;],&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The admin panel then becomes available&amp;nbsp;at &lt;code&gt;https://pad.example.com/admin/&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;CryptPad in a FreeBSD &lt;span class="caps"&gt;VNET&lt;/span&gt; jail provides a solid foundation for privacy-respecting collaboration. The layered security model gives defense in&amp;nbsp;depth:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;VNET&lt;/span&gt; jails&lt;/strong&gt; provide complete network stack&amp;nbsp;isolation&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; &lt;span class="caps"&gt;NAT&lt;/span&gt;&lt;/strong&gt; hides all jails behind &lt;span class="caps"&gt;RFC1918&lt;/span&gt;&amp;nbsp;addresses&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Selective port redirection&lt;/strong&gt; exposes only the Caddy&amp;nbsp;frontend&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;End-to-end encryption&lt;/strong&gt; means even you as administrator cannot read&amp;nbsp;documents&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The main gotcha is the three-step build process - if your installation hangs on loading, that&amp;#8217;s almost certainly the culprit. Get that right, ensure both domains are configured, and you&amp;#8217;ll have a working instance that&amp;#8217;s invisible to port scanners and attackers probing for Node.js&amp;nbsp;vulnerabilities.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.cryptpad.org/"&gt;CryptPad&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/cryptpad/cryptpad"&gt;CryptPad GitHub&amp;nbsp;Repository&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://caddyserver.com/docs/"&gt;Caddy Web&amp;nbsp;Server&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook -&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastillebsd.org/"&gt;BastilleBSD&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.openbsd.org/faq/pf/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; - The OpenBSD Packet&amp;nbsp;Filter&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="jails"/><category term="self-hosting"/><category term="privacy"/></entry><entry><title>Self-Hosting Email on FreeBSD: A Secure, Jailed Setup with Postfix and Dovecot</title><link href="https://hofstede.it/self-hosting-email-on-freebsd-a-secure-jailed-setup-with-postfix-and-dovecot/" rel="alternate"/><published>2026-01-18T00:00:00+01:00</published><updated>2026-01-18T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-01-18:/self-hosting-email-on-freebsd-a-secure-jailed-setup-with-postfix-and-dovecot/</id><summary type="html">&lt;p&gt;A technical deep-dive into running a modern, secure mail server on FreeBSD 15.0 using &lt;span class="caps"&gt;VNET&lt;/span&gt; jails, &lt;span class="caps"&gt;ZFS&lt;/span&gt; encryption, and GeoIP-aware&amp;nbsp;firewalling.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;div class="alert alert-info" role="alert"&gt;
&lt;p&gt;&lt;strong&gt;Update (2026-10-30):&lt;/strong&gt; The &lt;a href="#verification"&gt;Verification&lt;/a&gt; section now also mentions &lt;a href="https://email-spam-tester.com/"&gt;Email Spam Tester&lt;/a&gt;, which adds an optional inbox placement check at Gmail and Outlook to the usual &lt;span class="caps"&gt;SPF&lt;/span&gt;, &lt;span class="caps"&gt;DKIM&lt;/span&gt; and &lt;span class="caps"&gt;DMARC&lt;/span&gt; report. Thanks to Serg Tanichev, its founder, for the&amp;nbsp;suggestion.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;Self-hosting email is often described as a Sisyphean task. A constant battle against spam lists, deliverability issues, and security threats. While there&amp;#8217;s truth to that reputation, running your own mail infrastructure remains one of the best ways to reclaim privacy and truly understand how the internet&amp;#8217;s oldest federation protocol&amp;nbsp;works.&lt;/p&gt;
&lt;p&gt;In this article, I&amp;#8217;ll walk through my current mail server setup running on FreeBSD 15.0. The design philosophy prioritizes security through isolation (&lt;span class="caps"&gt;VNET&lt;/span&gt; jails), data protection (&lt;span class="caps"&gt;ZFS&lt;/span&gt; encryption), and attack surface reduction (GeoIP filtering). This isn&amp;#8217;t a beginner&amp;#8217;s tutorial - I&amp;#8217;ll assume familiarity with FreeBSD basics and focus on the architecture decisions and configuration details that make this setup&amp;nbsp;work.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Mailserver setup" src="https://hofstede.it/images/2026-01-18-mailserver-architecture.png" title="Architecture: VNET Jails for separation of concerns"&gt;&lt;/p&gt;
&lt;h2 id="architecture-overview"&gt;Architecture&amp;nbsp;Overview&lt;/h2&gt;
&lt;p&gt;Before diving into configuration files, here&amp;#8217;s how the pieces fit&amp;nbsp;together:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                        Internet
                            |
                            v
                +-----------------------+
                |     PF Firewall       |
                |  GeoIP + NAT + RDR    |
                +-----------+-----------+
                            |
            +---------------+---------------+
            |                               |
     Port 25 (SMTP)              Ports 143/587/4190
     Open to all                 GeoIP restricted
            |                               |
            v                               v
    +-------+-------------------------------+-------+
    |              VNET Jail: mailstack             |
    |                                               |
    |  +----------+  +----------+  +-----------+    |
    |  | Postfix  |  | Dovecot  |  |  Rspamd   |    |
    |  |   MTA    |  |IMAP/LMTP |  |  Filter   |    |
    |  +----+-----+  +----+-----+  +-----+-----+    |
    |       |             |              |          |
    |       +------+------+------+-------+          |
    |              |                                |
    |              v                                |
    |     +------------------+                      |
    |     | /var/vmail       |  (nullfs mount)      |
    |     | ZFS encrypted    |                      |
    |     +------------------+                      |
    +-----------------------------------------------+
                            |
            +---------------+---------------+
            |                               |
    +-------+-------+               +-------+-------+
    | VNET Jail:    |               |    Host:      |
    |   webmail     |               | zroot/secure  |
    | (Roundcube)   |               | (encrypted)   |
    +---------------+               +---------------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The key insight here is separation of concerns: the host handles networking and storage, while the jails run the actual services. Mail data lives on an encrypted &lt;span class="caps"&gt;ZFS&lt;/span&gt; dataset on the host and is mounted into the jail via nullfs. This means I can destroy and recreate the jail without touching any mail data, and backups of the encrypted dataset can be sent offsite without the backup server ever possessing the decryption&amp;nbsp;key.&lt;/p&gt;
&lt;h2 id="the-host-system-freebsd-150"&gt;The Host System: FreeBSD&amp;nbsp;15.0&lt;/h2&gt;
&lt;p&gt;The foundation is a hardened FreeBSD 15.0 system. Security starts at the kernel&amp;nbsp;level.&lt;/p&gt;
&lt;h3 id="security-hardening"&gt;Security&amp;nbsp;Hardening&lt;/h3&gt;
&lt;p&gt;I run&amp;nbsp;with &lt;code&gt;kern_securelevel="2"&lt;/code&gt;, which is a significant step up from the default. At this level, even the root user cannot write to immutable files, load kernel modules, or modify the firewall rules without a reboot. This means even if an attacker gains root access, they can&amp;#8217;t disable &lt;span class="caps"&gt;PF&lt;/span&gt; or load a&amp;nbsp;rootkit.&lt;/p&gt;
&lt;p&gt;Additional hardening via sysctl prevents users from seeing other processes and restricts access to kernel&amp;nbsp;internals:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/sysctl.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gh"&gt;#&lt;/span&gt; Prevent users from seeing processes of other UIDs
security.bsd.see_other_uids=0
security.bsd.see_other_gids=0
security.bsd.see_jail_proc=0

&lt;span class="gh"&gt;#&lt;/span&gt; Disable unprivileged access to kernel internals
security.bsd.unprivileged_read_msgbuf=0
security.bsd.unprivileged_proc_debug=0

&lt;span class="gh"&gt;#&lt;/span&gt; Randomize PIDs to make race conditions harder to exploit
kern.randompid=1

&lt;span class="gh"&gt;#&lt;/span&gt; ZFS tuning
vfs.zfs.vdev.min_auto_ashift=12

&lt;span class="gh"&gt;#&lt;/span&gt; Allow large PF tables for GeoIP filtering
net.pf.request_maxcount=500000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The last setting is crucial for GeoIP filtering - the default &lt;span class="caps"&gt;PF&lt;/span&gt; table size is too small to hold the ~273,000 &lt;span class="caps"&gt;CIDR&lt;/span&gt; blocks we&amp;#8217;ll be&amp;nbsp;loading.&lt;/p&gt;
&lt;h3 id="kernel-module-loading"&gt;Kernel Module&amp;nbsp;Loading&lt;/h3&gt;
&lt;p&gt;The necessary modules for jail networking are loaded at&amp;nbsp;boot:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/boot/loader.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;kern&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;geom&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;disk_ident&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;kern&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;geom&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;gptid&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;zfs_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Required for VNET jails&lt;/span&gt;
&lt;span class="n"&gt;nullfs_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;if_bridge_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;if_epair_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;kern&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;racct&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="network-and-zfs-layout"&gt;Network and &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;nbsp;Layout&lt;/h3&gt;
&lt;p&gt;The host handles the physical connection and bridges traffic to the jails. The network configuration uses a bridge interface that provides connectivity to all &lt;span class="caps"&gt;VNET&lt;/span&gt;&amp;nbsp;jails:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/rc.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;hostname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mail&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;keymap&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;us.kbd&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Security hardening&lt;/span&gt;
&lt;span class="nv"&gt;kern_securelevel_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;kern_securelevel&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# External Interface (VPS provider network)&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 192.0.2.50/32 -lro -tso&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:1c1c:4d2::1/65&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;172.31.1.1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fe80::1%vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Bridge for VNET Jails&lt;/span&gt;
&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bridge0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.0.0.1 netmask 255.255.255.0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:1c1c:4d2:8000::1/65&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Enable routing for jails&lt;/span&gt;
&lt;span class="nv"&gt;gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Services&lt;/span&gt;
&lt;span class="nv"&gt;dumpdev&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;jail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sshd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pflog_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;clear_tmp_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;moused_nondefault_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;ZFS&lt;/span&gt; layout separates sensitive data onto an encrypted dataset. Here&amp;#8217;s the relevant&amp;nbsp;portion:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;NAME                          USED  AVAIL  REFER  MOUNTPOINT
zroot/secure                 4.28G  29.2G   200K  /zroot/secure
zroot/secure/jails           4.04G  29.2G   264K  /jails
zroot/secure/jails/mailstack 1.40G  29.2G  1.00G  /jails/mailstack
zroot/secure/jails/webmail   1022M  29.2G   805M  /jails/webmail
zroot/secure/vmail            248M  29.2G   233M  /var/mail
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The encryption uses &lt;span class="caps"&gt;AES&lt;/span&gt;-256-&lt;span class="caps"&gt;GCM&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;encryption&lt;span class="w"&gt; &lt;/span&gt;zroot/secure
NAME&lt;span class="w"&gt;          &lt;/span&gt;PROPERTY&lt;span class="w"&gt;    &lt;/span&gt;VALUE&lt;span class="w"&gt;        &lt;/span&gt;SOURCE
zroot/secure&lt;span class="w"&gt;  &lt;/span&gt;encryption&lt;span class="w"&gt;  &lt;/span&gt;aes-256-gcm&lt;span class="w"&gt;  &lt;/span&gt;-
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This means all child datasets (jails and mail storage) inherit the encryption. After a reboot, the dataset must be manually unlocked before the jails can start - a deliberate trade-off between availability and&amp;nbsp;security.&lt;/p&gt;
&lt;h2 id="the-firewall-pf-with-geoip-filtering"&gt;The Firewall: &lt;span class="caps"&gt;PF&lt;/span&gt; with GeoIP&amp;nbsp;Filtering&lt;/h2&gt;
&lt;p&gt;I use &lt;span class="caps"&gt;PF&lt;/span&gt; (Packet Filter) to ruthlessly cut down on attack surface. The most effective measure has been GeoIP filtering. Since my users are all in Central Europe, I can block authentication attempts from everywhere else while keeping &lt;span class="caps"&gt;SMTP&lt;/span&gt; open for mail&amp;nbsp;delivery.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/pf.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# --- Macros ---&lt;/span&gt;
&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.0.0.0/24&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1c1c:4d2:8000::/65&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1c1c:4d2::1&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;mail_ipv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.0.0.3&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;mail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1c1c:4d2:8000::3&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;webmail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1c1c:4d2:8000::4&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# --- Tables ---&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;geoip_users&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;

&lt;span class="c1"&gt;# --- Options ---&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;limit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;entries&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1000000&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;skip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lo0&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loginterface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;

&lt;span class="c1"&gt;# --- Scrub ---&lt;/span&gt;
&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fragment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;reassemble&lt;/span&gt;
&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;

&lt;span class="c1"&gt;# --- NAT (IPv4 only for legacy support) ---&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- RDR for services ---&lt;/span&gt;
&lt;span class="c1"&gt;# Client ports: GeoIP restricted&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;geoip_users&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;143&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;587&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4190&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;mail_ipv4&lt;/span&gt;

&lt;span class="c1"&gt;# SMTP: Open to all (server-to-server)&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;mail_ipv4&lt;/span&gt;

&lt;span class="c1"&gt;# --- Filtering ---&lt;/span&gt;
&lt;span class="c1"&gt;# Block known bad actors immediately&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;

&lt;span class="c1"&gt;# Default deny&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;

&lt;span class="c1"&gt;# Allow all outbound (stateful)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Anti-spoofing&lt;/span&gt;
&lt;span class="n"&gt;antispoof&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bridge0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# SSH only from GeoIP restricted IP ranges&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;geoip_users&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Block and log all other SSH attempts&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh_not_trusted&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# IPv6: Mail client ports with GeoIP restriction&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;geoip_users&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;mail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;143&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;587&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4190&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# IPv6: SMTP open to all&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;mail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# IPv6: Webmail with GeoIP restriction&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;geoip_users&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;webmail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Essential ICMPv6&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echorep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbrsol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbradv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;toobig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;timex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paramprob&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# ICMP for IPv4 (ping, MTU discovery)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unreach&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Jail egress&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bridge0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;10.0&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;0.0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bridge0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;jails_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="n"&gt;c1c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;d2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The dual approach to traffic handling is visible in the&amp;nbsp;rules:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;SMTP&lt;/span&gt; (port 25)&lt;/strong&gt;: &lt;code&gt;from any&lt;/code&gt; - must be globally accessible for mail&amp;nbsp;delivery&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Client ports (143, 587, 4190)&lt;/strong&gt;: &lt;code&gt;from &amp;lt;geoip_users&amp;gt;&lt;/code&gt; - restricted to allowed&amp;nbsp;countries&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I covered the GeoIP update script in detail in my &lt;a href="https://hofstede.it/geoip-aware-firewalling-with-pf-on-freebsd/"&gt;previous article on GeoIP-aware firewalling&lt;/a&gt;. The short version: I parse MaxMind&amp;#8217;s GeoLite2 &lt;span class="caps"&gt;CSV&lt;/span&gt; data for my target countries (&lt;span class="caps"&gt;DE&lt;/span&gt;, &lt;span class="caps"&gt;AT&lt;/span&gt;, &lt;span class="caps"&gt;CH&lt;/span&gt;, &lt;span class="caps"&gt;NL&lt;/span&gt;, &lt;span class="caps"&gt;LU&lt;/span&gt;, &lt;span class="caps"&gt;FR&lt;/span&gt;) and load the resulting ~273,000 &lt;span class="caps"&gt;CIDR&lt;/span&gt; blocks into &lt;span class="caps"&gt;PF&lt;/span&gt; in chunks to avoid memory&amp;nbsp;exhaustion.&lt;/p&gt;
&lt;h2 id="jail-architecture-vnet-for-true-isolation"&gt;Jail Architecture: &lt;span class="caps"&gt;VNET&lt;/span&gt; for True&amp;nbsp;Isolation&lt;/h2&gt;
&lt;p&gt;I use &lt;span class="caps"&gt;VNET&lt;/span&gt; jails rather than traditional &lt;span class="caps"&gt;IP&lt;/span&gt; alias jails. Unlike &lt;span class="caps"&gt;IP&lt;/span&gt; aliases, &lt;span class="caps"&gt;VNET&lt;/span&gt; gives each jail its own fully virtualized network stack with dedicated interfaces. This allows firewall rules inside the jail if needed and keeps the host&amp;#8217;s networking&amp;nbsp;clean.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/jail.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;mailstack&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;enforce_statfs&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;devfs_ruleset&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;consolelog&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;jails&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mailstack_console&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/bin/sh /etc/rc&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/bin/sh /etc/rc.shutdown&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hostname&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mailstack&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;devfs&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/jails/mailstack&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Mount the mail storage from the host&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/mail /jails/mailstack/var/vmail nullfs rw 0 0&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# VNET configuration&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;vnet&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;vnet&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;e0b_mailstack&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;epair0=$(ifconfig epair create) &amp;amp;&amp;amp; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="s2"&gt;        ifconfig ${epair0} up name e0a_mailstack &amp;amp;&amp;amp; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="s2"&gt;        ifconfig ${epair0%a}b up name e0b_mailstack&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig bridge0 addm e0a_mailstack&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig e0a_mailstack description &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="s2"&gt;        &lt;/span&gt;&lt;span class="se"&gt;\&amp;quot;&lt;/span&gt;&lt;span class="s2"&gt;vnet0 host interface for Jail mailstack&lt;/span&gt;&lt;span class="se"&gt;\&amp;quot;&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;poststop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig e0a_mailstack destroy&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;webmail&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;enforce_statfs&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;devfs_ruleset&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clean&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;consolelog&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;jails&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;webmail_console&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/bin/sh /etc/rc&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;stop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;/bin/sh /etc/rc.shutdown&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;host&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;hostname&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;webmail&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;mount&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;devfs&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/jails/webmail&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;vnet&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;vnet&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;interface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;e0b_webmail&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;epair0=$(ifconfig epair create) &amp;amp;&amp;amp; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="s2"&gt;        ifconfig ${epair0} up name e0a_webmail &amp;amp;&amp;amp; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="s2"&gt;        ifconfig ${epair0%a}b up name e0b_webmail&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig bridge0 addm e0a_webmail&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;prestart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig e0a_webmail description &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="s2"&gt;        &lt;/span&gt;&lt;span class="se"&gt;\&amp;quot;&lt;/span&gt;&lt;span class="s2"&gt;vnet0 host interface for Jail webmail&lt;/span&gt;&lt;span class="se"&gt;\&amp;quot;&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;exec&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;poststop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ifconfig e0a_webmail destroy&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;To ensure the jails have only access to the /dev device nodes that they need, we need to have a devfs rule with the same id in /etc/devfs.rules and re-start the
devfs servive&amp;nbsp;with &lt;code&gt;service devfs restart&lt;/code&gt;. This is good practice for isolation of the&amp;nbsp;jails.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/devfs.rules&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[jail_vnet=4]&lt;/span&gt;
&lt;span class="na"&gt;add include \$devfsrules_hide_all&lt;/span&gt;
&lt;span class="na"&gt;add include \$devfsrules_unhide_basic&lt;/span&gt;
&lt;span class="na"&gt;add include \$devfsrules_unhide_login&lt;/span&gt;
&lt;span class="na"&gt;add include \$devfsrules_jail&lt;/span&gt;
&lt;span class="na"&gt;add include \$devfsrules_jail_vnet&lt;/span&gt;
&lt;span class="na"&gt;add path &amp;#39;bpf*&amp;#39; unhide&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The epair interfaces create a virtual cable - one end attached to the host&amp;#8217;s bridge, the other inside the jail. When the jail stops, the interface is automatically&amp;nbsp;destroyed.&lt;/p&gt;
&lt;p&gt;Inside the mailstack jail, the network is configured like any FreeBSD&amp;nbsp;system:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/jails/mailstack/etc/rc.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;ifconfig_e0b_mailstack_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vnet0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.0.0.3 netmask 255.255.255.0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:1c1c:4d2:8000::3/65&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_descr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;jail interface for bridge0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.0.0.1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:1c1c:4d2:8000::1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Disable sendmail (we use Postfix)&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_submit_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_outbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_msp_queue_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Services&lt;/span&gt;
&lt;span class="nv"&gt;syslogd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-ss&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;cron_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-J 60&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;postfix_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;dovecot_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rspamd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;local_unbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note that I run a local Unbound resolver inside the jail. This ensures &lt;span class="caps"&gt;DNS&lt;/span&gt; queries for &lt;span class="caps"&gt;DKIM&lt;/span&gt; verification and &lt;span class="caps"&gt;SPF&lt;/span&gt; checks don&amp;#8217;t leak to external resolvers and provides caching for the many lookups mail processing&amp;nbsp;requires.&lt;/p&gt;
&lt;h2 id="the-mail-stack-configuration"&gt;The Mail Stack&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Inside the mailstack jail, three components work together: Postfix handles &lt;span class="caps"&gt;SMTP&lt;/span&gt;, Dovecot manages storage and &lt;span class="caps"&gt;IMAP&lt;/span&gt; access, and Rspamd filters spam and signs outgoing mail with &lt;span class="caps"&gt;DKIM&lt;/span&gt;.&lt;/p&gt;
&lt;h3 id="postfix-the-mta"&gt;Postfix (The &lt;span class="caps"&gt;MTA&lt;/span&gt;)&lt;/h3&gt;
&lt;p&gt;Postfix handles both receiving mail from the internet and accepting submissions from authenticated users. The configuration enforces strict restrictions to reject spam early in the &lt;span class="caps"&gt;SMTP&lt;/span&gt;&amp;nbsp;conversation.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/usr/local/etc/postfix/main.cf&lt;/strong&gt; (key&amp;nbsp;settings)&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;compatibility_level&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m m-Double"&gt;3.10&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Identity&lt;/span&gt;
&lt;span class="nx"&gt;myhostname&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;mail&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;org&lt;/span&gt;
&lt;span class="nx"&gt;myorigin&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;myhostname&lt;/span&gt;
&lt;span class="nx"&gt;mydestination&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;localhost&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;mydomain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;localhost&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Network&lt;/span&gt;
&lt;span class="nx"&gt;inet_interfaces&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;
&lt;span class="nx"&gt;inet_protocols&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;
&lt;span class="nx"&gt;mynetworks_style&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;host&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;TLS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;Incoming&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Outgoing&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_tls_cert_file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;mail&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;certs&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;cert&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;pem&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_tls_key_file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;mail&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;certs&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;pem&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_tls_security_level&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;may&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_tls_auth_only&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;yes&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_tls_loglevel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="nx"&gt;smtp_tls_security_level&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;may&lt;/span&gt;
&lt;span class="nx"&gt;smtp_tls_loglevel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;span class="nx"&gt;smtp_tls_CApath&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;ssl&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;certs&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;SASL&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;Auth&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;via&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Dovecot&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_sasl_type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dovecot&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_sasl_path&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;private&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;auth&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_sasl_auth_enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;yes&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_sasl_security_options&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;noanonymous&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Virtual&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Domains&lt;/span&gt;
&lt;span class="nx"&gt;virtual_mailbox_domains&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;org&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;net&lt;/span&gt;
&lt;span class="nx"&gt;virtual_mailbox_maps&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hash&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;postfix&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;vmailbox&lt;/span&gt;
&lt;span class="nx"&gt;virtual_alias_maps&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hash&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;postfix&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="kd"&gt;virtual&lt;/span&gt;
&lt;span class="nx"&gt;virtual_uid_maps&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;static&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;
&lt;span class="nx"&gt;virtual_gid_maps&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;static&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;
&lt;span class="nx"&gt;virtual_minimum_uid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Hand&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;off&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Dovecot&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;local&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;delivery&lt;/span&gt;
&lt;span class="nx"&gt;virtual_transport&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;lmtp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;unix&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;private&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;dovecot&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;lmtp&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Rspamd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;integration&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;via&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;milter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;protocol&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_milters&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;localhost&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;11332&lt;/span&gt;
&lt;span class="nx"&gt;non_smtpd_milters&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;localhost&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;11332&lt;/span&gt;
&lt;span class="nx"&gt;milter_protocol&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;
&lt;span class="nx"&gt;milter_default_action&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Strict&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;HELO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;requirements&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_helo_required&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;yes&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_helo_restrictions&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;permit_mynetworks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;permit_sasl_authenticated&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;reject_invalid_helo_hostname&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;reject_non_fqdn_helo_hostname&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;permit&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Sender&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;restrictions&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_sender_restrictions&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;permit_mynetworks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;permit_sasl_authenticated&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;reject_non_fqdn_sender&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;reject_unknown_sender_domain&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;permit&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Recipient&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;restrictions&lt;/span&gt;
&lt;span class="nx"&gt;smtpd_recipient_restrictions&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;permit_mynetworks&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;permit_sasl_authenticated&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;reject_unauth_destination&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;reject_non_fqdn_recipient&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;reject_unknown_reverse_client_hostname&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The submission port (587) for authenticated users is configured&amp;nbsp;in &lt;code&gt;master.cf&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/usr/local/etc/postfix/master.cf&lt;/strong&gt; (submission&amp;nbsp;entry)&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;submission inet n       -       n       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_tls_auth_only=yes
  -o smtpd_milters=inet:localhost:11332
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The virtual mailbox and alias maps define which addresses are valid and where they&amp;#8217;re&amp;nbsp;delivered:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/usr/local/etc/postfix/vmailbox&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# example.org
admin@example.org        OK
backups@example.org      OK

# example.net
user@example.net         OK
accounts@example.net     OK
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;/usr/local/etc/postfix/virtual&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Aliases for example.org
postmaster@example.org   admin@example.org
abuse@example.org        admin@example.org
webmaster@example.org    admin@example.org

# Catch-all for specific services -&amp;gt; accounts
github@example.net       accounts@example.net
hetzner@example.net      accounts@example.net
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After modifying these files, rebuild the hash&amp;nbsp;databases:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;postmap&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/postfix/vmailbox
postmap&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/postfix/virtual
postfix&lt;span class="w"&gt; &lt;/span&gt;reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="dovecot-storage-authentication"&gt;Dovecot (Storage &lt;span class="amp"&gt;&amp;amp;&lt;/span&gt;&amp;nbsp;Authentication)&lt;/h3&gt;
&lt;p&gt;Dovecot is the workhorse. It serves &lt;span class="caps"&gt;IMAP&lt;/span&gt; to clients, handles authentication for Postfix via &lt;span class="caps"&gt;SASL&lt;/span&gt;, receives mail from Postfix via &lt;span class="caps"&gt;LMTP&lt;/span&gt;, and processes Sieve filter rules.
The special_use attributes implement &lt;span class="caps"&gt;RFC&lt;/span&gt; 6154. This signals to modern clients (iOS, Outlook) which folders are for Sent, Trash, and Drafts, preventing the chaos of mixed
&amp;#8216;Sent&amp;#8217; and &amp;#8216;Sent Messages&amp;#8217;&amp;nbsp;folders.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/usr/local/etc/dovecot/dovecot.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;protocols&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;imap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lmtp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sieve&lt;/span&gt;
&lt;span class="n"&gt;auth_mechanisms&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;plain&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;login&lt;/span&gt;

&lt;span class="c1"&gt;# Security - require TLS&lt;/span&gt;
&lt;span class="n"&gt;ssl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;required&lt;/span&gt;
&lt;span class="n"&gt;ssl_cert&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;certs&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;cert&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pem&lt;/span&gt;
&lt;span class="n"&gt;ssl_key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;/&lt;/span&gt;&lt;span class="n"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mail&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;certs&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pem&lt;/span&gt;
&lt;span class="n"&gt;ssl_min_protocol&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;TLSv1&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;

&lt;span class="c1"&gt;# Mail storage location&lt;/span&gt;
&lt;span class="n"&gt;mail_location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;maildir&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;vmail&lt;/span&gt;&lt;span class="o"&gt;/%&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;/%&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;Maildir&lt;/span&gt;
&lt;span class="n"&gt;mail_uid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;
&lt;span class="n"&gt;mail_gid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;
&lt;span class="n"&gt;mmap_disable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;

&lt;span class="c1"&gt;# Sieve for server-side filtering&lt;/span&gt;
&lt;span class="n"&gt;plugin&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;sieve&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;vmail&lt;/span&gt;&lt;span class="o"&gt;/%&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;/%&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/.&lt;/span&gt;&lt;span class="n"&gt;dovecot&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sieve&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;sieve_dir&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;vmail&lt;/span&gt;&lt;span class="o"&gt;/%&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;/%&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sieve&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Authentication (simple passwd-file for easy management)&lt;/span&gt;
&lt;span class="n"&gt;passdb&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;driver&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;passwd&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;file&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;scheme&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;BLF&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;CRYPT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;username_format&lt;/span&gt;&lt;span class="o"&gt;=%&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dovecot&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;users&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;userdb&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;driver&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;static&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;args&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;uid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;gid&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;5000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;home&lt;/span&gt;&lt;span class="o"&gt;=/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;vmail&lt;/span&gt;&lt;span class="o"&gt;/%&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="o"&gt;/%&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Socket for Postfix SASL authentication&lt;/span&gt;
&lt;span class="n"&gt;service&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;auth&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;unix_listener&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;spool&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;postfix&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;private&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;auth&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0660&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;postfix&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;postfix&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Socket for Postfix LMTP delivery&lt;/span&gt;
&lt;span class="n"&gt;service&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lmtp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;unix_listener&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;spool&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;postfix&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;private&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dovecot&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;lmtp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;mode&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0660&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;user&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;postfix&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;group&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;postfix&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;protocol&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lmtp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;mail_plugins&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;mail_plugins&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sieve&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Standard IMAP folders (RFC 6154)&lt;/span&gt;
&lt;span class="n"&gt;namespace&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inbox&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;inbox&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;yes&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;separator&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;mailbox&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Drafts&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;special_use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\&lt;span class="n"&gt;Drafts&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;subscribe&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;mailbox&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Junk&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;special_use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\&lt;span class="n"&gt;Junk&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;subscribe&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;mailbox&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Trash&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;special_use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\&lt;span class="n"&gt;Trash&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;subscribe&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;mailbox&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Sent&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;special_use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\&lt;span class="n"&gt;Sent&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;subscribe&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;mailbox&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Sent Messages&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;special_use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\&lt;span class="n"&gt;Sent&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;mailbox&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Archive&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;special_use&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\&lt;span class="n"&gt;Archive&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;auto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;subscribe&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The users file contains password hashes (generated&amp;nbsp;with &lt;code&gt;doveadm pw -s BLF-CRYPT&lt;/code&gt;):&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/usr/local/etc/dovecot/users&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;user@example.net:{BLF-CRYPT}$2y$05$...hash...
admin@example.org:{BLF-CRYPT}$2y$05$...hash...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="rspamd-the-gatekeeper"&gt;Rspamd (The&amp;nbsp;Gatekeeper)&lt;/h3&gt;
&lt;p&gt;Rspamd analyzes incoming messages and handles &lt;span class="caps"&gt;DKIM&lt;/span&gt; signing for outgoing mail. Without valid &lt;span class="caps"&gt;DKIM&lt;/span&gt; signatures, your mail will likely land in Gmail&amp;#8217;s spam&amp;nbsp;folder.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/usr/local/etc/rspamd/local.d/dkim_signing.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;path&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;/var/lib/rspamd/dkim/$domain.key&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;selector&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;mail&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="n"&gt;allow_username_mismatch&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Generate &lt;span class="caps"&gt;DKIM&lt;/span&gt; keys for each&amp;nbsp;domain:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;rspamadm&lt;span class="w"&gt; &lt;/span&gt;dkim_keygen&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;mail&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;example.org&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-k&lt;span class="w"&gt; &lt;/span&gt;/var/lib/rspamd/dkim/example.org.key&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;/var/lib/rspamd/dkim/example.org.pub
chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;640&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/var/lib/rspamd/dkim/example.org.key
chown&lt;span class="w"&gt; &lt;/span&gt;rspamd:rspamd&lt;span class="w"&gt; &lt;/span&gt;/var/lib/rspamd/dkim/example.org.key
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The public key goes into your &lt;span class="caps"&gt;DNS&lt;/span&gt; as a &lt;span class="caps"&gt;TXT&lt;/span&gt; record&amp;nbsp;at &lt;code&gt;mail._domainkey.example.org&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="boot-and-recovery-procedure"&gt;Boot and Recovery&amp;nbsp;Procedure&lt;/h2&gt;
&lt;p&gt;Because the mail data lives on an encrypted &lt;span class="caps"&gt;ZFS&lt;/span&gt; dataset, the system requires manual intervention after a reboot. This is a deliberate security&amp;nbsp;trade-off.&lt;/p&gt;
&lt;p&gt;After the system&amp;nbsp;boots:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Unlock the encrypted dataset&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;load-key&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;zroot/secure

&lt;span class="c1"&gt;# Mount all ZFS filesystems&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;mount&lt;span class="w"&gt; &lt;/span&gt;-a

&lt;span class="c1"&gt;# Start the jails&lt;/span&gt;
service&lt;span class="w"&gt; &lt;/span&gt;jail&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This sequence ensures that even if someone gains physical access to the server (or its disks), they cannot read the mail data without the encryption&amp;nbsp;passphrase.&lt;/p&gt;
&lt;h2 id="backups-encrypted-at-rest"&gt;Backups: Encrypted at&amp;nbsp;Rest&lt;/h2&gt;
&lt;p&gt;One of &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;#8217;s killer features for mail hosting is the ability to send encrypted raw streams. The backup server never sees unencrypted&amp;nbsp;data.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a recursive snapshot&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;zroot@backup_20260118

&lt;span class="c1"&gt;# Send encrypted stream to offsite backup&lt;/span&gt;
&lt;span class="c1"&gt;# The -w flag sends the raw encrypted blocks&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;-R&lt;span class="w"&gt; &lt;/span&gt;-w&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;@backup_20260117&lt;span class="w"&gt; &lt;/span&gt;zroot@backup_20260118&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;backup-user@backup.example.net&lt;span class="w"&gt; &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;recv&lt;span class="w"&gt; &lt;/span&gt;-F&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;mountpoint&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;none&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;canmount&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;off&lt;span class="w"&gt; &lt;/span&gt;data1/mailserver
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The backup server stores the data in encrypted form. Even if it&amp;#8217;s compromised, the attacker only gets encrypted blocks without the&amp;nbsp;key.&lt;/p&gt;
&lt;h2 id="verification"&gt;Verification&lt;/h2&gt;
&lt;p&gt;After everything is configured, verify the running services inside the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root@mailstack:/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# ps auxw | grep -E &amp;#39;postfix|dovecot|rspamd&amp;#39;&lt;/span&gt;
root&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="m"&gt;16095&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.3&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;130672&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;53916&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;SsJ&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:00.10&lt;span class="w"&gt; &lt;/span&gt;rspamd:&lt;span class="w"&gt; &lt;/span&gt;main&lt;span class="w"&gt; &lt;/span&gt;process
rspamd&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;16324&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.3&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;131056&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;54692&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;SJ&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:00.12&lt;span class="w"&gt; &lt;/span&gt;rspamd:&lt;span class="w"&gt; &lt;/span&gt;rspamd_proxy&lt;span class="w"&gt; &lt;/span&gt;process
rspamd&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;16521&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;131964&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;56244&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;SJ&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:01.09&lt;span class="w"&gt; &lt;/span&gt;rspamd:&lt;span class="w"&gt; &lt;/span&gt;controller&lt;span class="w"&gt; &lt;/span&gt;process
rspamd&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;16619&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;132592&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;58544&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;SJ&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:00.58&lt;span class="w"&gt; &lt;/span&gt;rspamd:&lt;span class="w"&gt; &lt;/span&gt;normal&lt;span class="w"&gt; &lt;/span&gt;process
root&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="m"&gt;35467&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.1&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;15984&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;4104&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;IsJ&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:00.15&lt;span class="w"&gt; &lt;/span&gt;/usr/local/sbin/dovecot
dovecot&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;36239&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.1&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;15916&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;3832&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;IJ&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:00.03&lt;span class="w"&gt; &lt;/span&gt;dovecot/anvil
root&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="m"&gt;69385&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.4&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;61164&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;14668&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;IsJ&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:00.11&lt;span class="w"&gt; &lt;/span&gt;/usr/local/libexec/postfix/master
postfix&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;69918&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.4&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;61188&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;14676&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;IJ&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:00.03&lt;span class="w"&gt; &lt;/span&gt;pickup&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;unix&lt;span class="w"&gt; &lt;/span&gt;-u
postfix&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;70681&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.4&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;61244&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;14744&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-&lt;span class="w"&gt;  &lt;/span&gt;IJ&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;13&lt;/span&gt;:21&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:00.03&lt;span class="w"&gt; &lt;/span&gt;qmgr&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;unix&lt;span class="w"&gt; &lt;/span&gt;-u
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Test &lt;span class="caps"&gt;SMTP&lt;/span&gt;&amp;nbsp;connectivity:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;telnet&lt;span class="w"&gt; &lt;/span&gt;mail.example.org&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;25&lt;/span&gt;
&lt;span class="m"&gt;220&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;mail.example.org&lt;span class="w"&gt; &lt;/span&gt;ESMTP&lt;span class="w"&gt; &lt;/span&gt;Postfix
EHLO&lt;span class="w"&gt; &lt;/span&gt;test.example.com
&lt;span class="m"&gt;250&lt;/span&gt;-mail.example.org
&lt;span class="m"&gt;250&lt;/span&gt;-STARTTLS
&lt;span class="m"&gt;250&lt;/span&gt;-AUTH&lt;span class="w"&gt; &lt;/span&gt;PLAIN&lt;span class="w"&gt; &lt;/span&gt;LOGIN
...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Test email deliverability using tools like &lt;a href="https://www.mail-tester.com/"&gt;mail-tester.com&lt;/a&gt; or &lt;a href="https://mxtoolbox.com/"&gt;MXToolbox&lt;/a&gt;. A properly configured server with &lt;span class="caps"&gt;SPF&lt;/span&gt;, &lt;span class="caps"&gt;DKIM&lt;/span&gt;, and &lt;span class="caps"&gt;DMARC&lt;/span&gt; should score&amp;nbsp;10/10.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://email-spam-tester.com/"&gt;Email Spam Tester&lt;/a&gt; works the same way as mail-tester.com: you send one real message to a disposable address and get a report covering &lt;span class="caps"&gt;SPF&lt;/span&gt;, &lt;span class="caps"&gt;DKIM&lt;/span&gt;, &lt;span class="caps"&gt;DMARC&lt;/span&gt; and blocklists, without signing up. It also offers an optional inbox placement test that shows whether the message lands in the inbox or the spam folder at Gmail, Outlook and other providers, which an authentication score alone can&amp;#8217;t tell&amp;nbsp;you.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Running a mail server in 2026 on FreeBSD 15.0 feels robust. The combination of &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s fine-grained control, &lt;span class="caps"&gt;VNET&lt;/span&gt; jails for network isolation, &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;#8217;s native encryption, and GeoIP filtering creates a platform that is secure by&amp;nbsp;design.&lt;/p&gt;
&lt;p&gt;While it requires initial effort to configure correctly - especially getting &lt;span class="caps"&gt;DNS&lt;/span&gt; records (&lt;span class="caps"&gt;SPF&lt;/span&gt;, &lt;span class="caps"&gt;DKIM&lt;/span&gt;, &lt;span class="caps"&gt;DMARC&lt;/span&gt;) aligned - the result is a private, secure communication hub that you truly own. You aren&amp;#8217;t mining your own data for ads, and you aren&amp;#8217;t subject to the whims of a provider who might lock your account without&amp;nbsp;recourse.&lt;/p&gt;
&lt;p&gt;The architecture also makes maintenance straightforward: update the host and jails independently, restore from encrypted backups without exposing plaintext data, and adjust GeoIP rules as your travel patterns&amp;nbsp;change.&lt;/p&gt;
&lt;p&gt;Is it more work than using a hosted provider? Absolutely. Is it worth it? For those who value understanding their infrastructure and maintaining control over their communications, unquestionably&amp;nbsp;yes.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.postfix.org/documentation.html"&gt;Postfix&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://doc.dovecot.org/"&gt;Dovecot&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://rspamd.com/doc/"&gt;Rspamd&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook:&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pf.conf"&gt;FreeBSD pf.conf(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://dev.maxmind.com/geoip/geolite2-free-geolocation-data"&gt;MaxMind GeoLite2 Free Geolocation&amp;nbsp;Data&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://openzfs.github.io/openzfs-docs/Getting%20Started/Ubuntu/Ubuntu%2020.04%20Root%20on%20ZFS.html#encryption"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; Encryption (OpenZFS&amp;nbsp;Documentation)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;The mail protocols we use today were designed in an era of mutual trust between operators. That trust has eroded, but the protocols remain remarkably resilient. With careful configuration and modern security layers, self-hosted email is not just viable - it&amp;#8217;s a statement of digital&amp;nbsp;autonomy.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="mail"/><category term="postfix"/><category term="dovecot"/><category term="rspamd"/><category term="jails"/><category term="zfs"/><category term="security"/></entry><entry><title>GeoIP-Aware Firewalling with PF on FreeBSD</title><link href="https://hofstede.it/geoip-aware-firewalling-with-pf-on-freebsd/" rel="alternate"/><published>2026-01-13T00:00:00+01:00</published><updated>2026-01-13T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-01-13:/geoip-aware-firewalling-with-pf-on-freebsd/</id><summary type="html">&lt;p&gt;Using MaxMind&amp;#8217;s GeoLite2 database with FreeBSD&amp;#8217;s &lt;span class="caps"&gt;PF&lt;/span&gt; firewall to restrict client-facing services to specific countries, reducing brute-force attempts and log noise while keeping essential services globally&amp;nbsp;accessible.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;Running a mail server on the public internet means dealing with a constant stream of brute-force attempts. Credential stuffers, password sprayers, and opportunistic bots hammer away at &lt;span class="caps"&gt;IMAP&lt;/span&gt;, submission ports, and webmail interfaces around the clock. While fail2ban-style rate limiting helps, the sheer volume of attempts still clutters logs and wastes resources on connection&amp;nbsp;handling.&lt;/p&gt;
&lt;p&gt;The solution I&amp;#8217;ve implemented takes a different approach: geographic restriction. My users are all located in Central Europe - Germany, Austria, Switzerland, and neighboring countries. There&amp;#8217;s no legitimate reason for someone in a botnet-heavy region to connect to my &lt;span class="caps"&gt;IMAP&lt;/span&gt; server. By restricting client-facing ports to &lt;span class="caps"&gt;IP&lt;/span&gt; ranges allocated to specific countries, I&amp;#8217;ve dramatically reduced both attack surface and log&amp;nbsp;noise.&lt;/p&gt;
&lt;p&gt;&lt;img alt="World map" src="https://hofstede.it/images/2026-01-13-freebsd-geoip-firewall.jpg" title="Geographic filtering: allowing only what you need"&gt;&lt;/p&gt;
&lt;h2 id="the-strategy"&gt;The&amp;nbsp;Strategy&lt;/h2&gt;
&lt;p&gt;The key insight is distinguishing between server-to-server and client-to-server&amp;nbsp;traffic:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;SMTP&lt;/span&gt; (port 25)&lt;/strong&gt; must remain globally accessible. Mail servers from anywhere in the world need to deliver messages to my server. Restricting this would break&amp;nbsp;email.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Client ports&lt;/strong&gt; - &lt;span class="caps"&gt;IMAP&lt;/span&gt; (143), Submission (587), Sieve (4190), and webmail (80/443) - only need to be accessible from where my users actually&amp;nbsp;are.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This creates a two-tier firewall policy: &lt;span class="caps"&gt;SMTP&lt;/span&gt; open to all, everything else restricted by&amp;nbsp;geography.&lt;/p&gt;
&lt;h2 id="architecture-overview"&gt;Architecture&amp;nbsp;Overview&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                    Internet
                        |
                        v
            +-----------------------+
            |    PF Firewall        |
            |                       |
            |  &amp;lt;geoip_users&amp;gt; table  |
            |  ~273,000 CIDR blocks |
            +-----------+-----------+
                        |
        +---------------+---------------+
        |                               |
        v                               v
   +----------+                   +-----------+
   | SMTP:25  |                   | Client    |
   | Open to  |                   | Ports     |
   | everyone |                   | GeoIP     |
   +----------+                   | filtered  |
                                  +-----------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;&amp;lt;geoip_users&amp;gt;&lt;/code&gt; table contains approximately 273,000 &lt;span class="caps"&gt;CIDR&lt;/span&gt; blocks covering the allowed countries. &lt;span class="caps"&gt;PF&lt;/span&gt; evaluates incoming connections against this table before allowing access to restricted&amp;nbsp;services.&lt;/p&gt;
&lt;h2 id="maxmind-geolite2-database"&gt;MaxMind GeoLite2&amp;nbsp;Database&lt;/h2&gt;
&lt;p&gt;The geographic data comes from &lt;a href="https://dev.maxmind.com/geoip/geolite2-free-geolocation-data"&gt;MaxMind&amp;#8217;s GeoLite2&lt;/a&gt; database, available free with registration. The &lt;span class="caps"&gt;CSV&lt;/span&gt; format works well for our&amp;nbsp;purposes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ ls -lah /usr/local/share/GeoIP/GeoLite2-Country-CSV_20260106
total 11 MB
-rw-r--r--  1 root wheel   22M Jan  6 07:42 GeoLite2-Country-Blocks-IPv4.csv
-rw-r--r--  1 root wheel   27M Jan  6 07:42 GeoLite2-Country-Blocks-IPv6.csv
-rw-r--r--  1 root wheel  9.6K Jan  6 07:42 GeoLite2-Country-Locations-en.csv
...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The database structure is&amp;nbsp;straightforward:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Locations files&lt;/strong&gt; map geoname IDs to country codes and&amp;nbsp;names&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Blocks files&lt;/strong&gt; map &lt;span class="caps"&gt;CIDR&lt;/span&gt; ranges to geoname&amp;nbsp;IDs&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;To find all &lt;span class="caps"&gt;IP&lt;/span&gt; ranges for Germany, you first look up Germany&amp;#8217;s geoname &lt;span class="caps"&gt;ID&lt;/span&gt; in the locations file, then find all &lt;span class="caps"&gt;CIDR&lt;/span&gt; blocks with that &lt;span class="caps"&gt;ID&lt;/span&gt; in the blocks&amp;nbsp;files.&lt;/p&gt;
&lt;h2 id="the-update-script"&gt;The Update&amp;nbsp;Script&lt;/h2&gt;
&lt;p&gt;Processing the GeoLite2 data and loading it into &lt;span class="caps"&gt;PF&lt;/span&gt; requires careful handling. With nearly 300,000 entries, you can&amp;#8217;t just dump everything at once - &lt;span class="caps"&gt;PF&lt;/span&gt; will reject the operation with a memory error. The solution is chunked&amp;nbsp;loading:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# Configuration&lt;/span&gt;
&lt;span class="nv"&gt;GEOIP_DIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/local/share/GeoIP/GeoLite2-Country-CSV_20260106&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;PF_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/db/pf/geoip_allowed&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;TEMP_OUT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/tmp/geoip_allowed.tmp&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;TEMP_IDS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/tmp/target_ids.txt&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;CHUNK_DIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/tmp/geoip_chunks&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Allowed countries (ISO Codes)&lt;/span&gt;
&lt;span class="nv"&gt;COUNTRIES&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;DE|AT|CH|NL|LU|FR&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Cleanup&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_OUT&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
rm&lt;span class="w"&gt; &lt;/span&gt;-rf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$CHUNK_DIR&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Processing GeoIP data for: &lt;/span&gt;&lt;span class="nv"&gt;$COUNTRIES&lt;/span&gt;&lt;span class="s2"&gt; ...&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Step 1: Extract geoname IDs for target countries&lt;/span&gt;
grep&lt;span class="w"&gt; &lt;/span&gt;-E&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;,(&lt;/span&gt;&lt;span class="nv"&gt;$COUNTRIES&lt;/span&gt;&lt;span class="s2"&gt;),&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$GEOIP_DIR&lt;/span&gt;&lt;span class="s2"&gt;/&amp;quot;&lt;/span&gt;*Locations-en.csv&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;cut&lt;span class="w"&gt; &lt;/span&gt;-d,&lt;span class="w"&gt; &lt;/span&gt;-f1&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_IDS&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Step 2: Extract CIDRs from block files&lt;/span&gt;
&lt;span class="c1"&gt;# IPv4&lt;/span&gt;
grep&lt;span class="w"&gt; &lt;/span&gt;-F&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_IDS&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$GEOIP_DIR&lt;/span&gt;&lt;span class="s2"&gt;/&amp;quot;&lt;/span&gt;*Blocks-IPv4.csv&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;cut&lt;span class="w"&gt; &lt;/span&gt;-d,&lt;span class="w"&gt; &lt;/span&gt;-f1&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_OUT&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# IPv6&lt;/span&gt;
grep&lt;span class="w"&gt; &lt;/span&gt;-F&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_IDS&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$GEOIP_DIR&lt;/span&gt;&lt;span class="s2"&gt;/&amp;quot;&lt;/span&gt;*Blocks-IPv6.csv&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;cut&lt;span class="w"&gt; &lt;/span&gt;-d,&lt;span class="w"&gt; &lt;/span&gt;-f1&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_OUT&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_OUT&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;dirname&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PF_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;mv&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_OUT&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PF_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Loading firewall table in 50k chunks...&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Flush existing entries&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;geoip_users&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;flush

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Split into manageable chunks&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$CHUNK_DIR&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;split&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;50000&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PF_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$CHUNK_DIR&lt;/span&gt;&lt;span class="s2"&gt;/chunk_&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Load each chunk&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;chunk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$CHUNK_DIR&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;/chunk_*&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;geoip_users&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$chunk&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Verify&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;COUNT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;geoip_users&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;wc&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39; &amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Success: &lt;/span&gt;&lt;span class="nv"&gt;$COUNT&lt;/span&gt;&lt;span class="s2"&gt; networks now active in memory.&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Cleanup&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-rf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$CHUNK_DIR&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_IDS&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Error: No data extracted.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TEMP_OUT&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The script processes both IPv4 and IPv6 ranges, creating a unified list that &lt;span class="caps"&gt;PF&lt;/span&gt; can use for dual-stack&amp;nbsp;filtering.&lt;/p&gt;
&lt;h2 id="kernel-tuning"&gt;Kernel&amp;nbsp;Tuning&lt;/h2&gt;
&lt;p&gt;By default, &lt;span class="caps"&gt;PF&lt;/span&gt; has a relatively low limit on table entries. Loading 273,000 &lt;span class="caps"&gt;CIDR&lt;/span&gt; blocks requires increasing this limit&amp;nbsp;via &lt;code&gt;sysctl.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Allow large PF tables for GeoIP filtering
net.pf.request_maxcount=500000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This setting must be applied before loading the GeoIP data. After adding it&amp;nbsp;to &lt;code&gt;/etc/sysctl.conf&lt;/code&gt;, either reboot or apply it&amp;nbsp;live:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;net.pf.request_maxcount&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;500000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="pf-configuration"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt;&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Here&amp;#8217;s the relevant portion&amp;nbsp;of &lt;code&gt;/etc/pf.conf&lt;/code&gt; showing how the GeoIP table integrates with the firewall&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Macros&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;10.0.0.0/24&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2001:db8:1c1c:4d2:8000::/65&amp;quot;&lt;/span&gt;

&lt;span class="nx"&gt;mail_ipv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;10.0.0.3&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;mail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2001:db8:1c1c:4d2:8000::3&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;webmail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2001:db8:1c1c:4d2:8000::4&amp;quot;&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Tables&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;bruteforce&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;persist&lt;/span&gt;
&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;jails_v4&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;jails_v6&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;geoip_users&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;persist&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Options&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;limit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;entries&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1000000&lt;/span&gt;
&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;skip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;lo0&lt;/span&gt;
&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;block&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;
&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;loginterface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="nx"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;fragment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;reassemble&lt;/span&gt;
&lt;span class="nx"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;random&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;
&lt;span class="nx"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;random&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;mss&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1500&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;NAT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;jails&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;IPv4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;only&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="nx"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;jails_v4&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;RDR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;services&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Client&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ports&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;GeoIP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;restricted&lt;/span&gt;
&lt;span class="nx"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;geoip_users&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;143&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;587&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4190&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;mail_ipv4&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;SMTP&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Open&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;server&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;server&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="nx"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;mail_ipv4&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Filtering&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="nx"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;bruteforce&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="nx"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;
&lt;span class="nx"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;

&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;span class="nx"&gt;antispoof&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;bridge0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv6&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Client&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ports&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;GeoIP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;restriction&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;geoip_users&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;mail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;143&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;587&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4190&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv6&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;SMTP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;open&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;all&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;mail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv6&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Webmail&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;GeoIP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;restriction&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;geoip_users&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="nx"&gt;webmail_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Essential&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ICMPv6&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;echorep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;neighbrsol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;neighbradv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;toobig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;timex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;paramprob&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ICMP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPv4&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;unreach&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Jail&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;egress&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;bridge0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;jails_v4&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;!&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m m-Double"&gt;10.0.0.0&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;span class="nx"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;bridge0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nx"&gt;jails_v6&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;!&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="nx"&gt;c1c&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="nx"&gt;d2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="o"&gt;::/&lt;/span&gt;&lt;span class="mi"&gt;65&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note&amp;nbsp;the &lt;code&gt;set limit table-entries 1000000&lt;/code&gt; directive - this is the &lt;span class="caps"&gt;PF&lt;/span&gt;-level configuration that complements the sysctl&amp;nbsp;tuning.&lt;/p&gt;
&lt;p&gt;The dual approach to traffic handling is visible in the&amp;nbsp;rules:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;SMTP&lt;/span&gt; (port 25)&lt;/strong&gt;: &lt;code&gt;from any&lt;/code&gt; - no source&amp;nbsp;restriction&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Client ports (143, 587, 4190, 80, 443)&lt;/strong&gt;: &lt;code&gt;from &amp;lt;geoip_users&amp;gt;&lt;/code&gt; - restricted to allowed&amp;nbsp;countries&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="verifying-the-setup"&gt;Verifying the&amp;nbsp;Setup&lt;/h2&gt;
&lt;p&gt;After loading the GeoIP data, verify the table is&amp;nbsp;populated:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;geoip_users&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;wc&lt;span class="w"&gt; &lt;/span&gt;-l
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;273460&lt;/span&gt;

$&lt;span class="w"&gt; &lt;/span&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;geoip_users&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tail&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;
&lt;span class="w"&gt;   &lt;/span&gt;2a1b:3e0:500::/64
&lt;span class="w"&gt;   &lt;/span&gt;2a1d:3e0:500::/64
&lt;span class="w"&gt;   &lt;/span&gt;2a1f:3e0:500::/64
&lt;span class="w"&gt;   &lt;/span&gt;2c0f:2c40:a310::/48
&lt;span class="w"&gt;   &lt;/span&gt;2c0f:fc04:2000::/64
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The loaded rules can be verified&amp;nbsp;with &lt;code&gt;pfctl -s r&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pass in quick on vtnet0 inet6 proto tcp from &amp;lt;geoip_users&amp;gt; \
    to 2001:db8:1c1c:4d2:8000::3 port = imap flags S/SA keep state
pass in quick on vtnet0 inet6 proto tcp from &amp;lt;geoip_users&amp;gt; \
    to 2001:db8:1c1c:4d2:8000::3 port = submission flags S/SA keep state
pass in quick on vtnet0 inet6 proto tcp from any \
    to 2001:db8:1c1c:4d2:8000::3 port = smtp flags S/SA keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The rules clearly show the differentiation: &lt;span class="caps"&gt;IMAP&lt;/span&gt; and submission require membership&amp;nbsp;in &lt;code&gt;&amp;lt;geoip_users&amp;gt;&lt;/code&gt;, while &lt;span class="caps"&gt;SMTP&lt;/span&gt;&amp;nbsp;allows &lt;code&gt;any&lt;/code&gt; source.&lt;/p&gt;
&lt;h2 id="automating-updates"&gt;Automating&amp;nbsp;Updates&lt;/h2&gt;
&lt;p&gt;MaxMind updates the GeoLite2 database weekly. A cron job keeps the firewall table&amp;nbsp;current:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Update GeoIP data weekly (Mondays at 3 AM)&lt;/span&gt;
&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;update_geoip_maxmind&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sh&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;geoip_update&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;amp;&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For the database download itself, MaxMind provides&amp;nbsp;a &lt;code&gt;geoipupdate&lt;/code&gt; tool that handles authentication and retrieval. The update script then processes the fresh&amp;nbsp;data.&lt;/p&gt;
&lt;h2 id="results"&gt;Results&lt;/h2&gt;
&lt;p&gt;The impact has been&amp;nbsp;significant:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Log volume&lt;/strong&gt;: Brute-force attempt logs dropped by roughly&amp;nbsp;90%&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Connection overhead&lt;/strong&gt;: Fewer &lt;span class="caps"&gt;TCP&lt;/span&gt; handshakes to reject means lower resource&amp;nbsp;usage&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Signal-to-noise ratio&lt;/strong&gt;: When something does appear in the logs, it&amp;#8217;s more likely to be worth&amp;nbsp;investigating&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The remaining attempts typically come from compromised hosts within allowed countries or &lt;span class="caps"&gt;VPN&lt;/span&gt; exit nodes. These still hit rate limiting, but at a manageable&amp;nbsp;volume.&lt;/p&gt;
&lt;h2 id="caveats"&gt;Caveats&lt;/h2&gt;
&lt;p&gt;This approach isn&amp;#8217;t without&amp;nbsp;trade-offs:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Traveling users&lt;/strong&gt;: Anyone traveling outside the allowed countries will be blocked. Communicate this clearly to users, or consider adding &lt;span class="caps"&gt;VPN&lt;/span&gt; access as a&amp;nbsp;bypass.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;VPN&lt;/span&gt; services&lt;/strong&gt;: Users connecting through &lt;span class="caps"&gt;VPN&lt;/span&gt; providers with exit nodes outside allowed countries will be&amp;nbsp;blocked.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Database accuracy&lt;/strong&gt;: GeoIP data isn&amp;#8217;t perfect. Some &lt;span class="caps"&gt;IP&lt;/span&gt; ranges are misattributed, and mobile carriers sometimes route traffic through unexpected&amp;nbsp;locations.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Table size&lt;/strong&gt;: 273,000 entries consume kernel memory. On memory-constrained systems, consider restricting to fewer&amp;nbsp;countries.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For my use case - a small mail server with users who rarely travel - these trade-offs are&amp;nbsp;acceptable.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Geographic filtering is a powerful layer in a defense-in-depth strategy. It doesn&amp;#8217;t replace proper authentication, &lt;span class="caps"&gt;TLS&lt;/span&gt;, or rate limiting, but it dramatically reduces the attack surface by eliminating entire classes of attackers before they even reach the application&amp;nbsp;layer.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s table mechanism handles large GeoIP datasets efficiently, and the chunked loading approach works around memory limitations during updates. Combined with MaxMind&amp;#8217;s regularly updated databases, this provides a low-maintenance way to keep your servers visible only where they need to&amp;nbsp;be.&lt;/p&gt;
&lt;p&gt;The bots scanning from halfway around the world? They now see nothing but a silent drop. Your logs stay clean, your resources stay available for legitimate users, and your threat surface shrinks to a fraction of what it&amp;nbsp;was.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://dev.maxmind.com/geoip/geolite2-free-geolocation-data"&gt;MaxMind GeoLite2 Free Geolocation&amp;nbsp;Data&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.openbsd.org/faq/pf/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; - The OpenBSD Packet Filter&lt;/a&gt; (FreeBSD&amp;#8217;s &lt;span class="caps"&gt;PF&lt;/span&gt; derives from&amp;nbsp;OpenBSD)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/firewalls/"&gt;FreeBSD Handbook:&amp;nbsp;Firewalls&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=pf.conf"&gt;FreeBSD pf.conf(5) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;Sometimes the best security is simply not being there. If an attacker can&amp;#8217;t reach your service, they can&amp;#8217;t attack it. Geographic filtering makes your server invisible to most of the internet while remaining fully accessible to the people who actually need&amp;nbsp;it.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="security"/><category term="firewall"/><category term="pf"/><category term="geoip"/><category term="mail"/></entry><entry><title>Managing FreeBSD Jails with Ansible: The jailexec Connection Plugin</title><link href="https://hofstede.it/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/" rel="alternate"/><published>2025-12-31T00:00:00+01:00</published><updated>2025-12-31T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-12-31:/managing-freebsd-jails-with-ansible-the-jailexec-connection-plugin/</id><summary type="html">&lt;p&gt;A custom Ansible connection plugin that enables native management of FreeBSD jails via jexec, without requiring &lt;span class="caps"&gt;SSH&lt;/span&gt; inside each&amp;nbsp;jail.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;FreeBSD jails are elegant isolation containers, but managing them with Ansible has traditionally required either running &lt;span class="caps"&gt;SSH&lt;/span&gt; daemons inside each jail or using awkward workarounds. The &lt;strong&gt;jailexec&lt;/strong&gt; connection plugin solves this by connecting to the jail host via &lt;span class="caps"&gt;SSH&lt;/span&gt; and&amp;nbsp;using &lt;code&gt;jexec&lt;/code&gt; to execute commands inside jails - just like you would&amp;nbsp;manually.&lt;/p&gt;
&lt;p&gt;&lt;img alt="jailexec in action" src="https://hofstede.it/images/2025-12-31-ansible-jailexec.png" title="The jailexec connection plugin executing Ansible ping against multiple FreeBSD jails"&gt;&lt;/p&gt;
&lt;h2 id="the-problem-with-jail-management"&gt;The Problem with Jail&amp;nbsp;Management&lt;/h2&gt;
&lt;p&gt;When automating FreeBSD jail infrastructure, you face a&amp;nbsp;choice:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Run &lt;span class="caps"&gt;SSH&lt;/span&gt; in every jail&lt;/strong&gt; - Works, but defeats much of the security benefit of jails and adds management&amp;nbsp;overhead&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Use&amp;nbsp;the &lt;code&gt;jail&lt;/code&gt; connection plugin&lt;/strong&gt; - Requires Ansible to run directly on the jail host, limiting remote&amp;nbsp;management&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Manual jexec wrapper scripts&lt;/strong&gt; - Brittle, hard to maintain, and doesn&amp;#8217;t integrate cleanly with Ansible&amp;#8217;s&amp;nbsp;ecosystem&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;What I wanted was simple: &lt;span class="caps"&gt;SSH&lt;/span&gt; to my jail host, then&amp;nbsp;use &lt;code&gt;jexec&lt;/code&gt; to run commands inside any jail. That&amp;#8217;s exactly what the jailexec plugin&amp;nbsp;does.&lt;/p&gt;
&lt;h2 id="how-it-works"&gt;How It&amp;nbsp;Works&lt;/h2&gt;
&lt;p&gt;The plugin extends Ansible&amp;#8217;s &lt;span class="caps"&gt;SSH&lt;/span&gt; connection to create a two-hop execution&amp;nbsp;model:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[ Control Machine ] --SSH--&amp;gt; [ Jail Host ] --jexec--&amp;gt; [ Jail ]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;When you run an Ansible task against a&amp;nbsp;jail:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The plugin establishes an &lt;span class="caps"&gt;SSH&lt;/span&gt; connection to the jail&amp;nbsp;host&lt;/li&gt;
&lt;li&gt;Commands are wrapped with privilege escalation&amp;nbsp;(&lt;code&gt;doas&lt;/code&gt; or &lt;code&gt;sudo&lt;/code&gt;)&amp;nbsp;and &lt;code&gt;jexec&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Output is captured and returned to Ansible as if running directly in the&amp;nbsp;jail&lt;/li&gt;
&lt;li&gt;File transfers use a two-stage process: upload to the host, then move into the jail&amp;#8217;s&amp;nbsp;filesystem&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;This means your jails don&amp;#8217;t need &lt;span class="caps"&gt;SSH&lt;/span&gt; or any additional services - just a working FreeBSD&amp;nbsp;environment.&lt;/p&gt;
&lt;h2 id="installation"&gt;Installation&lt;/h2&gt;
&lt;p&gt;The plugin is a single Python file. Drop it into your Ansible plugins&amp;nbsp;directory:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# User-specific installation&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;~/.ansible/plugins/connection/
curl&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;jailexec.py&lt;span class="w"&gt; &lt;/span&gt;https://raw.githubusercontent.com/chofstede/ansible_jailexec/main/jailexec.py
mv&lt;span class="w"&gt; &lt;/span&gt;jailexec.py&lt;span class="w"&gt; &lt;/span&gt;~/.ansible/plugins/connection/

&lt;span class="c1"&gt;# Or project-specific&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;connection_plugins/
cp&lt;span class="w"&gt; &lt;/span&gt;jailexec.py&lt;span class="w"&gt; &lt;/span&gt;connection_plugins/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For project-specific installation, add to&amp;nbsp;your &lt;code&gt;ansible.cfg&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[defaults]&lt;/span&gt;
&lt;span class="na"&gt;connection_plugins&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;./connection_plugins&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="inventory-configuration"&gt;Inventory&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;The key difference from standard Ansible inventory is specifying the jail host separately from the jail&amp;nbsp;itself:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[freebsd_hosts]&lt;/span&gt;
&lt;span class="na"&gt;jail-host.example.com ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;ssh ansible_user=ansible ansible_port=22&lt;/span&gt;

&lt;span class="k"&gt;[freebsd_jails]&lt;/span&gt;
&lt;span class="na"&gt;web-jail    ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;jailexec  ansible_jail_host=jail-host.example.com ansible_user=ansible&lt;/span&gt;
&lt;span class="na"&gt;db-jail     ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;jailexec  ansible_jail_host=jail-host.example.com ansible_user=ansible&lt;/span&gt;
&lt;span class="na"&gt;app-jail    ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;jailexec  ansible_jail_host=jail-host.example.com ansible_user=ansible&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The inventory hostname becomes the jail name by default. &lt;span class="caps"&gt;SSH&lt;/span&gt; authentication settings are inherited from your &lt;span class="caps"&gt;SSH&lt;/span&gt; configuration or can be specified&amp;nbsp;per-host.&lt;/p&gt;
&lt;h3 id="configuration-variables"&gt;Configuration&amp;nbsp;Variables&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Variable&lt;/th&gt;
&lt;th&gt;Default&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ansible_jail_host&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;(required)&lt;/td&gt;
&lt;td&gt;FreeBSD host running the jails&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ansible_jail_name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;inventory_hostname&lt;/td&gt;
&lt;td&gt;Override jail name&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ansible_jail_user&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;root&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;User for command execution in jail&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ansible_jail_privilege_escalation&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;doas&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Privilege escalation method&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ansible_jail_remote_tmp&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/tmp/.ansible/tmp&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Temporary directory path&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="jail-host-setup"&gt;Jail Host&amp;nbsp;Setup&lt;/h2&gt;
&lt;p&gt;The &lt;span class="caps"&gt;SSH&lt;/span&gt; user on your jail host needs privilege escalation configured for jail management commands. I&amp;nbsp;recommend &lt;code&gt;doas&lt;/code&gt; for its&amp;nbsp;simplicity:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Install doas&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;doas

&lt;span class="c1"&gt;# Configure /usr/local/etc/doas.conf&lt;/span&gt;
permit&lt;span class="w"&gt; &lt;/span&gt;nopass&lt;span class="w"&gt; &lt;/span&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;as&lt;span class="w"&gt; &lt;/span&gt;root&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;jls
permit&lt;span class="w"&gt; &lt;/span&gt;nopass&lt;span class="w"&gt; &lt;/span&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;as&lt;span class="w"&gt; &lt;/span&gt;root&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;jexec
permit&lt;span class="w"&gt; &lt;/span&gt;nopass&lt;span class="w"&gt; &lt;/span&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;as&lt;span class="w"&gt; &lt;/span&gt;root&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;mkdir
permit&lt;span class="w"&gt; &lt;/span&gt;nopass&lt;span class="w"&gt; &lt;/span&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;as&lt;span class="w"&gt; &lt;/span&gt;root&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;mv
permit&lt;span class="w"&gt; &lt;/span&gt;nopass&lt;span class="w"&gt; &lt;/span&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;as&lt;span class="w"&gt; &lt;/span&gt;root&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;rm
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For sudo, the equivalent&amp;nbsp;in &lt;code&gt;/usr/local/etc/sudoers&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;ALL&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;root&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;NOPASSWD:&lt;span class="w"&gt; &lt;/span&gt;/usr/sbin/jls,&lt;span class="w"&gt; &lt;/span&gt;/usr/sbin/jexec,&lt;span class="w"&gt; &lt;/span&gt;/bin/mkdir,&lt;span class="w"&gt; &lt;/span&gt;/bin/mv,&lt;span class="w"&gt; &lt;/span&gt;/bin/rm
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="basic-usage"&gt;Basic&amp;nbsp;Usage&lt;/h2&gt;
&lt;p&gt;Test&amp;nbsp;connectivity:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;hosts.ini&lt;span class="w"&gt; &lt;/span&gt;freebsd_jails&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;ping
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Expected&amp;nbsp;output:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;web&lt;/span&gt;&lt;span class="mi"&gt;-&lt;/span&gt;&lt;span class="err"&gt;jail&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;SUCCESS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;changed&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;ping&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pong&amp;quot;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Run commands inside&amp;nbsp;jails:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check FreeBSD version in all jails&lt;/span&gt;
ansible&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;hosts.ini&lt;span class="w"&gt; &lt;/span&gt;freebsd_jails&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;command&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;freebsd-version&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Install packages&lt;/span&gt;
ansible&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;hosts.ini&lt;span class="w"&gt; &lt;/span&gt;freebsd_jails&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;community.general.pkgng&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;name=nginx state=present&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="example-playbook"&gt;Example&amp;nbsp;Playbook&lt;/h2&gt;
&lt;p&gt;A typical playbook for configuring a web server&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Configure web server jail&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;web-jail&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;connection&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;jailexec&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;become&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Install nginx&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;community.general.pkgng&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;nginx&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;present&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Copy nginx configuration&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.copy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;src&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;files/nginx.conf&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/usr/local/etc/nginx/nginx.conf&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;root&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;group&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;wheel&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;0644&amp;#39;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;notify&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;restart nginx&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Enable and start nginx&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.service&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;nginx&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;started&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;enabled&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;handlers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;restart nginx&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.service&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;nginx&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;restarted&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="multi-environment-inventory"&gt;Multi-Environment&amp;nbsp;Inventory&lt;/h2&gt;
&lt;p&gt;For more complex setups with multiple jail&amp;nbsp;hosts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[production_jails]&lt;/span&gt;
&lt;span class="na"&gt;prod-web-01   ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;jailexec  ansible_jail_host=prod-host-01.example.com&lt;/span&gt;
&lt;span class="na"&gt;prod-web-02   ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;jailexec  ansible_jail_host=prod-host-02.example.com&lt;/span&gt;
&lt;span class="na"&gt;prod-db-01    ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;jailexec  ansible_jail_host=prod-host-01.example.com  ansible_jail_user=postgres&lt;/span&gt;

&lt;span class="k"&gt;[staging_jails]&lt;/span&gt;
&lt;span class="na"&gt;stage-web     ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;jailexec  ansible_jail_host=stage-host.example.com&lt;/span&gt;
&lt;span class="na"&gt;stage-db      ansible_connection&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;jailexec  ansible_jail_host=stage-host.example.com&lt;/span&gt;

&lt;span class="k"&gt;[all_jails:children]&lt;/span&gt;
&lt;span class="na"&gt;production_jails&lt;/span&gt;
&lt;span class="na"&gt;staging_jails&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="security-design"&gt;Security&amp;nbsp;Design&lt;/h2&gt;
&lt;p&gt;The plugin implements several security&amp;nbsp;measures:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Input validation&lt;/strong&gt;: Jail names are validated against FreeBSD naming conventions. Path traversal attempts&amp;nbsp;(&lt;code&gt;..&lt;/code&gt;) and shell injection patterns are&amp;nbsp;blocked.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Two-stage file transfers&lt;/strong&gt;: Files are first uploaded to a temporary location the &lt;span class="caps"&gt;SSH&lt;/span&gt; user can access, then moved into the jail using privilege escalation. This prevents direct writes to privileged&amp;nbsp;paths.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Secure temporary files&lt;/strong&gt;: Temporary files use mode 600 and are cleaned up on&amp;nbsp;failure.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Connection reuse&lt;/strong&gt;: &lt;span class="caps"&gt;SSH&lt;/span&gt; connections are pooled, reducing authentication overhead and&amp;nbsp;exposure.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Privilege escalation control&lt;/strong&gt;: You choose&amp;nbsp;between &lt;code&gt;doas&lt;/code&gt; or &lt;code&gt;sudo&lt;/code&gt;, and can lock down exactly which commands are&amp;nbsp;permitted.&lt;/p&gt;
&lt;h2 id="troubleshooting"&gt;Troubleshooting&lt;/h2&gt;
&lt;p&gt;Enable verbose output for&amp;nbsp;debugging:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Maximum verbosity&lt;/span&gt;
ansible&lt;span class="w"&gt; &lt;/span&gt;-vvv&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;hosts.ini&lt;span class="w"&gt; &lt;/span&gt;freebsd_jails&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;ping

&lt;span class="c1"&gt;# With Ansible debugging&lt;/span&gt;
&lt;span class="nv"&gt;ANSIBLE_DEBUG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;-vvv&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;hosts.ini&lt;span class="w"&gt; &lt;/span&gt;freebsd_jails&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;ping
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="common-issues"&gt;Common&amp;nbsp;Issues&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;No jail host specified&amp;#8221;&lt;/strong&gt;&amp;nbsp;Add &lt;code&gt;ansible_jail_host=your-host.example.com&lt;/code&gt; to your&amp;nbsp;inventory.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Permission denied accessing jail&amp;#8221;&lt;/strong&gt;
Verify doas/sudo configuration. Test manually on the jail&amp;nbsp;host:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;doas&lt;span class="w"&gt; &lt;/span&gt;jls
doas&lt;span class="w"&gt; &lt;/span&gt;jexec&lt;span class="w"&gt; &lt;/span&gt;web-jail&lt;span class="w"&gt; &lt;/span&gt;/bin/sh&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;id&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Jail &amp;#8216;name&amp;#8217; not found&amp;#8221;&lt;/strong&gt;
Check that the jail is&amp;nbsp;running:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;jls
service&lt;span class="w"&gt; &lt;/span&gt;jail&lt;span class="w"&gt; &lt;/span&gt;onestart&lt;span class="w"&gt; &lt;/span&gt;web-jail
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;File transfer failures&lt;/strong&gt;
Verify the &lt;span class="caps"&gt;SSH&lt;/span&gt; user can write&amp;nbsp;to &lt;code&gt;/tmp&lt;/code&gt; on the jail host and that disk space is&amp;nbsp;available.&lt;/p&gt;
&lt;h2 id="future-development"&gt;Future&amp;nbsp;Development&lt;/h2&gt;
&lt;p&gt;The plugin is actively maintained. Potential future&amp;nbsp;additions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Ansible Collection packaging for easier&amp;nbsp;distribution&lt;/li&gt;
&lt;li&gt;Support for jail templates and cloning&amp;nbsp;operations&lt;/li&gt;
&lt;li&gt;Parallel execution&amp;nbsp;optimizations&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Contributions are welcome on &lt;a href="https://github.com/chofstede/ansible_jailexec"&gt;GitHub&lt;/a&gt; or &lt;a href="https://codeberg.org/Larvitz/ansible_jailexec"&gt;Codeberg&lt;/a&gt;.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The jailexec plugin fills a gap in FreeBSD automation tooling. It brings proper Ansible support to jail management without compromising the security benefits of running minimal jail&amp;nbsp;environments.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re managing FreeBSD jails at any scale, this plugin lets you treat them as first-class Ansible targets - the same playbooks, roles, and patterns you use everywhere else, now working seamlessly with FreeBSD&amp;#8217;s native&amp;nbsp;containerization.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/chofstede/ansible_jailexec"&gt;jailexec on&amp;nbsp;GitHub&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://codeberg.org/Larvitz/ansible_jailexec"&gt;jailexec on&amp;nbsp;Codeberg&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook -&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.ansible.com/ansible/latest/plugins/connection.html"&gt;Ansible Connection Plugins&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.openbsd.org/doas"&gt;doas - dedicated OpenBSD application&amp;nbsp;subexecutor&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="ansible"/><category term="jails"/><category term="automation"/><category term="devops"/></entry><entry><title>FreeBSD 15 Cloud-Init on Proxmox: Working Around nuageinit’s Network-Config Gap</title><link href="https://hofstede.it/freebsd-15-cloud-init-on-proxmox-working-around-nuageinits-network-config-gap/" rel="alternate"/><published>2025-12-28T00:00:00+01:00</published><updated>2025-12-28T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-12-28:/freebsd-15-cloud-init-on-proxmox-working-around-nuageinits-network-config-gap/</id><summary type="html">&lt;p&gt;Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; generates network-config v1, but FreeBSD 15&amp;#8217;s nuageinit only speaks v2. Here&amp;#8217;s a script that bridges the gap for static &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;configuration.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;FreeBSD 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; shipped earlier this month with &lt;span class="caps"&gt;VM&lt;/span&gt; images that are &amp;#8220;cloud-init compatible&amp;#8221; with a catch. Instead of using the Python-based cloud-init that Linux distributions rely on, FreeBSD implemented their own solution: nuageinit(7). It&amp;#8217;s lighter, written in Lua, and fits FreeBSD&amp;#8217;s philosophy of minimal dependencies. The only problem? It doesn&amp;#8217;t understand network-config version&amp;nbsp;1.&lt;/p&gt;
&lt;p&gt;This matters because Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt;-even in version 9.1 still generates network-config v1 format when you configure cloud-init through the web &lt;span class="caps"&gt;UI&lt;/span&gt;. The result: your carefully configured static &lt;span class="caps"&gt;IP&lt;/span&gt; settings are silently ignored, and the &lt;span class="caps"&gt;VM&lt;/span&gt; falls back to &lt;span class="caps"&gt;DHCP&lt;/span&gt;. For production environments where predictable addressing matters, this is a&amp;nbsp;deal-breaker.&lt;/p&gt;
&lt;h2 id="the-version-mismatch"&gt;The Version&amp;nbsp;Mismatch&lt;/h2&gt;
&lt;p&gt;Cloud-init&amp;#8217;s network configuration has evolved through several&amp;nbsp;versions:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Version 1&lt;/strong&gt;: The original format, with nested dictionaries and explicit type&amp;nbsp;fields&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Version 2 (Netplan)&lt;/strong&gt;: A cleaner &lt;span class="caps"&gt;YAML&lt;/span&gt; structure that became the de facto&amp;nbsp;standard&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Here&amp;#8217;s what Proxmox generates&amp;nbsp;(v1):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;1&lt;/span&gt;
&lt;span class="nt"&gt;config&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;physical&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;vtnet0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;subnets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;type&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;static&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;address&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10.254.254.42/24&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;gateway&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10.254.254.1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And what nuageinit expects&amp;nbsp;(v2):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2&lt;/span&gt;
&lt;span class="nt"&gt;ethernets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;vtnet0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;addresses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10.254.254.42/24&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;gateway4&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10.254.254.1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The structures are fundamentally different. nuageinit simply doesn&amp;#8217;t parse v1, leaving your &lt;span class="caps"&gt;VM&lt;/span&gt; with &lt;span class="caps"&gt;DHCP&lt;/span&gt;-assigned addresses instead of the static configuration you&amp;nbsp;specified.&lt;/p&gt;
&lt;h2 id="the-workaround"&gt;The&amp;nbsp;Workaround&lt;/h2&gt;
&lt;p&gt;Until Proxmox adds v2 support or nuageinit gains v1 compatibility, the solution is to generate your own cloud-init &lt;span class="caps"&gt;ISO&lt;/span&gt;. The script below runs on a Proxmox host and creates a properly formatted &lt;span class="caps"&gt;ISO&lt;/span&gt; that FreeBSD will actually&amp;nbsp;understand.&lt;/p&gt;
&lt;h3 id="installation"&gt;Installation&lt;/h3&gt;
&lt;p&gt;Save the script&amp;nbsp;as &lt;code&gt;/usr/local/bin/freebsd-cloudinit-iso&lt;/code&gt; and make it&amp;nbsp;executable:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;+x&lt;span class="w"&gt; &lt;/span&gt;/usr/local/bin/freebsd-cloudinit-iso
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Dependencies are&amp;nbsp;minimal-just &lt;code&gt;genisoimage&lt;/code&gt; or &lt;code&gt;mkisofs&lt;/code&gt;, which are typically already present on Proxmox&amp;nbsp;hosts.&lt;/p&gt;
&lt;h3 id="basic-usage"&gt;Basic&amp;nbsp;Usage&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-cloudinit-iso&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--vmid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;203&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--hostname&lt;span class="w"&gt; &lt;/span&gt;myfreebsd&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--domain&lt;span class="w"&gt; &lt;/span&gt;example.com&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;admin&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--password&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;secretpassword&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ip4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.254.254.42/24&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--gw4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.254.254.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--dns&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.1.1.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--dns&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.0.0.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--storage&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;local&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This creates an &lt;span class="caps"&gt;ISO&lt;/span&gt;&amp;nbsp;with:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Hostname set&amp;nbsp;to &lt;code&gt;myfreebsd.example.com&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;A&amp;nbsp;user &lt;code&gt;admin&lt;/code&gt; with sudo privileges and the specified&amp;nbsp;password&lt;/li&gt;
&lt;li&gt;Static IPv4&amp;nbsp;configuration&lt;/li&gt;
&lt;li&gt;Cloudflare &lt;span class="caps"&gt;DNS&lt;/span&gt;&amp;nbsp;servers&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The &lt;span class="caps"&gt;ISO&lt;/span&gt; is automatically attached to &lt;span class="caps"&gt;VM&lt;/span&gt; 203 as a &lt;span class="caps"&gt;CD&lt;/span&gt;-&lt;span class="caps"&gt;ROM&lt;/span&gt;&amp;nbsp;drive.&lt;/p&gt;
&lt;h3 id="dual-stack-configuration"&gt;Dual-Stack&amp;nbsp;Configuration&lt;/h3&gt;
&lt;p&gt;For IPv6-first&amp;nbsp;environments:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-cloudinit-iso&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--vmid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;204&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--hostname&lt;span class="w"&gt; &lt;/span&gt;webserver&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--domain&lt;span class="w"&gt; &lt;/span&gt;prod.example.com&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;deploy&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ssh-key-file&lt;span class="w"&gt; &lt;/span&gt;~/.ssh/id_ed25519.pub&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ip6&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8::50/64&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--gw6&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8::1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ip4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.0.0.50/24&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--gw4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.0.0.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--dns&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8::1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--dns&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.0.0.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--storage&lt;span class="w"&gt; &lt;/span&gt;ceph-nvme
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note that IPv6 addresses are listed first in the generated config-nuageinit applies them in order, and IPv6-first makes sense for modern&amp;nbsp;networks.&lt;/p&gt;
&lt;h3 id="ssh-key-authentication"&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; Key&amp;nbsp;Authentication&lt;/h3&gt;
&lt;p&gt;For production deployments, prefer &lt;span class="caps"&gt;SSH&lt;/span&gt; keys over&amp;nbsp;passwords:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;freebsd-cloudinit-iso&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--vmid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;205&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--hostname&lt;span class="w"&gt; &lt;/span&gt;secure&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--domain&lt;span class="w"&gt; &lt;/span&gt;internal.lan&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ssh-key-file&lt;span class="w"&gt; &lt;/span&gt;~/.ssh/automation.pub&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ip4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.100.0.10/24&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--gw4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.100.0.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--dns&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.100.0.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--storage&lt;span class="w"&gt; &lt;/span&gt;local-zfs
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You can also pass the key directly&amp;nbsp;with &lt;code&gt;--ssh-key "ssh-ed25519 AAAA..."&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="what-the-script-generates"&gt;What the Script&amp;nbsp;Generates&lt;/h2&gt;
&lt;p&gt;The script creates three files in the &lt;span class="caps"&gt;ISO&lt;/span&gt;:&lt;/p&gt;
&lt;h3 id="meta-data"&gt;meta-data&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;instance-id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;203-1735400000&lt;/span&gt;
&lt;span class="nt"&gt;local-hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;myfreebsd&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The instance-id uses the &lt;span class="caps"&gt;VM&lt;/span&gt; &lt;span class="caps"&gt;ID&lt;/span&gt; and timestamp, ensuring cloud-init recognizes each boot as potentially needing&amp;nbsp;reconfiguration.&lt;/p&gt;
&lt;h3 id="user-data"&gt;user-data&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;#cloud-config&lt;/span&gt;
&lt;span class="nt"&gt;hostname&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;myfreebsd&lt;/span&gt;
&lt;span class="nt"&gt;fqdn&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;myfreebsd.example.com&lt;/span&gt;
&lt;span class="nt"&gt;manage_etc_hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;

&lt;span class="nt"&gt;users&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;admin&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;groups&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;wheel&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;shell&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/bin/sh&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;sudo&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ALL=(ALL) NOPASSWD:ALL&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;lock_passwd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;passwd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;$6$rounds=5000$...&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;ssh_authorized_keys&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ssh-ed25519 AAAA...&lt;/span&gt;

&lt;span class="nt"&gt;ssh_pwauth&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="nt"&gt;chpasswd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;expire&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Passwords are hashed using &lt;span class="caps"&gt;SHA&lt;/span&gt;-512 before being written-plaintext credentials never touch the &lt;span class="caps"&gt;ISO&lt;/span&gt;.&lt;/p&gt;
&lt;h3 id="network-config"&gt;network-config&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2&lt;/span&gt;
&lt;span class="nt"&gt;ethernets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;vtnet0&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;addresses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2001:db8::50/64&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10.0.0.50/24&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;gateway6&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2001:db8::1&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;gateway4&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10.0.0.1&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;nameservers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;addresses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;2001:db8::1&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;10.0.0.1&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;search&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;prod.example.com&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is the crucial part-version 2 format that nuageinit actually&amp;nbsp;parses.&lt;/p&gt;
&lt;h2 id="full-option-reference"&gt;Full Option&amp;nbsp;Reference&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Required:
  --vmid &amp;lt;id&amp;gt;           VM ID in Proxmox
  --hostname &amp;lt;name&amp;gt;     Hostname (without domain)
  --user &amp;lt;username&amp;gt;     Username to create

Network (at least one required):
  --ip4 &amp;lt;addr/prefix&amp;gt;   IPv4 address with prefix (e.g., 10.0.0.50/24)
  --gw4 &amp;lt;addr&amp;gt;          IPv4 gateway
  --ip6 &amp;lt;addr/prefix&amp;gt;   IPv6 address with prefix
  --gw6 &amp;lt;addr&amp;gt;          IPv6 gateway

Authentication (at least one required):
  --password &amp;lt;pass&amp;gt;       Password for user (will be hashed)
  --root-password &amp;lt;pass&amp;gt;  Password for root (will be hashed)
  --ssh-key &amp;lt;key&amp;gt;         SSH public key string
  --ssh-key-file &amp;lt;file&amp;gt;   Path to SSH public key file

DNS:
  --dns &amp;lt;addr&amp;gt;          DNS server (repeatable)
  --search &amp;lt;domain&amp;gt;     Search domain (repeatable)
  --domain &amp;lt;domain&amp;gt;     Domain (also added to search)

Storage:
  --storage &amp;lt;name&amp;gt;      Proxmox storage for ISO (default: auto-detect)

Options:
  --iface &amp;lt;name&amp;gt;        Network interface (default: vtnet0)
  --groups &amp;lt;groups&amp;gt;     Comma-separated groups (default: wheel)
  --output-dir &amp;lt;dir&amp;gt;    Output directory instead of storage
  --no-attach           Create ISO but don&amp;#39;t attach to VM
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="workflow-integration"&gt;Workflow&amp;nbsp;Integration&lt;/h2&gt;
&lt;h3 id="with-terraformopentofu"&gt;With&amp;nbsp;Terraform/OpenTofu&lt;/h3&gt;
&lt;p&gt;If you&amp;#8217;re provisioning FreeBSD VMs with Terraform, call the script as a local-exec provisioner after creating the &lt;span class="caps"&gt;VM&lt;/span&gt; but before first&amp;nbsp;boot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kr"&gt;resource&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;&amp;quot;proxmox_vm_qemu&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;&amp;quot;freebsd&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;freebsd-web&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;vmid&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;210&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;target_node&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pve1&amp;quot;&lt;/span&gt;
&lt;span class="c1"&gt;  # ... other config ...&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="kr"&gt;resource&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;&amp;quot;null_resource&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;&amp;quot;cloudinit&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="na"&gt;depends_on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nv"&gt;proxmox_vm_qemu.freebsd&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="kr"&gt;provisioner&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;&amp;quot;local-exec&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;command&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;lt;-&lt;/span&gt;&lt;span class="dl"&gt;EOT&lt;/span&gt;
&lt;span class="sh"&gt;      ssh root@pve1 &amp;#39;freebsd-cloudinit-iso \&lt;/span&gt;
&lt;span class="sh"&gt;        --vmid 210 \&lt;/span&gt;
&lt;span class="sh"&gt;        --hostname freebsd-web \&lt;/span&gt;
&lt;span class="sh"&gt;        --domain example.com \&lt;/span&gt;
&lt;span class="sh"&gt;        --user terraform \&lt;/span&gt;
&lt;span class="sh"&gt;        --ssh-key-file /root/.ssh/terraform.pub \&lt;/span&gt;
&lt;span class="sh"&gt;        --ip4 10.0.0.210/24 \&lt;/span&gt;
&lt;span class="sh"&gt;        --gw4 10.0.0.1 \&lt;/span&gt;
&lt;span class="sh"&gt;        --dns 10.0.0.1 \&lt;/span&gt;
&lt;span class="sh"&gt;        --storage local&amp;#39;&lt;/span&gt;
&lt;span class="dl"&gt;    EOT&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="with-ansible"&gt;With&amp;nbsp;Ansible&lt;/h3&gt;
&lt;p&gt;For Ansible-driven&amp;nbsp;provisioning:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Generate FreeBSD cloud-init ISO&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;delegate_to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;proxmox_host&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;command&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;&amp;gt;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;freebsd-cloudinit-iso&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--vmid {{ vm_id }}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--hostname {{ inventory_hostname_short }}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--domain {{ domain }}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--user {{ ansible_user }}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--ssh-key-file /root/.ssh/ansible.pub&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--ip4 {{ ansible_host }}/24&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--gw4 {{ gateway }}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--dns {{ dns_server }}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;--storage {{ proxmox_storage }}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="limitations-and-caveats"&gt;Limitations and&amp;nbsp;Caveats&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Single interface only&lt;/strong&gt;: The script currently supports one network interface. Multiple NICs would require extending the network-config&amp;nbsp;generation.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;No &lt;span class="caps"&gt;DHCP&lt;/span&gt; option&lt;/strong&gt;: This is intentionally for static configurations. If you want &lt;span class="caps"&gt;DHCP&lt;/span&gt;, the default Proxmox cloud-init works fine (nuageinit falls back to &lt;span class="caps"&gt;DHCP&lt;/span&gt; when it can&amp;#8217;t parse the network&amp;nbsp;config).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;VirtIO interface name&lt;/strong&gt;: The&amp;nbsp;default &lt;code&gt;vtnet0&lt;/code&gt; assumes VirtIO network devices. If you&amp;#8217;re using emulated e1000 or another driver,&amp;nbsp;specify &lt;code&gt;--iface&lt;/code&gt; accordingly.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Storage detection&lt;/strong&gt;: Auto-detection looks for storage&amp;nbsp;with &lt;code&gt;iso&lt;/code&gt; content type. If your Proxmox setup is non-standard,&amp;nbsp;specify &lt;code&gt;--storage&lt;/code&gt; explicitly.&lt;/p&gt;
&lt;h2 id="why-not-just-fix-proxmox"&gt;Why Not Just Fix&amp;nbsp;Proxmox?&lt;/h2&gt;
&lt;p&gt;Fair question. There&amp;#8217;s an open feature request for network-config v2 support in Proxmox, but it&amp;#8217;s not trivial-the entire cloud-init subsystem assumes v1 internally. FreeBSD&amp;#8217;s nuageinit could also add v1 parsing, but the project explicitly chose v2 as the modern&amp;nbsp;format.&lt;/p&gt;
&lt;p&gt;In the meantime, this script fills the gap. It&amp;#8217;s not elegant, but it works reliably and integrates cleanly with existing Proxmox&amp;nbsp;workflows.&lt;/p&gt;
&lt;h2 id="the-script"&gt;The&amp;nbsp;Script&lt;/h2&gt;
&lt;p&gt;The full script is available below. It&amp;#8217;s a single Bash file with no external dependencies beyond standard Proxmox&amp;nbsp;tools:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="c1"&gt;#&lt;/span&gt;
&lt;span class="c1"&gt;# freebsd-cloudinit-iso - Generate cloud-init ISO for FreeBSD VMs on Proxmox&lt;/span&gt;
&lt;span class="c1"&gt;#&lt;/span&gt;
&lt;span class="c1"&gt;# This works around FreeBSD 15&amp;#39;s nuageinit not supporting network-config v1&lt;/span&gt;
&lt;span class="c1"&gt;# by generating v2 format that nuageinit actually understands.&lt;/span&gt;

&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e

&lt;span class="c1"&gt;# Defaults&lt;/span&gt;
&lt;span class="nv"&gt;VMID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;HOSTNAME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;DOMAIN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;USER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;PASSWORD&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ROOT_PASSWORD&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;SSH_KEY_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;IP4&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;GW4&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;IP6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;GW6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;DNS&lt;/span&gt;&lt;span class="o"&gt;=()&lt;/span&gt;
&lt;span class="nv"&gt;SEARCH&lt;/span&gt;&lt;span class="o"&gt;=()&lt;/span&gt;
&lt;span class="nv"&gt;STORAGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;IFACE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;OUTPUT_DIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ATTACH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
&lt;span class="nv"&gt;USER_GROUPS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;wheel&amp;quot;&lt;/span&gt;

usage&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF&lt;/span&gt;
&lt;span class="s"&gt;Usage: $(basename &amp;quot;$0&amp;quot;) [options]&lt;/span&gt;

&lt;span class="s"&gt;Required:&lt;/span&gt;
&lt;span class="s"&gt;  --vmid &amp;lt;id&amp;gt;           VM ID&lt;/span&gt;
&lt;span class="s"&gt;  --hostname &amp;lt;name&amp;gt;     Hostname (without domain)&lt;/span&gt;
&lt;span class="s"&gt;  --user &amp;lt;username&amp;gt;     Username to create&lt;/span&gt;

&lt;span class="s"&gt;Network (at least one of --ip4 or --ip6 required):&lt;/span&gt;
&lt;span class="s"&gt;  --ip4 &amp;lt;addr/prefix&amp;gt;   IPv4 address with prefix (e.g., 10.0.0.50/24)&lt;/span&gt;
&lt;span class="s"&gt;  --gw4 &amp;lt;addr&amp;gt;          IPv4 gateway&lt;/span&gt;
&lt;span class="s"&gt;  --ip6 &amp;lt;addr/prefix&amp;gt;   IPv6 address with prefix (e.g., 2001:db8::50/64)&lt;/span&gt;
&lt;span class="s"&gt;  --gw6 &amp;lt;addr&amp;gt;          IPv6 gateway&lt;/span&gt;

&lt;span class="s"&gt;Authentication (at least one required):&lt;/span&gt;
&lt;span class="s"&gt;  --password &amp;lt;pass&amp;gt;     Password for user (plaintext, will be hashed)&lt;/span&gt;
&lt;span class="s"&gt;  --root-password &amp;lt;pass&amp;gt; Password for root (plaintext, will be hashed)&lt;/span&gt;
&lt;span class="s"&gt;  --ssh-key &amp;lt;key&amp;gt;       SSH public key string&lt;/span&gt;
&lt;span class="s"&gt;  --ssh-key-file &amp;lt;file&amp;gt; Path to SSH public key file&lt;/span&gt;

&lt;span class="s"&gt;DNS:&lt;/span&gt;
&lt;span class="s"&gt;  --dns &amp;lt;addr&amp;gt;          DNS server (can be specified multiple times)&lt;/span&gt;
&lt;span class="s"&gt;  --search &amp;lt;domain&amp;gt;     Search domain (can be specified multiple times)&lt;/span&gt;
&lt;span class="s"&gt;  --domain &amp;lt;domain&amp;gt;     Domain name (also added to search if not present)&lt;/span&gt;

&lt;span class="s"&gt;Storage:&lt;/span&gt;
&lt;span class="s"&gt;  --storage &amp;lt;name&amp;gt;      Proxmox storage name (default: auto-detect)&lt;/span&gt;

&lt;span class="s"&gt;Options:&lt;/span&gt;
&lt;span class="s"&gt;  --iface &amp;lt;name&amp;gt;        Network interface name (default: vtnet0)&lt;/span&gt;
&lt;span class="s"&gt;  --groups &amp;lt;groups&amp;gt;     Comma-separated groups (default: wheel)&lt;/span&gt;
&lt;span class="s"&gt;  --output-dir &amp;lt;dir&amp;gt;    Output directory for ISO (default: storage path)&lt;/span&gt;
&lt;span class="s"&gt;  --no-attach           Don&amp;#39;t attach ISO to VM, just create it&lt;/span&gt;
&lt;span class="s"&gt;  --help                Show this help&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

error&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Error: &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

warn&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Warning: &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Parse arguments&lt;/span&gt;
&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-gt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--vmid&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="nv"&gt;VMID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--hostname&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="nv"&gt;HOSTNAME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--domain&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="nv"&gt;DOMAIN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--user&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="nv"&gt;USER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--password&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="nv"&gt;PASSWORD&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--root-password&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;ROOT_PASSWORD&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--ssh-key&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--ssh-key-file&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--ip4&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;IP4&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--gw4&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;GW4&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--ip6&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;IP6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--gw6&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;GW6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--dns&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;DNS&lt;/span&gt;&lt;span class="o"&gt;+=(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--search&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="nv"&gt;SEARCH&lt;/span&gt;&lt;span class="o"&gt;+=(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--storage&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;STORAGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--iface&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nv"&gt;IFACE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--groups&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="nv"&gt;USER_GROUPS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--output-dir&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;OUTPUT_DIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$2&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--no-attach&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nv"&gt;ATTACH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;false&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--help&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;usage&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Unknown option: &lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;esac&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;

&lt;span class="c1"&gt;# Validation&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$VMID&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--vmid is required&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOSTNAME&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--hostname is required&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$USER&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--user is required&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$IP4&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$IP6&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;At least one of --ip4 or --ip6 is required&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PASSWORD&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$SSH_KEY&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$SSH_KEY_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;At least one of --password, --ssh-key, or --ssh-key-file is required&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Check for required tools&lt;/span&gt;
&lt;span class="nb"&gt;command&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;genisoimage&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;command&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;mkisofs&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;genisoimage or mkisofs required&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;command&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;qm&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;warn&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;qm not found - not running on Proxmox?&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Load SSH key from file if specified&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$SSH_KEY_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$SSH_KEY_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;SSH key file not found: &lt;/span&gt;&lt;span class="nv"&gt;$SSH_KEY_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$SSH_KEY_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="c1"&gt;# Find storage path&lt;/span&gt;
find_storage_path&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;local&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;storage&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$OUTPUT_DIR&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$OUTPUT_DIR&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$storage&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;storage&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;pvesm&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;--content&lt;span class="w"&gt; &lt;/span&gt;iso&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;NR&amp;gt;1 {print $1; exit}&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$storage&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;No storage with &amp;#39;iso&amp;#39; content type found. Specify --storage or --output-dir&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;local&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;path&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;pvesm&lt;span class="w"&gt; &lt;/span&gt;path&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;storage&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:iso/dummy.iso&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sed&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;s|/dummy\.iso$||&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$path&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Could not determine path for storage: &lt;/span&gt;&lt;span class="nv"&gt;$storage&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;STORAGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$storage&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$path&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="nv"&gt;SNIPPETS_PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;find_storage_path&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$STORAGE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$SNIPPETS_PATH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Create temp directory&lt;/span&gt;
&lt;span class="nv"&gt;TMPDIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;trap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;rm -rf &amp;#39;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;&amp;#39;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;EXIT

&lt;span class="c1"&gt;# Generate password hashes&lt;/span&gt;
&lt;span class="nv"&gt;PASSWORD_HASH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PASSWORD&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;PASSWORD_HASH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;openssl&lt;span class="w"&gt; &lt;/span&gt;passwd&lt;span class="w"&gt; &lt;/span&gt;-6&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PASSWORD&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;PASSWORD_HASH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;python3&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;import crypt; print(crypt.crypt(&amp;#39;&lt;/span&gt;&lt;span class="nv"&gt;$PASSWORD&lt;/span&gt;&lt;span class="s2"&gt;&amp;#39;, crypt.mksalt(crypt.METHOD_SHA512)))&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Could not hash password&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="nv"&gt;ROOT_PASSWORD_HASH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ROOT_PASSWORD&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;ROOT_PASSWORD_HASH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;openssl&lt;span class="w"&gt; &lt;/span&gt;passwd&lt;span class="w"&gt; &lt;/span&gt;-6&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ROOT_PASSWORD&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;ROOT_PASSWORD_HASH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;python3&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;import crypt; print(crypt.crypt(&amp;#39;&lt;/span&gt;&lt;span class="nv"&gt;$ROOT_PASSWORD&lt;/span&gt;&lt;span class="s2"&gt;&amp;#39;, crypt.mksalt(crypt.METHOD_SHA512)))&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;error&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Could not hash root password&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="c1"&gt;# Build FQDN&lt;/span&gt;
&lt;span class="nv"&gt;FQDN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$HOSTNAME&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$DOMAIN&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;FQDN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;HOSTNAME&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;DOMAIN&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Add domain to search if not present&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$DOMAIN&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;domain_in_search&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SEARCH&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$s&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$DOMAIN&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;domain_in_search&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;$domain_in_search&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;SEARCH&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$DOMAIN&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SEARCH&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="c1"&gt;# Create meta-data&lt;/span&gt;
cat&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/meta-data&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF&lt;/span&gt;
&lt;span class="s"&gt;instance-id: $(uuidgen 2&amp;gt;/dev/null || echo &amp;quot;${VMID}-$(date +%s)&amp;quot;)&lt;/span&gt;
&lt;span class="s"&gt;local-hostname: ${HOSTNAME}&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;

&lt;span class="c1"&gt;# Create user-data&lt;/span&gt;
cat&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/user-data&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF&lt;/span&gt;
&lt;span class="s"&gt;#cloud-config&lt;/span&gt;
&lt;span class="s"&gt;hostname: ${HOSTNAME}&lt;/span&gt;
&lt;span class="s"&gt;fqdn: ${FQDN}&lt;/span&gt;
&lt;span class="s"&gt;manage_etc_hosts: true&lt;/span&gt;

&lt;span class="s"&gt;users:&lt;/span&gt;
&lt;span class="s"&gt;  - name: ${USER}&lt;/span&gt;
&lt;span class="s"&gt;    groups: ${USER_GROUPS}&lt;/span&gt;
&lt;span class="s"&gt;    shell: /bin/sh&lt;/span&gt;
&lt;span class="s"&gt;    sudo: ALL=(ALL) NOPASSWD:ALL&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PASSWORD_HASH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/user-data&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF&lt;/span&gt;
&lt;span class="s"&gt;    lock_passwd: false&lt;/span&gt;
&lt;span class="s"&gt;    passwd: ${PASSWORD_HASH}&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$SSH_KEY&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/user-data&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF&lt;/span&gt;
&lt;span class="s"&gt;    ssh_authorized_keys:&lt;/span&gt;
&lt;span class="s"&gt;      - ${SSH_KEY}&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PASSWORD_HASH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/user-data&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF&lt;/span&gt;

&lt;span class="s"&gt;ssh_pwauth: true&lt;/span&gt;
&lt;span class="s"&gt;chpasswd:&lt;/span&gt;
&lt;span class="s"&gt;  expire: false&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ROOT_PASSWORD_HASH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/user-data&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF&lt;/span&gt;

&lt;span class="s"&gt;chpasswd:&lt;/span&gt;
&lt;span class="s"&gt;  expire: false&lt;/span&gt;
&lt;span class="s"&gt;  users:&lt;/span&gt;
&lt;span class="s"&gt;    - name: root&lt;/span&gt;
&lt;span class="s"&gt;      password: ${ROOT_PASSWORD_HASH}&lt;/span&gt;
&lt;span class="s"&gt;      type: HASH&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="c1"&gt;# Create network-config (v2 format)&lt;/span&gt;
cat&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;lt;&amp;lt;EOF&lt;/span&gt;
&lt;span class="s"&gt;version: 2&lt;/span&gt;
&lt;span class="s"&gt;ethernets:&lt;/span&gt;
&lt;span class="s"&gt;  ${IFACE}:&lt;/span&gt;
&lt;span class="s"&gt;EOF&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;    addresses:&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$IP6&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;      - &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IP6&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$IP4&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;      - &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IP4&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;

&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$GW6&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;    gateway6: &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;GW6&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$GW4&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;    gateway4: &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;GW4&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;DNS&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-gt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;SEARCH&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-gt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;    nameservers:&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;DNS&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-gt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;      addresses:&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;dns&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;DNS&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;        - &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;dns&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${#&lt;/span&gt;&lt;span class="nv"&gt;SEARCH&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-gt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;      search:&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;search&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SEARCH&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;        - &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;search&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;/network-config&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="c1"&gt;# Generate ISO&lt;/span&gt;
&lt;span class="nv"&gt;ISO_NAME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;freebsd-ci-&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;VMID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.iso&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ISO_PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SNIPPETS_PATH&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;ISO_NAME&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;command&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;genisoimage&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;MKISO&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;genisoimage&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;MKISO&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mkisofs&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="nv"&gt;$MKISO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-output&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ISO_PATH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-volid&lt;span class="w"&gt; &lt;/span&gt;cidata&lt;span class="w"&gt; &lt;/span&gt;-joliet&lt;span class="w"&gt; &lt;/span&gt;-rock&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TMPDIR&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Created: &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;ISO_PATH&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Attach to VM&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$ATTACH&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;command&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;qm&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;qm&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$VMID&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;qm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$VMID&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--delete&lt;span class="w"&gt; &lt;/span&gt;ide2&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;qm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$VMID&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--ide2&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;STORAGE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:iso/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;ISO_NAME&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;,media=cdrom&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Attached to VM &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;VMID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; as ide2&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;else&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;warn&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;VM &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;VMID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; not found, ISO created but not attached&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;FreeBSD 15&amp;#8217;s nuageinit and Proxmox&amp;#8217;s cloud-init generator speak different dialects of the same configuration language. Until upstream fixes arrive, this script provides a reliable workaround for anyone running FreeBSD VMs on Proxmox who needs static network&amp;nbsp;configuration.&lt;/p&gt;
&lt;p&gt;The approach is straightforward: generate the &lt;span class="caps"&gt;ISO&lt;/span&gt; yourself with the correct format. It integrates with existing automation tools, requires no modifications to either Proxmox or FreeBSD, and produces predictable&amp;nbsp;results.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.freebsd.org/releases/15.0R/announce/"&gt;FreeBSD 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt;&amp;nbsp;Announcement&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man.freebsd.org/cgi/man.cgi?query=nuageinit&amp;amp;sektion=7"&gt;nuageinit(7) man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://cloudinit.readthedocs.io/en/latest/reference/network-config-format-v2.html"&gt;Cloud-init Network Config Version&amp;nbsp;2&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pve.proxmox.com/wiki/Cloud-Init_Support"&gt;Proxmox Cloud-Init&amp;nbsp;Support&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="proxmox"/><category term="cloud-init"/><category term="virtualization"/><category term="automation"/></entry><entry><title>Running a Factorio Headless Server on FreeBSD with the Linuxulator</title><link href="https://hofstede.it/running-a-factorio-headless-server-on-freebsd-with-the-linuxulator/" rel="alternate"/><published>2025-12-20T00:00:00+01:00</published><updated>2025-12-20T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-12-20:/running-a-factorio-headless-server-on-freebsd-with-the-linuxulator/</id><summary type="html">&lt;p&gt;How to run a Factorio dedicated server inside a FreeBSD jail using Linux binary compatibility, with no native port&amp;nbsp;required.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;Factorio doesn&amp;#8217;t have a native FreeBSD build, but that doesn&amp;#8217;t mean you can&amp;#8217;t run it on FreeBSD. The Linuxulator, FreeBSD&amp;#8217;s Linux binary compatibility layer, handles Linux &lt;span class="caps"&gt;ELF&lt;/span&gt; binaries seamlessly. This article walks through setting up a Factorio headless server inside a Bastille jail, complete with firewall rules for public&amp;nbsp;access.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Factorio" src="https://hofstede.it/images/2025-12-20-factorio-freebsd-linuxulator.png" title="Factorio: The factory must grow"&gt;&lt;/p&gt;
&lt;h2 id="prerequisites"&gt;Prerequisites&lt;/h2&gt;
&lt;p&gt;This guide assumes you already&amp;nbsp;have:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;A FreeBSD host (14.x or later) with Bastille&amp;nbsp;installed&lt;/li&gt;
&lt;li&gt;A working bridge network for jails (see my &lt;a href="https://hofstede.it/hosting-a-static-blog-on-freebsd-with-bastille-jails-and-automated-deployment/"&gt;previous article on FreeBSD jail infrastructure&lt;/a&gt;)&lt;/li&gt;
&lt;li&gt;A Factorio account to download the headless&amp;nbsp;server&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="enabling-the-64-bit-linuxulator"&gt;Enabling the 64-bit&amp;nbsp;Linuxulator&lt;/h2&gt;
&lt;p&gt;The Factorio server is a 64-bit Linux binary. FreeBSD&amp;#8217;s Linuxulator needs to be loaded on the &lt;strong&gt;host&lt;/strong&gt; system before jails can use&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;Add&amp;nbsp;to &lt;code&gt;/boot/loader.conf&lt;/code&gt; on the&amp;nbsp;host:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;linux64_load&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Reboot, or load it&amp;nbsp;immediately:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;kldload&lt;span class="w"&gt; &lt;/span&gt;linux64
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You can verify it&amp;#8217;s&amp;nbsp;loaded:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;kldstat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;linux
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;9&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;0xffffffff827fa000&lt;span class="w"&gt;    &lt;/span&gt;619f8&lt;span class="w"&gt; &lt;/span&gt;linux64.ko
&lt;span class="m"&gt;10&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;0xffffffff8285c000&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="m"&gt;20970&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;linux_common.ko
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="creating-the-jail"&gt;Creating the&amp;nbsp;Jail&lt;/h2&gt;
&lt;p&gt;Create a standard &lt;span class="caps"&gt;VNET&lt;/span&gt; jail attached to your existing bridge. In this example, I&amp;#8217;m using&amp;nbsp;the &lt;code&gt;bastille0&lt;/code&gt; bridge with a private IPv4/IPv6 address&amp;nbsp;pair:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-B&lt;span class="w"&gt; &lt;/span&gt;factorio&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;14&lt;/span&gt;.3-RELEASE&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.254.252.98&lt;span class="w"&gt; &lt;/span&gt;bastille0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Configure the jail&amp;#8217;s network in&amp;nbsp;its &lt;code&gt;/etc/rc.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;ifconfig_e0b_factorio_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vnet0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.254.252.98 netmask 255.255.255.0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:8000::98/64&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.254.252.1&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# That&amp;#39;s the hosts IP address on the bridge bastille0&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:8000::1&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# That&amp;#39;s the hosts IPv6 address on the bridge bastille0&lt;/span&gt;

&lt;span class="nv"&gt;syslogd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-ss&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_submit_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_outbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_msp_queue_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;cron_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-J 60&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;linux_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The key line&amp;nbsp;is &lt;code&gt;linux_enable="YES"&lt;/code&gt; - this enables the Linux compatibility layer inside the&amp;nbsp;jail.&lt;/p&gt;
&lt;p&gt;Start the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;factorio
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="installing-linux-userland"&gt;Installing Linux&amp;nbsp;Userland&lt;/h2&gt;
&lt;p&gt;Inside the jail, install the Rocky Linux 9 base package. This provides the necessary Linux shared&amp;nbsp;libraries:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;factorio&lt;span class="w"&gt; &lt;/span&gt;pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;linux_base-rl9
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This pulls in a minimal Linux userland including glibc, which the Factorio binary needs. The installed packages are&amp;nbsp;lightweight:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;linux_base-rl9-9.6_1           Base set of packages needed in Linux mode (Rocky Linux 9.6)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="creating-the-factorio-user"&gt;Creating the Factorio&amp;nbsp;User&lt;/h2&gt;
&lt;p&gt;Create a dedicated user to run the&amp;nbsp;server:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;cmd&lt;span class="w"&gt; &lt;/span&gt;factorio&lt;span class="w"&gt; &lt;/span&gt;adduser
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Follow the prompts - the defaults are fine. I&amp;nbsp;used &lt;code&gt;factorio&lt;/code&gt; as the username&amp;nbsp;with &lt;code&gt;/home/factorio&lt;/code&gt; as the home directory. When prompted for the&amp;nbsp;shell, &lt;code&gt;/bin/sh&lt;/code&gt; is a safe&amp;nbsp;choice.&lt;/p&gt;
&lt;h2 id="downloading-and-extracting-factorio"&gt;Downloading and Extracting&amp;nbsp;Factorio&lt;/h2&gt;
&lt;p&gt;Enter the jail and switch to the factorio&amp;nbsp;user:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;console&lt;span class="w"&gt; &lt;/span&gt;factorio
su&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;factorio
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Download and extract the server. You can use FreeBSD&amp;#8217;s&amp;nbsp;built-in &lt;code&gt;fetch&lt;/code&gt; or&amp;nbsp;install &lt;code&gt;wget&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;fetch&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;factorio-headless.tar.xz&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://factorio.com/get-download/stable/headless/linux64&amp;quot;&lt;/span&gt;
tar&lt;span class="w"&gt; &lt;/span&gt;xf&lt;span class="w"&gt; &lt;/span&gt;factorio-headless.tar.xz
rm&lt;span class="w"&gt; &lt;/span&gt;factorio-headless.tar.xz
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This creates&amp;nbsp;a &lt;code&gt;factorio/&lt;/code&gt; directory with the server files. You can verify the binary is a Linux &lt;span class="caps"&gt;ELF&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;file&lt;span class="w"&gt; &lt;/span&gt;factorio/bin/x64/factorio
factorio/bin/x64/factorio:&lt;span class="w"&gt; &lt;/span&gt;ELF&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;64&lt;/span&gt;-bit&lt;span class="w"&gt; &lt;/span&gt;LSB&lt;span class="w"&gt; &lt;/span&gt;pie&lt;span class="w"&gt; &lt;/span&gt;executable,&lt;span class="w"&gt; &lt;/span&gt;x86-64,&lt;span class="w"&gt; &lt;/span&gt;version&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;SYSV&lt;span class="o"&gt;)&lt;/span&gt;,
dynamically&lt;span class="w"&gt; &lt;/span&gt;linked,&lt;span class="w"&gt; &lt;/span&gt;interpreter&lt;span class="w"&gt; &lt;/span&gt;/lib64/ld-linux-x86-64.so.2,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;GNU/Linux&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;.2.0,
with&lt;span class="w"&gt; &lt;/span&gt;debug_info,&lt;span class="w"&gt; &lt;/span&gt;not&lt;span class="w"&gt; &lt;/span&gt;stripped
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="configuring-the-server"&gt;Configuring the&amp;nbsp;Server&lt;/h2&gt;
&lt;p&gt;Factorio stores its configuration&amp;nbsp;in &lt;code&gt;factorio/data/server-settings.json&lt;/code&gt;. Copy the example and customize&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;factorio
cp&lt;span class="w"&gt; &lt;/span&gt;data/server-settings.example.json&lt;span class="w"&gt; &lt;/span&gt;data/server-settings.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Edit at&amp;nbsp;minimum:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;name&lt;/code&gt;: Your server&amp;#8217;s public&amp;nbsp;name&lt;/li&gt;
&lt;li&gt;&lt;code&gt;description&lt;/code&gt;: What players will see in the server&amp;nbsp;browser&lt;/li&gt;
&lt;li&gt;&lt;code&gt;visibility&lt;/code&gt;: Set&amp;nbsp;to &lt;code&gt;{"public": true, "lan": true}&lt;/code&gt; if you want it&amp;nbsp;listed&lt;/li&gt;
&lt;li&gt;&lt;code&gt;username&lt;/code&gt; and &lt;code&gt;password&lt;/code&gt;: Your Factorio account credentials (required for public&amp;nbsp;servers)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;game_password&lt;/code&gt;: Optional password for players to&amp;nbsp;join&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="running-the-server"&gt;Running the&amp;nbsp;Server&lt;/h2&gt;
&lt;p&gt;When running under the Linuxulator, Factorio can&amp;#8217;t automatically detect its installation directory and defaults to looking&amp;nbsp;in &lt;code&gt;/usr/share/factorio&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;--executable-path&lt;/code&gt; flag tells it where to find the game&amp;nbsp;data.&lt;/p&gt;
&lt;h3 id="quick-start-with-tmux"&gt;Quick Start with&amp;nbsp;tmux&lt;/h3&gt;
&lt;p&gt;The simplest approach is running the server in a tmux session. This keeps the console accessible and survives disconnections. Note&amp;nbsp;that &lt;code&gt;~/factorio&lt;/code&gt; expands&amp;nbsp;to &lt;code&gt;/home/factorio/factorio&lt;/code&gt; since the home directory contains the&amp;nbsp;extracted &lt;code&gt;factorio/&lt;/code&gt; folder:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;tmux&lt;span class="w"&gt; &lt;/span&gt;new-session&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;factorio
tmux&lt;span class="w"&gt; &lt;/span&gt;send-keys&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;factorio&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;cd ~/factorio &amp;amp;&amp;amp; bin/x64/factorio --executable-path ~/factorio/bin/x64/ --start-server-load-latest --server-settings ./data/server-settings.json&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;Enter
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;To attach to the console&amp;nbsp;later:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;tmux&lt;span class="w"&gt; &lt;/span&gt;attach&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;factorio
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;To create a new map (run from&amp;nbsp;within &lt;code&gt;~/factorio&lt;/code&gt;):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bin/x64/factorio&lt;span class="w"&gt; &lt;/span&gt;--executable-path&lt;span class="w"&gt; &lt;/span&gt;~/factorio/bin/x64/&lt;span class="w"&gt; &lt;/span&gt;--create&lt;span class="w"&gt; &lt;/span&gt;./saves/myworld.zip
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then start the server with that&amp;nbsp;save:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bin/x64/factorio&lt;span class="w"&gt; &lt;/span&gt;--executable-path&lt;span class="w"&gt; &lt;/span&gt;~/factorio/bin/x64/&lt;span class="w"&gt; &lt;/span&gt;--start-server&lt;span class="w"&gt; &lt;/span&gt;./saves/myworld.zip&lt;span class="w"&gt; &lt;/span&gt;--server-settings&lt;span class="w"&gt; &lt;/span&gt;./data/server-settings.json
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="proper-rcd-service-script"&gt;Proper rc.d Service&amp;nbsp;Script&lt;/h3&gt;
&lt;p&gt;For production use, an rc.d script integrates with FreeBSD&amp;#8217;s service management.&amp;nbsp;Create &lt;code&gt;/usr/local/etc/rc.d/factorio&lt;/code&gt; inside the&amp;nbsp;jail:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# PROVIDE: factorio&lt;/span&gt;
&lt;span class="c1"&gt;# REQUIRE: LOGIN&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;

.&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.subr

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;factorio&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rcvar&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;factorio_enable

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;

:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_enable&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_user&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;factorio&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_dir&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/home/factorio/factorio&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_savefile&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--start-server-load-latest&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_flags&lt;/span&gt;&lt;span class="p"&gt;:=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;

&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/factorio/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;start_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_start&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;stop_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_stop&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;status_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_status&amp;quot;&lt;/span&gt;

factorio_start&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_user&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_user&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;755&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;/var/run/factorio
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Starting &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# -S: don&amp;#39;t send SIGHUP to child on exit, -T: set process title for ps&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/usr/sbin/daemon&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_user&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;-T&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_dir&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;/bin/x64/factorio&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--executable-path&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_dir&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;/bin/x64/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_savefile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--server-settings&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_dir&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;/data/server-settings.json&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;factorio_flags&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

factorio_status&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-0&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; is running as pid &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; is not running.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

factorio_stop&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Stopping &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-TERM&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null
&lt;span class="w"&gt;        &lt;/span&gt;sleep&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;else&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; is not running.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Make it executable and enable&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;+x&lt;span class="w"&gt; &lt;/span&gt;/usr/local/etc/rc.d/factorio
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;factorio_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;YES
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now you can manage the server with standard&amp;nbsp;commands:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;factorio&lt;span class="w"&gt; &lt;/span&gt;start
service&lt;span class="w"&gt; &lt;/span&gt;factorio&lt;span class="w"&gt; &lt;/span&gt;stop
service&lt;span class="w"&gt; &lt;/span&gt;factorio&lt;span class="w"&gt; &lt;/span&gt;status
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;To specify a particular save file instead of loading the&amp;nbsp;latest:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;factorio_savefile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--start-server /home/factorio/factorio/saves/myworld.zip&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="firewall-configuration"&gt;Firewall&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Factorio uses &lt;span class="caps"&gt;UDP&lt;/span&gt; port 34197 by default. Add redirect and pass rules to your&amp;nbsp;host&amp;#8217;s &lt;code&gt;/etc/pf.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;# Macros&lt;/span&gt;
&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2001:db8::f3d1&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;factorio_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;10.254.252.98&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;factorio_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;2001:db8:8000::98&amp;quot;&lt;/span&gt;

&lt;span class="cp"&gt;# Redirect incoming Factorio traffic to the jail&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;34197&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;factorio_v4&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;34197&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;factorio_v6&lt;/span&gt;

&lt;span class="cp"&gt;# Allow the traffic through&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;factorio_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;34197&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="n"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;udp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="n"&gt;factorio_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;34197&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kr"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Test and reload &lt;span class="caps"&gt;PF&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-nf&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.conf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pfctl&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/pf.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="verifying-the-setup"&gt;Verifying the&amp;nbsp;Setup&lt;/h2&gt;
&lt;p&gt;If using the rc.d script, start the server and check the logs (inside the&amp;nbsp;jail):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;factorio&lt;span class="w"&gt; &lt;/span&gt;start
tail&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/home/factorio/factorio/factorio-current.log
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A successful startup looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;   0.000 2025-12-20 12:02:46; Factorio 2.0.72 (build 84292, linux64, headless)
   0.000 Operating system: Linux
   0.000 Config path: /usr/home/factorio/factorio/config/config.ini
   0.000 Read data path: /usr/home/factorio/factorio/data
   0.000 Write data path: /usr/home/factorio/factorio [108461/109184MB]
   0.009 System info: [CPU: AMD EPYC-Rome Processor, 8 cores, RAM: 15957 MB]
   0.010 Running in headless mode
   ...
   2.404 Hosting game at IP ADDR:({0.0.0.0:34197})
   2.739 Info ServerMultiplayerManager.cpp:808: changing state to(InGame)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &amp;#8220;Operating system: Linux&amp;#8221; line is the key indicator - it proves the Linuxulator translation layer is active and the binary genuinely believes it&amp;#8217;s running on Linux. Players can now connect via your server&amp;#8217;s public &lt;span class="caps"&gt;IP&lt;/span&gt; on port&amp;nbsp;34197.&lt;/p&gt;
&lt;h2 id="troubleshooting"&gt;Troubleshooting&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;error while loading shared libraries&amp;#8221;&lt;/strong&gt;: The Linux userland isn&amp;#8217;t installed&amp;nbsp;or &lt;code&gt;linux_enable&lt;/code&gt; isn&amp;#8217;t set.&amp;nbsp;Install &lt;code&gt;linux_base-rl9&lt;/code&gt; and&amp;nbsp;ensure &lt;code&gt;/etc/rc.conf&lt;/code&gt; has &lt;code&gt;linux_enable="YES"&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Server starts but players can&amp;#8217;t connect&lt;/strong&gt;: Check your &lt;span class="caps"&gt;PF&lt;/span&gt; rules. Ensure both&amp;nbsp;the &lt;code&gt;rdr&lt;/code&gt; and &lt;code&gt;pass&lt;/code&gt; rules are in place for &lt;span class="caps"&gt;UDP&lt;/span&gt; 34197.&amp;nbsp;Use &lt;code&gt;tcpdump -i vtnet0 udp port 34197&lt;/code&gt; on the host to verify traffic is&amp;nbsp;arriving.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Failed to find the system certificate authority file&amp;#8221;&lt;/strong&gt;: This warning is harmless. Factorio falls back to its bundled certificate file for &lt;span class="caps"&gt;HTTPS&lt;/span&gt; requests to the auth&amp;nbsp;server.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Error configuring paths: There is no package core in /usr/share/factorio&amp;#8221;&lt;/strong&gt;:&amp;nbsp;The &lt;code&gt;--executable-path&lt;/code&gt; flag is missing. Under the Linuxulator, Factorio can&amp;#8217;t auto-detect its installation directory. Always&amp;nbsp;specify &lt;code&gt;--executable-path /path/to/factorio/bin/x64/&lt;/code&gt; when starting the&amp;nbsp;server.&lt;/p&gt;
&lt;h2 id="summary"&gt;Summary&lt;/h2&gt;
&lt;p&gt;Running Factorio on FreeBSD is straightforward thanks to the&amp;nbsp;Linuxulator:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Load &lt;code&gt;linux64.ko&lt;/code&gt; on the&amp;nbsp;host&lt;/li&gt;
&lt;li&gt;Create a jail&amp;nbsp;with &lt;code&gt;linux_enable="YES"&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Install &lt;code&gt;linux_base-rl9&lt;/code&gt; for the Linux shared&amp;nbsp;libraries&lt;/li&gt;
&lt;li&gt;Download the headless server and run&amp;nbsp;it&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The Linuxulator handles the translation transparently - the Factorio binary thinks it&amp;#8217;s running on Linux. Combined with FreeBSD jails for isolation and &lt;span class="caps"&gt;PF&lt;/span&gt; for traffic control, you get a clean, manageable game server&amp;nbsp;setup.&lt;/p&gt;
&lt;p&gt;The factory must grow. Even on&amp;nbsp;FreeBSD.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://factorio.com/download"&gt;Factorio Headless Server&amp;nbsp;Download&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://wiki.factorio.com/Multiplayer#Dedicated_Server"&gt;Factorio Dedicated Server&amp;nbsp;Wiki&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/linuxemu/"&gt;FreeBSD Handbook: Linux Binary&amp;nbsp;Compatibility&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastillebsd.org"&gt;BastilleBSD&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.openbsd.org/faq/pf/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; - The OpenBSD Packet&amp;nbsp;Filter&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;Thanks to the FreeBSD team for maintaining the Linuxulator, making it possible to run Linux-only software without virtualization&amp;nbsp;overhead.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="gaming"/><category term="jails"/><category term="bastille"/><category term="factorio"/></entry><entry><title>Hosting a Static Blog on FreeBSD with Bastille Jails and Automated Deployment</title><link href="https://hofstede.it/hosting-a-static-blog-on-freebsd-with-bastille-jails-and-automated-deployment/" rel="alternate"/><published>2025-12-14T00:00:00+01:00</published><updated>2025-12-14T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-12-14:/hosting-a-static-blog-on-freebsd-with-bastille-jails-and-automated-deployment/</id><summary type="html">&lt;p&gt;A full-stack overview of hosting a Pelican blog on FreeBSD 15.0 using Bastille jails, Caddy reverse proxy, and automated &lt;span class="caps"&gt;CI&lt;/span&gt;/&lt;span class="caps"&gt;CD&lt;/span&gt; deployment via Forgejo&amp;nbsp;Actions.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;Self-hosting a blog might seem like overkill in the age of managed platforms, but there&amp;#8217;s something deeply satisfying about controlling your entire stack. This article walks through how this very blog is hosted: a FreeBSD 15.0 server running multiple Bastille jails, with automated deployments triggered by git pushes to a self-hosted Forgejo instance. The setup prioritizes security through isolation, simplicity through static site generation, and reliability through&amp;nbsp;automation.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Safety deposit boxes" src="https://hofstede.it/images/2025-12-14-freebsd-blog-infrastructure.png" title="Jails: isolated compartments within a secure host"&gt;&lt;/p&gt;
&lt;h2 id="architecture-overview"&gt;Architecture&amp;nbsp;Overview&lt;/h2&gt;
&lt;p&gt;Before diving into the details, here&amp;#8217;s how the pieces fit&amp;nbsp;together:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                    Internet
                        |
                        v
            +-----------------------+
            |    PF Firewall        |
            |    NAT + RDR          |
            +-----------+-----------+
                        |
           +------------+------------+
           |                         |
           v                         v
    +-------------+           +-------------+
    | Caddy Jail  |           | Transporter |
    | TLS + Proxy |           |    Jail     |
    +------+------+           | rsync only  |
           |                  +------+------+
           v                         ^
    +-------------+                  |
    |  Blog Jail  |    nullfs mount  |
    |   Nginx     +------------------+
    +-------------+                  |
                              (rsync over SSH)
                                     |
                            +--------+--------+
                            | Forgejo Runner  |
                            | (remote server) |
                            +-----------------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Traffic flows from the internet through &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s packet filter, gets redirected to the Caddy jail for &lt;span class="caps"&gt;TLS&lt;/span&gt; termination, then proxied to the blog jail where Nginx serves static files. Deployments come in separately: a Forgejo runner builds the site and rsyncs it to the transporter jail, which has the blog&amp;#8217;s webroot mounted via&amp;nbsp;nullfs.&lt;/p&gt;
&lt;p&gt;The Forgejo Git-Forge installation is not covered in this article. This might be a topic for a future post&amp;nbsp;:-)&lt;/p&gt;
&lt;h2 id="the-host-system"&gt;The Host&amp;nbsp;System&lt;/h2&gt;
&lt;p&gt;The server runs FreeBSD 15.0-&lt;span class="caps"&gt;RELEASE&lt;/span&gt; on a &lt;span class="caps"&gt;VPS&lt;/span&gt; with dual-stack networking. Here&amp;#8217;s the relevant portion&amp;nbsp;of &lt;code&gt;/etc/rc.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;hostname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;server.example.com&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Security hardening&lt;/span&gt;
&lt;span class="nv"&gt;kern_securelevel_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;kern_securelevel&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Network setup - dual stack&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 192.0.2.10 netmask 255.255.252.0 -lro -tso&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8::2 prefixlen 68&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;192.0.2.1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fe80::1%vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Bastille jail network bridge&lt;/span&gt;
&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bridge0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.254.254.1/24&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:1000::1 prefixlen 68&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Services&lt;/span&gt;
&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;bastille_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Setting &lt;code&gt;kern_securelevel="2"&lt;/code&gt; is a simple but effective hardening measure. At securelevel 2, the system prevents loading kernel modules, writing to mounted filesystems marked immutable, and modifying the firewall rules without a reboot. This means even if an attacker gains root access, they can&amp;#8217;t disable &lt;span class="caps"&gt;PF&lt;/span&gt; or load a rootkit&amp;nbsp;module.&lt;/p&gt;
&lt;p&gt;The Bastille bridge network&amp;nbsp;(&lt;code&gt;bastille0&lt;/code&gt;) provides a private network for all jails. Each jail gets an &lt;span class="caps"&gt;IP&lt;/span&gt; from the 10.254.254.0/24 range for IPv4 and a corresponding address in the 2001:db8:1000::/68 range for&amp;nbsp;IPv6.&lt;/p&gt;
&lt;h2 id="jail-architecture"&gt;Jail&amp;nbsp;Architecture&lt;/h2&gt;
&lt;p&gt;Bastille manages three jails relevant to this&amp;nbsp;setup:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;all
&lt;span class="w"&gt; &lt;/span&gt;JID&lt;span class="w"&gt;  &lt;/span&gt;State&lt;span class="w"&gt;  &lt;/span&gt;IP&lt;span class="w"&gt; &lt;/span&gt;Address&lt;span class="w"&gt;       &lt;/span&gt;Hostname&lt;span class="w"&gt;       &lt;/span&gt;Release
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;Up&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.254.254.20&lt;span class="w"&gt;    &lt;/span&gt;blog&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.0-RELEASE
&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8:1000::20
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;Up&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.254.254.10&lt;span class="w"&gt;    &lt;/span&gt;caddy&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.0-RELEASE
&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8:1000::10
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;3&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;Up&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.254.254.25&lt;span class="w"&gt;    &lt;/span&gt;transporter&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="m"&gt;15&lt;/span&gt;.0-RELEASE
&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8:1000::25
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="the-caddy-jail"&gt;The Caddy&amp;nbsp;Jail&lt;/h3&gt;
&lt;p&gt;This jail handles all incoming &lt;span class="caps"&gt;HTTPS&lt;/span&gt; traffic. Caddy automatically obtains and renews Let&amp;#8217;s Encrypt certificates, terminates &lt;span class="caps"&gt;TLS&lt;/span&gt;, and proxies requests to the appropriate backend. It&amp;#8217;s the only jail that needs to be reachable from the internet on ports 80 and&amp;nbsp;443.&lt;/p&gt;
&lt;h3 id="the-blog-jail"&gt;The Blog&amp;nbsp;Jail&lt;/h3&gt;
&lt;p&gt;A minimal jail running Nginx to serve static files. The webroot&amp;nbsp;at &lt;code&gt;/usr/local/www/&lt;/code&gt; contains the generated Pelican output. The Nginx configuration is intentionally vanilla - just a&amp;nbsp;basic &lt;code&gt;server&lt;/code&gt; block pointing at the webroot with no special tuning required for static content. This jail has no public network exposure - all traffic comes through Caddy&amp;#8217;s reverse&amp;nbsp;proxy.&lt;/p&gt;
&lt;h3 id="the-transporter-jail"&gt;The Transporter&amp;nbsp;Jail&lt;/h3&gt;
&lt;p&gt;This is the clever part. The transporter jail exists solely to receive deployments. It runs &lt;span class="caps"&gt;SSH&lt;/span&gt; but with heavily restricted access: only rsync is allowed, only from the Forgejo runner&amp;#8217;s &lt;span class="caps"&gt;IP&lt;/span&gt;, and only to a specific directory. The magic happens through nullfs&amp;nbsp;mounts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# /usr/local/bastille/jails/transporter/fstab
/usr/local/bastille/jails/blog/root/usr/local/www/blog.example.com \
    /usr/local/bastille/jails/transporter/root/usr/local/deploy/blog nullfs 0 0
/usr/local/bastille/jails/blog/root/usr/local/www/staging.example.com \
    /usr/local/bastille/jails/transporter/root/usr/local/deploy/stageblog nullfs 0 0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;When the &lt;span class="caps"&gt;CI&lt;/span&gt; runner rsyncs files&amp;nbsp;to &lt;code&gt;/usr/local/deploy/blog/&lt;/code&gt; inside the transporter jail, they&amp;#8217;re actually being written directly to the blog jail&amp;#8217;s webroot. The transporter never stores anything - it&amp;#8217;s just a secure&amp;nbsp;gateway.&lt;/p&gt;
&lt;h2 id="pf-firewall-configuration"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; Firewall&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; ties everything together with &lt;span class="caps"&gt;NAT&lt;/span&gt; for outbound jail traffic and redirects for inbound&amp;nbsp;services:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# --- Macros ---
ext_if = &amp;quot;vtnet0&amp;quot;
jail_net = &amp;quot;10.254.254.0/24&amp;quot;
jail_net6 = &amp;quot;2001:db8:1000::/68&amp;quot;
host_ipv6 = &amp;quot;2001:db8::2&amp;quot;

frontend_v4 = &amp;quot;10.254.254.10&amp;quot;
frontend_v6 = &amp;quot;2001:db8:1000::10&amp;quot;
transport_v4 = &amp;quot;10.254.254.25&amp;quot;
transport_v6 = &amp;quot;2001:db8:1000::25&amp;quot;
forgejo_runner_v4 = &amp;quot;198.51.100.50&amp;quot;

# --- Tables ---
table &amp;lt;bruteforce&amp;gt; persist
table &amp;lt;jails_v4&amp;gt; { $jail_net }
table &amp;lt;jails_v6&amp;gt; { $jail_net6 }

# --- Options ---
set skip on lo0
set block-policy drop

# --- NAT for jail egress ---
nat on $ext_if inet from &amp;lt;jails_v4&amp;gt; to any -&amp;gt; ($ext_if)
nat on $ext_if inet6 from &amp;lt;jails_v6&amp;gt; to any -&amp;gt; $host_ipv6

# --- RDR for services ---
rdr pass on $ext_if inet proto tcp to ($ext_if) port {80,443} -&amp;gt; $frontend_v4
rdr pass on $ext_if inet6 proto tcp to $host_ipv6 port {80,443} -&amp;gt; $frontend_v6

# Deployment SSH - only from Forgejo runner
rdr pass on $ext_if inet proto tcp from $forgejo_runner_v4 \
    to ($ext_if) port 2225 -&amp;gt; $transport_v4 port 22

# --- Filtering ---
block quick from &amp;lt;bruteforce&amp;gt;
block drop in log all
block drop out log all

pass out quick all keep state
antispoof quick for { $ext_if, bastille0 }

# Essential ICMP
pass in inet proto icmp icmp-type { echoreq, unreach }
pass in quick inet6 proto ipv6-icmp icmp6-type { echoreq, echorep, neighbrsol, neighbradv }

# Jail egress
pass in quick on bastille0 from &amp;lt;jails_v4&amp;gt; to ! 10.254.254.0/24 keep state
pass in quick on bastille0 inet6 from &amp;lt;jails_v6&amp;gt; to ! 2001:db8:1000::/68 keep state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few things worth&amp;nbsp;noting:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Default deny&lt;/strong&gt;: Everything is blocked unless explicitly&amp;nbsp;allowed&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Deployment &lt;span class="caps"&gt;SSH&lt;/span&gt; on port 2225&lt;/strong&gt;: The transporter jail&amp;#8217;s &lt;span class="caps"&gt;SSH&lt;/span&gt; is only reachable from the Forgejo runner&amp;#8217;s &lt;span class="caps"&gt;IP&lt;/span&gt; address. Anyone else hitting that port gets silently&amp;nbsp;dropped.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Brute-force protection&lt;/strong&gt;:&amp;nbsp;The &lt;code&gt;&amp;lt;bruteforce&amp;gt;&lt;/code&gt; table can be populated by rules elsewhere (e.g., for the host&amp;#8217;s &lt;span class="caps"&gt;SSH&lt;/span&gt;) to automatically block repeat&amp;nbsp;offenders&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt; for jails&lt;/strong&gt;: Jails can reach the internet for package updates and certificate validation, but they appear to come from the host&amp;#8217;s public &lt;span class="caps"&gt;IP&lt;/span&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="the-deployment-pipeline"&gt;The Deployment&amp;nbsp;Pipeline&lt;/h2&gt;
&lt;p&gt;The deployment relies on an &lt;span class="caps"&gt;SSH&lt;/span&gt; keypair generated beforehand: the public key lives in the transporter&amp;nbsp;jail&amp;#8217;s &lt;code&gt;authorized_keys&lt;/code&gt; (with the rrsync restriction shown later), while the private key is stored as a secret in Forgejo. When I push to the blog&amp;#8217;s git repository, a Forgejo Actions workflow kicks&amp;nbsp;off:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;push&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;branches&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;#39;**&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;]&lt;/span&gt;

&lt;span class="nt"&gt;jobs&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;deploy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;runs-on&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;docker&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;container&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;python:3.11-slim&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Install Dependencies&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;apt-get update&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;apt-get install -y rsync openssh-client git&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;pip install pelican pelican-sitemap markdown typogrify&lt;/span&gt;

&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Checkout Code&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;actions/checkout@v3&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;submodules&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;recursive&lt;/span&gt;

&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Configure Target&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;id&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;meta&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;SHORT_SHA=$(git rev-parse --short HEAD)&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;if [[ &amp;quot;${{ github.ref }}&amp;quot; == &amp;quot;refs/heads/main&amp;quot; ]]; then&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="no"&gt;echo &amp;quot;url=https://blog.example.com&amp;quot; &amp;gt;&amp;gt; $GITHUB_OUTPUT&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="no"&gt;echo &amp;quot;dest=blog/&amp;quot; &amp;gt;&amp;gt; $GITHUB_OUTPUT&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;else&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="no"&gt;echo &amp;quot;url=https://staging.example.com/$SHORT_SHA&amp;quot; &amp;gt;&amp;gt; $GITHUB_OUTPUT&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="no"&gt;echo &amp;quot;dest=stageblog/$SHORT_SHA/&amp;quot; &amp;gt;&amp;gt; $GITHUB_OUTPUT&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;fi&lt;/span&gt;

&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Build Site&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;SITEURL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;${{ steps.meta.outputs.url }}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;pelican content -o output -s publishconf.py&lt;/span&gt;

&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Deploy via Rsync&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="no"&gt;rsync -avz --delete -e &amp;quot;ssh -p 2225&amp;quot; \&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="no"&gt;output/ deploy@server.example.com:${{ steps.meta.outputs.dest }}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The branch strategy is simple: pushes&amp;nbsp;to &lt;code&gt;main&lt;/code&gt; deploy to production, pushes to any other branch deploy to a staging &lt;span class="caps"&gt;URL&lt;/span&gt; with the commit hash in the path. This lets me preview changes before&amp;nbsp;merging.&lt;/p&gt;
&lt;p&gt;The transporter&amp;nbsp;jail&amp;#8217;s &lt;code&gt;authorized_keys&lt;/code&gt; file locks down what the deploy user can&amp;nbsp;do:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;command=&amp;quot;/usr/local/sbin/rrsync -wo /usr/local/deploy/&amp;quot;,no-agent-forwarding,no-port-forwarding,no-pty,no-user-rc,no-X11-forwarding ssh-ed25519 AAAA... deploy@runner
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;rrsync&lt;/code&gt; wrapper (restricted rsync) ensures the &lt;span class="caps"&gt;SSH&lt;/span&gt; key can only write&amp;nbsp;to &lt;code&gt;/usr/local/deploy/&lt;/code&gt; and nothing else. Combined with the &lt;span class="caps"&gt;PF&lt;/span&gt; rules limiting source IPs, this creates a minimal attack surface for&amp;nbsp;deployments.&lt;/p&gt;
&lt;h2 id="caddy-configuration"&gt;Caddy&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Caddy&amp;#8217;s configuration is refreshingly simple. Here&amp;#8217;s the production site&amp;nbsp;block:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;blog.example.com {
    reverse_proxy 10.254.254.20:80  # blog jail

    header {
        Strict-Transport-Security &amp;quot;max-age=31536000; includeSubDomains; preload&amp;quot;
        X-Content-Type-Options &amp;quot;nosniff&amp;quot;
        X-Frame-Options &amp;quot;DENY&amp;quot;
        Referrer-Policy &amp;quot;strict-origin-when-cross-origin&amp;quot;
    }

    log {
        output file /var/log/caddy/blog.access.log {
            roll_size 100mb
            roll_keep 10
        }
        format json
    }
}

staging.example.com {
    reverse_proxy 10.254.254.20:80

    header {
        Strict-Transport-Security &amp;quot;max-age=31536000; includeSubDomains; preload&amp;quot;
        X-Content-Type-Options &amp;quot;nosniff&amp;quot;
        X-Frame-Options &amp;quot;DENY&amp;quot;
        Referrer-Policy &amp;quot;strict-origin-when-cross-origin&amp;quot;
    }
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Caddy handles certificate management automatically via &lt;span class="caps"&gt;ACME&lt;/span&gt;. The security headers provide defense-in-depth: &lt;span class="caps"&gt;HSTS&lt;/span&gt; ensures browsers always use &lt;span class="caps"&gt;HTTPS&lt;/span&gt;, X-Frame-Options prevents clickjacking, and the other headers mitigate various web attacks. For a static blog these might seem excessive, but they cost nothing and establish good&amp;nbsp;habits.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;This setup might look complex at first glance, but each piece serves a&amp;nbsp;purpose:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Bastille jails&lt;/strong&gt; provide lightweight isolation without the overhead of full&amp;nbsp;VMs&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;PF&lt;/span&gt;&lt;/strong&gt; gives fine-grained control over traffic flow and enforces the principle of least&amp;nbsp;privilege&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The transporter pattern&lt;/strong&gt; allows secure deployments without exposing the blog jail&amp;nbsp;directly&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Forgejo Actions&lt;/strong&gt; automates the entire build-and-deploy&amp;nbsp;cycle&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Caddy&lt;/strong&gt; simplifies &lt;span class="caps"&gt;TLS&lt;/span&gt; and reverse proxying to a few lines of&amp;nbsp;config&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The result is a blog that deploys automatically on every git push, runs in isolated jails with minimal attack surface, and costs just a few euros per month on a small &lt;span class="caps"&gt;VPS&lt;/span&gt;. FreeBSD&amp;#8217;s jails remain one of the most elegant isolation mechanisms available - simpler than containers, more lightweight than VMs, and battle-tested over two&amp;nbsp;decades.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re considering self-hosting, I&amp;#8217;d encourage giving FreeBSD and Bastille a try. The learning curve is worth&amp;nbsp;it.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook:&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastillebsd.org"&gt;Bastille&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://caddyserver.com/docs/"&gt;Caddy web&amp;nbsp;server&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.getpelican.com/"&gt;Pelican static site&amp;nbsp;generator&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://forgejo.org/docs/latest/user/actions/"&gt;Forgejo&amp;nbsp;Actions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.openbsd.org/faq/pf/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; - The OpenBSD Packet&amp;nbsp;Filter&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;Thanks to everyone behind FreeBSD, Bastille, Caddy, Pelican, Forgejo, and the countless other open source projects that make setups like this possible. Standing on the shoulders of giants has never been&amp;nbsp;easier.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="jails"/><category term="deployment"/><category term="caddy"/><category term="ci-cd"/><category term="bastille"/><category term="nginx"/><category term="infrastructure"/></entry><entry><title>Migrating burningboard.net Mastodon instance to a Multi-Jail FreeBSD Setup</title><link href="https://hofstede.it/migrating-burningboardnet-mastodon-instance-to-a-multi-jail-freebsd-setup/" rel="alternate"/><published>2025-12-07T00:00:00+01:00</published><updated>2025-12-07T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-12-07:/migrating-burningboardnet-mastodon-instance-to-a-multi-jail-freebsd-setup/</id><summary type="html">&lt;p&gt;Migrating a Mastodon instance to FreeBSD with BastilleBSD - a multi-jail architecture with aggressive service separation, centralized &lt;span class="caps"&gt;PF&lt;/span&gt; firewalling, and a fully dual-stack network&amp;nbsp;design.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Over the last few weeks, I’ve been working on migrating our Mastodon instance &lt;strong&gt;burningboard.net&lt;/strong&gt; from its current Linux host to a modular FreeBSD jail-based setup powered by &lt;a href="https://bastillebsd.org"&gt;BastilleBSD&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;This post walks through the architecture and design rationale of my new multi-jail Mastodon system, with aggressive separation of concerns, centralized firewalling, and a fully dual-stack network&amp;nbsp;design.&lt;/p&gt;
&lt;h2 id="acknowledgements"&gt;Acknowledgements&lt;/h2&gt;
&lt;p&gt;This work is based on the excellent post by &lt;strong&gt;Stefano Marinelli&lt;/strong&gt;:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;a href="https://it-notes.dragas.net/2022/11/23/installing-mastodon-on-a-freebsd-jail/"&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Installing Mastodon on a FreeBSD&amp;nbsp;jail&amp;#8221;&lt;/a&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Stefano’s article inspired me to try Mastodon on FreeBSD. My implementation takes that foundation and extends it for a more maintainable, production-ready&amp;nbsp;architecture.&lt;/p&gt;
&lt;h2 id="design-goals"&gt;Design&amp;nbsp;Goals&lt;/h2&gt;
&lt;p&gt;The motivation behind this&amp;nbsp;move:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Central &lt;span class="caps"&gt;PF&lt;/span&gt; firewall&lt;/strong&gt; – all filtering, &lt;span class="caps"&gt;NAT&lt;/span&gt;, and routing are handled by the host only. Jails see a clean, local L2 view - no &lt;span class="caps"&gt;PF&lt;/span&gt; inside jails, no double &lt;span class="caps"&gt;NAT&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Separation of concerns&lt;/strong&gt; – every jail runs exactly one functional service:&lt;ul&gt;
&lt;li&gt;&lt;code&gt;nginx&lt;/code&gt; — reverse proxy + &lt;span class="caps"&gt;TLS&lt;/span&gt;&amp;nbsp;termination  &lt;/li&gt;
&lt;li&gt;&lt;code&gt;mastodonweb&lt;/code&gt; — Puma / Rails web&amp;nbsp;backend  &lt;/li&gt;
&lt;li&gt;&lt;code&gt;mastodonsidekiq&lt;/code&gt; — background&amp;nbsp;jobs  &lt;/li&gt;
&lt;li&gt;&lt;code&gt;database&lt;/code&gt; — PostgreSQL and Valkey (Redis&amp;nbsp;fork)&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Host‑managed source&lt;/strong&gt; – Mastodon source tree shared&amp;nbsp;via &lt;code&gt;nullfs&lt;/code&gt; between web and sidekiq jails.&amp;nbsp;Common &lt;code&gt;.env.production&lt;/code&gt;, shared dependencies, single codebase to&amp;nbsp;maintain.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Clean dual‑stack (IPv4 + IPv6)&lt;/strong&gt; – every component visible under both protocols; no &lt;span class="caps"&gt;NAT66&lt;/span&gt; or translation&amp;nbsp;hacks.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Predictable networking&lt;/strong&gt; – each functional group lives on its own bridge with private address&amp;nbsp;space.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="jail-and-network-overview"&gt;Jail and Network&amp;nbsp;Overview&lt;/h2&gt;
&lt;p&gt;Example address plan (using &lt;span class="caps"&gt;RFC&lt;/span&gt; 5737 and 3849 documentation&amp;nbsp;spaces):&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Jail&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;th&gt;IPv4&lt;/th&gt;
&lt;th&gt;IPv6&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;nginx&lt;/td&gt;
&lt;td&gt;Reverse proxy&lt;/td&gt;
&lt;td&gt;192.0.2.13&lt;/td&gt;
&lt;td&gt;2001:db8:8000::13&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;mastodonweb&lt;/td&gt;
&lt;td&gt;Rails backend&lt;/td&gt;
&lt;td&gt;198.51.100.9&lt;/td&gt;
&lt;td&gt;2001:db8:9000::9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;mastodonsidekiq&lt;/td&gt;
&lt;td&gt;Workers&lt;/td&gt;
&lt;td&gt;198.51.100.8&lt;/td&gt;
&lt;td&gt;2001:db8:b000::8&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;database&lt;/td&gt;
&lt;td&gt;PostgreSQL + Valkey&lt;/td&gt;
&lt;td&gt;198.51.100.6&lt;/td&gt;
&lt;td&gt;2001:db8:a000::6&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Host&lt;/td&gt;
&lt;td&gt;&amp;#8220;burningboard.example.net&amp;#8221;&lt;/td&gt;
&lt;td&gt;203.0.113.1&lt;/td&gt;
&lt;td&gt;2001:db8::f3d1&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Each functional bucket gets its&amp;nbsp;own &lt;code&gt;bridge(4)&lt;/code&gt; interface on the host&amp;nbsp;(&lt;code&gt;bastille0&lt;/code&gt;..&lt;code&gt;bastille3&lt;/code&gt;) and its&amp;nbsp;own &lt;code&gt;/24&lt;/code&gt; and &lt;code&gt;/64&lt;/code&gt; subnet.&lt;br&gt;
Jails are created and attached to the corresponding&amp;nbsp;bridge.&lt;/p&gt;
&lt;h2 id="schematic-diagram"&gt;Schematic&amp;nbsp;diagram&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[ Internet ]
     |
     v
 [ PF Host ]
     ├── bridge0 — nginx (192.0.2.13 / 2001:db8:8000::13)
     ├── bridge1 — mastodonweb (198.51.100.9 / 2001:db8:9000::9)
     ├── bridge2 — database (198.51.100.6 / 2001:db8:a000::6)
     └── bridge3 — sidekiq (198.51.100.8 / 2001:db8:b000::8)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With the address plan established, the next step is creating the individual jails and assigning virtual network&amp;nbsp;interfaces.&lt;/p&gt;
&lt;h2 id="jail-creation-and-perjail-configuration"&gt;Jail Creation and Per‑Jail&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;Each jail was created directly through Bastille using &lt;span class="caps"&gt;VNET&lt;/span&gt; support, attaching it to its respective bridge.&lt;br&gt;
For example, creating&amp;nbsp;the &lt;code&gt;nginx&lt;/code&gt; frontend jail on&amp;nbsp;the &lt;code&gt;bastille0&lt;/code&gt; bridge:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;bastille&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-B&lt;span class="w"&gt; &lt;/span&gt;nginx&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;14&lt;/span&gt;.3-RELEASE&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.13&lt;span class="w"&gt; &lt;/span&gt;bastille0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Bastille automatically provisions a &lt;span class="caps"&gt;VNET&lt;/span&gt; interface inside the jail&amp;nbsp;(&lt;code&gt;vnet0&lt;/code&gt;) and associates it with the corresponding bridge on the host.&lt;br&gt;
Inside each jail,&amp;nbsp;the &lt;code&gt;/etc/rc.conf&lt;/code&gt; defines its own network interface, IPv4/IPv6 addresses, default routes, and any service daemons enabled for that&amp;nbsp;jail.&lt;/p&gt;
&lt;p&gt;Example configuration for&amp;nbsp;the &lt;code&gt;database&lt;/code&gt; jail (substituted with documentation&amp;nbsp;addresses):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;ifconfig_e0b_database_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vnet0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 198.51.100.6 netmask 255.255.255.0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:a000::6/64&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_descr&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;database jail interface on bastille2&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;198.51.100.1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:a000::1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;syslogd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-ss&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_submit_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_outbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_msp_queue_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;cron_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-J 60&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;valkey_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;postgresql_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Each jail is therefore a &lt;strong&gt;fully self‑contained FreeBSD environment&lt;/strong&gt; with native rc(8) configuration, its own routing table, and service definition. Bastille’s role ends at boot‑time network attachment - the rest is standard FreeBSD&amp;nbsp;administration.&lt;/p&gt;
&lt;h2 id="host-etcrcconf"&gt;Host &lt;code&gt;/etc/rc.conf&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;Below is a simplified version of the host configuration that ties everything together.&lt;br&gt;
Each jail bridge subnet is assigned both IPv4 and IPv6 space; the host acts as&amp;nbsp;gateway.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Basic host config&lt;/span&gt;
&lt;span class="nv"&gt;hostname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;burningboard.example.net&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;keymap&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;us.kbd&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Networking&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 203.0.113.1 netmask 255.255.255.255&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8::f3d1/64&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;203.0.113.254&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8::1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Bridges for jails&lt;/span&gt;
&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bridge0 bridge1 bridge2 bridge3&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge1_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge2_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille2&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge3_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille3&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Bridge interfaces for individual networks&lt;/span&gt;

&lt;span class="c1"&gt;# Frontend (nginx)&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 192.0.2.1/24&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:8000::1/64&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Mastodon Web (Rails / Puma)&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille1&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 198.51.100.1/24&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille1_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:9000::1/64&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Database&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille2&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 198.51.100.2/24&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille2_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:a000::1/64&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Sidekiq (workers)&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille3&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 198.51.100.3/24&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille3_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:b000::1/64&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Services&lt;/span&gt;
&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pflog_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;bastille_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sshd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ntpd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ntpd_sync_on_start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This provides proper L3 separation for each functional group.&lt;br&gt;
In this&amp;nbsp;layout, &lt;code&gt;bastille0&lt;/code&gt; →&amp;nbsp;frontend, &lt;code&gt;bastille1&lt;/code&gt; →&amp;nbsp;app, &lt;code&gt;bastille2&lt;/code&gt; → &lt;span class="caps"&gt;DB&lt;/span&gt;, &lt;code&gt;bastille3&lt;/code&gt; → worker&amp;nbsp;pool.&lt;/p&gt;
&lt;h2 id="etcpfconf"&gt;&lt;code&gt;/etc/pf.conf&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;The host firewall serves the dual purpose of &lt;span class="caps"&gt;NAT&lt;/span&gt; gateway and service ingress&amp;nbsp;controller.&lt;/p&gt;
&lt;p&gt;Below is an anonymized but structurally identical&amp;nbsp;configuration.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# --- Macros ---&lt;/span&gt;
&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;198.51.100.0/20&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:8000::/64&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8::f3d1&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;frontend_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;192.0.2.13&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;frontend_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:8000::13&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Trusted management networks (example)&lt;/span&gt;
&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;{ 203.0.113.42, 192.0.2.222 }&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;{ 2001:db8:beef::/64 }&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;

&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;skip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lo0&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loginterface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;

&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fragment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;reassemble&lt;/span&gt;
&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;mss&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1500&lt;/span&gt;

&lt;span class="c1"&gt;# --- NAT ---&lt;/span&gt;
&lt;span class="c1"&gt;# Jails -&amp;gt; egress internet (IPv4)&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- Port redirection ---&lt;/span&gt;
&lt;span class="c1"&gt;# Incoming HTTP/HTTPS -&amp;gt; nginx jail&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v4&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v6&lt;/span&gt;

&lt;span class="c1"&gt;# --- Filtering policy ---&lt;/span&gt;

&lt;span class="c1"&gt;# Default deny (log for audit)&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;

&lt;span class="c1"&gt;# Allow existing stateful flows out&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Allow management SSH (example port 30822) only from trusted subnets&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;30822&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;30822&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Block all other SSH&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;30822&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh_blocked&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# ICMP/ICMPv6 essentials&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unreach&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echorep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbrsol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbradv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;toobig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;timex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paramprob&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Inter-jail traffic&lt;/span&gt;
&lt;span class="c1"&gt;# nginx -&amp;gt; mastodonweb&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.13&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;198.51&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;100.9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;3000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;4000&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;13&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;9000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;3000&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;4000&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# mastodonweb -&amp;gt; database (Postgres + Valkey)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;198.51&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;100.9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;198.51&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;100.6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;5432&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;6379&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;9000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;a000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;5432&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;6379&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# sidekiq -&amp;gt; database&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;198.51&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;100.8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;198.51&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;100.6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;5432&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;6379&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;b000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;a000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;5432&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;6379&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Optional: temporary egress blocking during testing&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;

&lt;span class="c1"&gt;# External access&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;frontend_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This &lt;span class="caps"&gt;PF&lt;/span&gt; configuration centralizes control at the host. The jails have no firewall logic - just clean &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;connectivity.&lt;/p&gt;
&lt;h2 id="shared-source-design"&gt;Shared Source&amp;nbsp;Design&lt;/h2&gt;
&lt;p&gt;Both &lt;code&gt;mastodonweb&lt;/code&gt; and &lt;code&gt;mastodonsidekiq&lt;/code&gt; jails&amp;nbsp;mount &lt;code&gt;/usr/local/mastodon&lt;/code&gt; from the&amp;nbsp;host:&lt;/p&gt;
&lt;p&gt;/usr/local/mastodon -&amp;gt;&amp;nbsp;/usr/local/bastille/jails/mastodonweb/root/usr/home/mastodon&lt;/p&gt;
&lt;p&gt;/usr/local/mastodon -&amp;gt;&amp;nbsp;/usr/local/bastille/jails/mastodonsidekiq/root/usr/home/mastodon&lt;/p&gt;
&lt;p&gt;Example &lt;code&gt;fstab&lt;/code&gt; entry:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/usr/local/mastodon /usr/local/bastille/jails/mastodonweb/root/usr/home/mastodon nullfs rw 0 0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That way, only one source tree needs updates after a git pull or bundle/yarn operation. The jails simply see the current state of that&amp;nbsp;directory.&lt;/p&gt;
&lt;p&gt;Logs and tmp directories are symlinked to /var/log/mastodon and /var/tmp/mastodon inside each jail for persistence and&amp;nbsp;cleanup.&lt;/p&gt;
&lt;h2 id="service-boot-integration"&gt;Service Boot&amp;nbsp;Integration&lt;/h2&gt;
&lt;p&gt;Each Mastodon jail defines lightweight /usr/local/etc/rc.d&amp;nbsp;scripts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# PROVIDE: mastodon_web&lt;/span&gt;
&lt;span class="c1"&gt;# KEYWORD: shutdown&lt;/span&gt;

.&lt;span class="w"&gt; &lt;/span&gt;/etc/rc.subr

&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastodon_web&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;rcvar&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;mastodon_web_enable
&lt;span class="nv"&gt;pidfile&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/run/mastodon/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;start_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastodon_web_start&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;stop_cmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastodon_web_stop&amp;quot;&lt;/span&gt;

mastodon_web_start&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon
&lt;span class="w"&gt;    &lt;/span&gt;chown&lt;span class="w"&gt; &lt;/span&gt;mastodon:mastodon&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon
&lt;span class="w"&gt;    &lt;/span&gt;su&lt;span class="w"&gt; &lt;/span&gt;mastodon&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;export PATH=/usr/local/bin:/usr/bin:/bin; \&lt;/span&gt;
&lt;span class="s2"&gt;        export RAILS_ENV=production; export PORT=3000; \&lt;/span&gt;
&lt;span class="s2"&gt;        cd /home/mastodon/live &amp;amp;&amp;amp; \&lt;/span&gt;
&lt;span class="s2"&gt;        /usr/sbin/daemon -T &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; -P /var/run/mastodon/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;_supervisor.pid \&lt;/span&gt;
&lt;span class="s2"&gt;        -p /var/run/mastodon/&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.pid -f -S -r \&lt;/span&gt;
&lt;span class="s2"&gt;        /usr/local/bin/bundle exec puma -C config/puma.rb&amp;quot;&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;

mastodon_web_stop&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-9&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon/&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;_supervisor.pid&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;kill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-15&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sb"&gt;`&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/var/run/mastodon/&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;name&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;.pid&lt;span class="sb"&gt;`&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null
&lt;span class="o"&gt;}&lt;/span&gt;

load_rc_config&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$name&lt;/span&gt;
run_rc_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Equivalent scripts exist for mastodon_streaming and the Sidekiq&amp;nbsp;worker.&lt;/p&gt;
&lt;p&gt;Everything integrates seamlessly with FreeBSD’s native service&amp;nbsp;management:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;service&lt;span class="w"&gt; &lt;/span&gt;mastodon_web&lt;span class="w"&gt; &lt;/span&gt;start
service&lt;span class="w"&gt; &lt;/span&gt;mastodon_streaming&lt;span class="w"&gt; &lt;/span&gt;restart
service&lt;span class="w"&gt; &lt;/span&gt;mastodonsidekiq&lt;span class="w"&gt; &lt;/span&gt;status
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;No Docker, no systemd, no exotic process&amp;nbsp;supervisors.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="why-it-matters"&gt;Why It&amp;nbsp;Matters&lt;/h2&gt;
&lt;p&gt;The resulting system is simple, observable, and&amp;nbsp;robust:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Firewall rules are centralized and&amp;nbsp;auditable.&lt;/li&gt;
&lt;li&gt;Each jail is a clean service container (pure FreeBSD primitives, no overlay&amp;nbsp;complexity).&lt;/li&gt;
&lt;li&gt;IPv4/IPv6 connectivity is symmetrical and&amp;nbsp;clear.&lt;/li&gt;
&lt;li&gt;Source and configs are under full administrator control, not hidden in&amp;nbsp;containers.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;It’s also easy to snapshot with &lt;span class="caps"&gt;ZFS&lt;/span&gt; or promote new releases jail-by-jail using Bastille’s clone/deploy&amp;nbsp;model.&lt;/p&gt;
&lt;h2 id="summary"&gt;Summary&lt;/h2&gt;
&lt;p&gt;In&amp;nbsp;short:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Host does &lt;span class="caps"&gt;PF&lt;/span&gt;, routing, &lt;span class="caps"&gt;NAT&lt;/span&gt;,&amp;nbsp;bridges&lt;/li&gt;
&lt;li&gt;Each jail has exactly one&amp;nbsp;purpose&lt;/li&gt;
&lt;li&gt;Source code lives once on the&amp;nbsp;host&lt;/li&gt;
&lt;li&gt;Dual-stack networking, no&amp;nbsp;translation&lt;/li&gt;
&lt;li&gt;Everything&amp;nbsp;FreeBSD-native&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This structure makes it easy to reason about - each moving part has one&amp;nbsp;job. &lt;/p&gt;
&lt;p&gt;That’s how I like my infrastructure: boringly&amp;nbsp;reliable.&lt;/p&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://it-notes.dragas.net/2022/11/23/installing-mastodon-on-a-freebsd-jail/"&gt;Stefano Marinelli — Installing Mastodon on a FreeBSD&amp;nbsp;jail&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastillebsd.org/"&gt;The BastilleBSD&amp;nbsp;Project&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook –&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://datatracker.ietf.org/doc/html/rfc3849"&gt;&lt;span class="caps"&gt;RFC&lt;/span&gt; 3849 – IPv6 Documentation&amp;nbsp;Prefix&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://datatracker.ietf.org/doc/html/rfc5737"&gt;&lt;span class="caps"&gt;RFC&lt;/span&gt; 5737 – IPv4 Documentation&amp;nbsp;Space&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="networking"/><category term="mastodon"/><category term="ipv6"/></entry><entry><title>FreeBSD Dual-Stack Jails on Hetzner Cloud</title><link href="https://hofstede.it/freebsd-dual-stack-jails-on-hetzner-cloud/" rel="alternate"/><published>2025-11-12T00:00:00+01:00</published><updated>2025-11-12T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-11-12:/freebsd-dual-stack-jails-on-hetzner-cloud/</id><summary type="html">&lt;p&gt;A reproducible dual-stack configuration for FreeBSD on Hetzner Cloud with &lt;span class="caps"&gt;VNET&lt;/span&gt; jails, &lt;span class="caps"&gt;PF&lt;/span&gt; &lt;span class="caps"&gt;NAT&lt;/span&gt; for IPv4, and a /65 split trick to give jails native globally-routable IPv6 from a single&amp;nbsp;/64.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Hetzner Cloud networking has a few quirks that complicate otherwise standard FreeBSD&amp;nbsp;setups:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;IPv4 arrives as a /32 with a pseudo on-link next-hop at&amp;nbsp;172.31.1.1.&lt;/li&gt;
&lt;li&gt;You typically get exactly one routed IPv6 /64 per&amp;nbsp;server.&lt;/li&gt;
&lt;li&gt;There’s no shared L2 domain; your &lt;span class="caps"&gt;VM&lt;/span&gt; doesn’t &lt;span class="caps"&gt;ARP&lt;/span&gt;/&lt;span class="caps"&gt;NDP&lt;/span&gt; for other&amp;nbsp;tenants.&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;NIC&lt;/span&gt; offload features can misbehave with &lt;span class="caps"&gt;VNET&lt;/span&gt; + bridges on&amp;nbsp;FreeBSD.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This guide shows a reproducible dual-stack configuration for a FreeBSD 14 host with &lt;span class="caps"&gt;VNET&lt;/span&gt; jails managed by Bastille. The&amp;nbsp;pattern:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;IPv4: &lt;span class="caps"&gt;RFC1918&lt;/span&gt; on the jail bridge; &lt;span class="caps"&gt;NAT&lt;/span&gt; to host’s public&amp;nbsp;IPv4.&lt;/li&gt;
&lt;li&gt;IPv6: Split the single routed /64 into two /65s: one for the host, one routed to the jails (no &lt;span class="caps"&gt;ULA&lt;/span&gt;, no &lt;span class="caps"&gt;NAT66&lt;/span&gt;).&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; for IPv4 &lt;span class="caps"&gt;NAT&lt;/span&gt; and port forwards; pure routing for&amp;nbsp;IPv6.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All addresses below use documentation&amp;nbsp;ranges:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Host IPv4:&amp;nbsp;203.0.113.10/32&lt;/li&gt;
&lt;li&gt;Hetzner pseudo-gateway (IPv4):&amp;nbsp;172.31.1.1&lt;/li&gt;
&lt;li&gt;Assigned IPv6 /64:&amp;nbsp;2001:db8:abcd:1000::/64&lt;/li&gt;
&lt;li&gt;Host half (/65):&amp;nbsp;2001:db8:abcd:1000::/65&lt;/li&gt;
&lt;li&gt;Jails half (/65):&amp;nbsp;2001:db8:abcd:1000:8000::/65&lt;/li&gt;
&lt;li&gt;Bastille bridge IPv4:&amp;nbsp;10.100.0.1/24&lt;/li&gt;
&lt;li&gt;Jail example IPv4:&amp;nbsp;10.100.0.100/24&lt;/li&gt;
&lt;li&gt;Host IPv6 (example):&amp;nbsp;2001:db8:abcd:1000::10/65&lt;/li&gt;
&lt;li&gt;Bridge IPv6 (gateway for jails):&amp;nbsp;2001:db8:abcd:1000:8000::1/65&lt;/li&gt;
&lt;li&gt;Jail example IPv6:&amp;nbsp;2001:db8:abcd:1000:8000::100/65&lt;/li&gt;
&lt;/ul&gt;
&lt;blockquote&gt;
&lt;p&gt;Why split the /64 into two&amp;nbsp;/65s?&lt;/p&gt;
&lt;p&gt;Hetzner Cloud gives you a single routed /64. To put global IPv6 addresses in jails without &lt;span class="caps"&gt;NAT66&lt;/span&gt;/&lt;span class="caps"&gt;ULA&lt;/span&gt;, split that /64:
- Keep the lower /65 on the host interface.
- Route the upper /65 to your jail bridge and use it for jails.
This mirrors the “host + routed downstream” model and keeps IPv6 fully&amp;nbsp;end-to-end.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;hr&gt;
&lt;h2 id="0-design-philosophy"&gt;0) Design&amp;nbsp;Philosophy&lt;/h2&gt;
&lt;p&gt;This guide follows modern networking&amp;nbsp;principles:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;IPv6 is &lt;span class="caps"&gt;IP&lt;/span&gt;.&lt;/strong&gt; IPv4 is the legacy compatibility&amp;nbsp;layer.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No &lt;span class="caps"&gt;NAT66&lt;/span&gt;.&lt;/strong&gt; &lt;span class="caps"&gt;NAT&lt;/span&gt; was an IPv4 workaround for address exhaustion. 
  IPv6 has 340 undecillion addresses - use&amp;nbsp;them.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;End-to-end connectivity.&lt;/strong&gt; Jails get real, globally routable 
  IPv6 addresses. Firewalling provides security, not address&amp;nbsp;translation.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;IPv4 best-effort.&lt;/strong&gt; &lt;span class="caps"&gt;NAT&lt;/span&gt; is acceptable for IPv4 because we have 
  no choice. It&amp;#8217;s temporary&amp;nbsp;infrastructure.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If this offends you, you&amp;#8217;re reading the wrong guide.&amp;nbsp;🙂&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="1-host-etcrcconf"&gt;1) Host&amp;nbsp;/etc/rc.conf&lt;/h2&gt;
&lt;p&gt;Configure the /32 IPv4 with the on-link host route to the pseudo gateway, enable forwarding, define the Bastille bridge, and assign both v4 and v6 (using the /65&amp;nbsp;split).&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;hostname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;freebsd-hcloud&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;keymap&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;us.kbd&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Forwarding for jails&lt;/span&gt;
&lt;span class="nv"&gt;gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Hetzner Cloud NIC: /32 IPv4 and routed IPv6&lt;/span&gt;
&lt;span class="c1"&gt;# Disable LRO/TSO to avoid VNET/bridge issues&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 203.0.113.10 netmask 255.255.255.255 -tso -lro&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:abcd:1000::10/65&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# IPv4 pseudo-gateway and default route&lt;/span&gt;
&lt;span class="nv"&gt;static_routes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;hcloud default&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;route_hcloud&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-host 172.31.1.1 -interface vtnet0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;route_default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;default 172.31.1.1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# IPv6 default router (Hetzner uses link-local on your NIC)&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fe80::1%vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Bastille VNET bridge&lt;/span&gt;
&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bridge0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.100.0.1/24&amp;quot;&lt;/span&gt;
&lt;span class="c1"&gt;# Routed upper /65 goes to the bridge (acts as gateway for jails)&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:abcd:1000:8000::1/65&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Services&lt;/span&gt;
&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pflog_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sshd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ntpd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ntpd_sync_on_start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Notes:&amp;nbsp;- &lt;code&gt;-tso -lro&lt;/code&gt; on vtnet0 avoids checksum/segmentation oddities with bridged epairs.
- The host’s IPv6 address lives in the lower /65; the bridge gets a gateway address in the upper&amp;nbsp;/65.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="2-etcsysctlconf"&gt;2)&amp;nbsp;/etc/sysctl.conf&lt;/h2&gt;
&lt;p&gt;Ensure forwarding is on and (optionally) disable&amp;nbsp;redirects.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;net.inet.ip.forwarding&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
net.inet6.ip6.forwarding&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
net.inet.ip.redirect&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
net.inet6.ip6.redirect&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr&gt;
&lt;h2 id="3-pf-etcpfconf"&gt;3) &lt;span class="caps"&gt;PF&lt;/span&gt;:&amp;nbsp;/etc/pf.conf&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt; IPv4 from the jail network to the host’s public IPv4. IPv6 is routed (no &lt;span class="caps"&gt;NAT&lt;/span&gt;). Forward &lt;span class="caps"&gt;HTTP&lt;/span&gt;/&lt;span class="caps"&gt;HTTPS&lt;/span&gt; to a specific jail as an example. Lock down &lt;span class="caps"&gt;SSH&lt;/span&gt; to trusted&amp;nbsp;sources.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# --- Macros ---&lt;/span&gt;
&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Jail networks&lt;/span&gt;
&lt;span class="n"&gt;jail_net_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.100.0.0/24&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;jail_net_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:abcd:1000:8000::/65&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Host IPv6 on the external interface (lower /65)&lt;/span&gt;
&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:abcd:1000::10&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Example trusted admin sources&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;198.51&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;100.22&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;203.0&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;113.50&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;ffff&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Brute-force tracking&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;

&lt;span class="c1"&gt;# --- Options ---&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;skip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lo0&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loginterface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;

&lt;span class="c1"&gt;# --- Scrub ---&lt;/span&gt;
&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fragment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;reassemble&lt;/span&gt;
&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;mss&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1500&lt;/span&gt;

&lt;span class="c1"&gt;# --- NAT (IPv4 only) ---&lt;/span&gt;
&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# --- RDR for public services to a &amp;quot;web&amp;quot; jail ---&lt;/span&gt;
&lt;span class="c1"&gt;# v4: to host IPv4, forward 80/443 to the jail v4&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;10.100&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;0.100&lt;/span&gt;
&lt;span class="c1"&gt;# v6: to the host&amp;#39;s IPv6, forward 80/443 to the jail v6&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;abcd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;

&lt;span class="c1"&gt;# --- Policy ---&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;

&lt;span class="c1"&gt;# Allow established and all outbound by default&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Anti-spoof&lt;/span&gt;
&lt;span class="n"&gt;antispoof&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# SSH allowlist (example nonstandard port 22222)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22222&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22222&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# Log and drop other SSH attempts&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22222&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh_blocked&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Essential ICMPv6 (ND, pMTU, echo)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echorep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbrsol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbradv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;toobig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;timex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paramprob&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Useful ICMPv4 (echo, unreach for pMTU)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unreach&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Allow from jails bridge (host-facing side)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="c1"&gt;# Public HTTP/HTTPS to forwarded jail&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr&gt;
&lt;h2 id="4-bastillejail-networking"&gt;4) Bastille/Jail&amp;nbsp;networking&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;VNET&lt;/span&gt; jails get an epair; the host end attaches&amp;nbsp;to &lt;code&gt;bastille0&lt;/code&gt;. Inside the jail, rename the epair&amp;nbsp;to &lt;code&gt;vnet0&lt;/code&gt; and assign addresses. The bridge addresses act as the default&amp;nbsp;gateways.&lt;/p&gt;
&lt;p&gt;Example&amp;nbsp;jail &lt;code&gt;rc.conf&lt;/code&gt; (inside the&amp;nbsp;jail):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;ifconfig_e0b_webjail_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vnet0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.100.0.100 netmask 255.255.255.0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:abcd:1000:8000::100/65&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.100.0.1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:abcd:1000:8000::1&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Optional: quiet base daemons&lt;/span&gt;
&lt;span class="nv"&gt;syslogd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-ss&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_submit_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_outbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_msp_queue_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If templating with Bastille, a minimal jail.conf&amp;nbsp;fragment:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;vnet=&amp;quot;on&amp;quot;;
vnet.interface=&amp;quot;e0a_${name}&amp;quot;;

exec.prestart=&amp;quot;ifconfig e0a_${name} create up&amp;quot;;
exec.prestart=&amp;quot;ifconfig bastille0 addm e0a_${name}&amp;quot;;

exec.start=&amp;quot;/bin/sh /etc/rc&amp;quot;;
exec.stop=&amp;quot;/bin/sh /etc/rc.shutdown&amp;quot;;

mount.devfs;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Bastille will hand the jail end&amp;nbsp;as &lt;code&gt;e0b_${name}&lt;/code&gt;, which you rename&amp;nbsp;to &lt;code&gt;vnet0&lt;/code&gt; in the jail’s&amp;nbsp;rc.conf.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="5-bring-up-and-verification"&gt;5) Bring-up and&amp;nbsp;verification&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Reload &lt;span class="caps"&gt;PF&lt;/span&gt; and confirm&amp;nbsp;rules:&lt;/li&gt;
&lt;li&gt;&lt;code&gt;service pf restart&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;pfctl -sr -v&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Verify routes on the&amp;nbsp;host:&lt;/li&gt;
&lt;li&gt;IPv4 default&amp;nbsp;via &lt;code&gt;172.31.1.1&lt;/code&gt; on &lt;code&gt;vtnet0&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;IPv6 default&amp;nbsp;via &lt;code&gt;fe80::1%vtnet0&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Local routes&amp;nbsp;to &lt;code&gt;10.100.0.0/24&lt;/code&gt; and &lt;code&gt;2001:db8:abcd:1000:8000::/65&lt;/code&gt; on &lt;code&gt;bastille0&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Start Bastille and your&amp;nbsp;jail:&lt;/li&gt;
&lt;li&gt;&lt;code&gt;service bastille start&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;bastille start webjail&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Inside the&amp;nbsp;jail:&lt;/li&gt;
&lt;li&gt;IPv4: &lt;code&gt;ping -c 3 1.1.1.1&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;IPv6: &lt;code&gt;ping -c 3 2001:4860:4860::8888&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;From the internet:
- &lt;span class="caps"&gt;HTTP&lt;/span&gt;/&lt;span class="caps"&gt;HTTPS&lt;/span&gt; should land on the jail via rdr.
- &lt;span class="caps"&gt;SSH&lt;/span&gt; should only accept from your trusted&amp;nbsp;sources.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="6-hetzner-cloud-quirks-and-fixes"&gt;6) Hetzner Cloud quirks and&amp;nbsp;fixes&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;IPv4 /32 with pseudo&amp;nbsp;next-hop:&lt;/li&gt;
&lt;li&gt;You must add a host route&amp;nbsp;to &lt;code&gt;172.31.1.1&lt;/code&gt; on &lt;code&gt;vtnet0&lt;/code&gt; and set it as&amp;nbsp;default.&lt;/li&gt;
&lt;li&gt;Offload&amp;nbsp;features:&lt;/li&gt;
&lt;li&gt;&lt;code&gt;-tso -lro&lt;/code&gt; on &lt;code&gt;vtnet0&lt;/code&gt; prevents oddities with bridged epairs and IPv6&amp;nbsp;checksums.&lt;/li&gt;
&lt;li&gt;IPv6 routing&amp;nbsp;model:&lt;/li&gt;
&lt;li&gt;IPv6 is routed to your &lt;span class="caps"&gt;VM&lt;/span&gt;. Don’t expect &lt;span class="caps"&gt;SLAAC&lt;/span&gt; for jails. Assign static v6 from your routed half (/65) and route via the&amp;nbsp;host.&lt;/li&gt;
&lt;li&gt;If IPv6 works on the host but not in&amp;nbsp;jails:&lt;/li&gt;
&lt;li&gt;Ensure &lt;code&gt;net.inet6.ip6.forwarding=1&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Make sure jails&amp;nbsp;use &lt;code&gt;2001:db8:abcd:1000:8000::1&lt;/code&gt; as their default&amp;nbsp;router&lt;/li&gt;
&lt;li&gt;Confirm &lt;span class="caps"&gt;PF&lt;/span&gt; isn’t blocking &lt;span class="caps"&gt;NDP&lt;/span&gt; or pMTU&amp;nbsp;(&lt;code&gt;ipv6-icmp&lt;/code&gt; types are allowed&amp;nbsp;above)&lt;/li&gt;
&lt;li&gt;If IPv4 &lt;span class="caps"&gt;NAT&lt;/span&gt; seems&amp;nbsp;flaky:&lt;/li&gt;
&lt;li&gt;Check that no later &lt;span class="caps"&gt;PF&lt;/span&gt; rules&amp;nbsp;override &lt;code&gt;pass out&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Verify &lt;span class="caps"&gt;MTU&lt;/span&gt;/&lt;span class="caps"&gt;MSS&lt;/span&gt;&amp;nbsp;(scrub &lt;code&gt;max-mss 1500&lt;/code&gt; is&amp;nbsp;set)&lt;/li&gt;
&lt;li&gt;Inspect&amp;nbsp;states: &lt;code&gt;pfctl -ss | grep 10.100.0.&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;h2 id="7-security-notes"&gt;7) Security&amp;nbsp;notes&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Restrict &lt;span class="caps"&gt;SSH&lt;/span&gt; with allowlists and non-default ports; consider WireGuard for&amp;nbsp;admin.&lt;/li&gt;
&lt;li&gt;Enable pflog and inspect with&amp;nbsp;tcpdump:&lt;/li&gt;
&lt;li&gt;&lt;code&gt;tcpdump -n -e -ttt -r /var/log/pflog&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Rate-limit public services at your reverse proxy jail if&amp;nbsp;needed.&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;h2 id="8-quick-reference-full-files"&gt;8) Quick reference: full&amp;nbsp;files&lt;/h2&gt;
&lt;h3 id="etcrcconf"&gt;/etc/rc.conf&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;hostname&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;freebsd-hcloud&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;keymap&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;us.kbd&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_gateway_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;ifconfig_vtnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 203.0.113.10 netmask 255.255.255.255 -tso -lro&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vtnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:abcd:1000::10/65&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;static_routes&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;hcloud default&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;route_hcloud&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-host 172.31.1.1 -interface vtnet0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;route_default&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;default 172.31.1.1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fe80::1%vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bridge0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bridge0_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bastille0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.100.0.1/24&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_bastille0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:abcd:1000:8000::1/65&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;pf_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;pflog_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sshd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;zfs_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ntpd_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ntpd_sync_on_start&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;YES&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="etcsysctlconf"&gt;/etc/sysctl.conf&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;net.inet.ip.forwarding&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
net.inet6.ip6.forwarding&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
net.inet.ip.redirect&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
net.inet6.ip6.redirect&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="etcpfconf"&gt;/etc/pf.conf&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vtnet0&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;jail_net_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.100.0.0/24&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;jail_net_v6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:abcd:1000:8000::/65&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:abcd:1000::10&amp;quot;&lt;/span&gt;

&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;198.51&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;100.22&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;203.0&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;113.50&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;ffff&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;64&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="n"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;persist&lt;/span&gt;

&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;skip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;lo0&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;drop&lt;/span&gt;
&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loginterface&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;

&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fragment&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;reassemble&lt;/span&gt;
&lt;span class="n"&gt;scrub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;random&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;mss&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1500&lt;/span&gt;

&lt;span class="n"&gt;nat&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;jail_net_v4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;any&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;10.100&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;0.100&lt;/span&gt;
&lt;span class="n"&gt;rdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;host_ipv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;abcd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1000&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;8000&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;

&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;
&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;out&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="n"&gt;antispoof&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v4&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22222&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;trusted_v6&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22222&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;S&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;SA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;max&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;src&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;conn&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;overload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;bruteforce&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;global&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="n"&gt;block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22222&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh_blocked&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echorep&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbrsol&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neighbradv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;toobig&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;timex&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paramprob&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;icmp&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;echoreq&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unreach&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bastille0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;

&lt;span class="k"&gt;pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quick&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="n"&gt;ext_if&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;port&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;keep&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="jail-rcconf-inside-the-jail"&gt;Jail rc.conf (inside the&amp;nbsp;jail)&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;ifconfig_e0b_webjail_name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vnet0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet 10.100.0.100 netmask 255.255.255.0&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ifconfig_vnet0_ipv6&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;inet6 2001:db8:abcd:1000:8000::100/65&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10.100.0.1&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;ipv6_defaultrouter&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2001:db8:abcd:1000:8000::1&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;syslogd_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-ss&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_submit_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_outbound_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;sendmail_msp_queue_enable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;NO&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;hr&gt;
&lt;h2 id="9-troubleshooting-checklist"&gt;9) Troubleshooting&amp;nbsp;checklist&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Host&amp;nbsp;routing:&lt;/li&gt;
&lt;li&gt;&lt;code&gt;netstat -rn&lt;/code&gt; shows default v4 via 172.31.1.1; default v6 via&amp;nbsp;fe80::1%vtnet0&lt;/li&gt;
&lt;li&gt;Bridge&amp;nbsp;membership:&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ifconfig bastille0&lt;/code&gt; lists e0a_ interfaces for running&amp;nbsp;jails&lt;/li&gt;
&lt;li&gt;IPv6 neighbors and&amp;nbsp;pMTU:&lt;/li&gt;
&lt;li&gt;&lt;code&gt;ndp -an&lt;/code&gt; on host shows upstream neighbors; ensure ipv6-icmp is&amp;nbsp;allowed&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt;&amp;nbsp;state:&lt;/li&gt;
&lt;li&gt;&lt;code&gt;pfctl -ss | grep 10.100.0.&lt;/code&gt; confirms IPv4 &lt;span class="caps"&gt;NAT&lt;/span&gt;&amp;nbsp;states&lt;/li&gt;
&lt;li&gt;Throughput/stalls:&lt;/li&gt;
&lt;li&gt;Verify &lt;code&gt;-tso -lro&lt;/code&gt; on&amp;nbsp;vtnet0; &lt;code&gt;scrub out max-mss 1500&lt;/code&gt; is&amp;nbsp;present&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;h2 id="wrap-up"&gt;Wrap-up&lt;/h2&gt;
&lt;p&gt;This pattern—IPv4 NATed jails, IPv6 routed using a /64 split into two /65s—fits Hetzner Cloud’s model and avoids &lt;span class="caps"&gt;NAT66&lt;/span&gt;/&lt;span class="caps"&gt;ULA&lt;/span&gt; while keeping end-to-end IPv6. It’s simple, robust, and friendly to &lt;span class="caps"&gt;PF&lt;/span&gt;.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.freebsd.org/en/books/handbook/jails/"&gt;FreeBSD Handbook:&amp;nbsp;Jails&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bastillebsd.org"&gt;Bastille&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.hetzner.com/cloud/servers/primary-ips/primary-ip-configuration#ipv6"&gt;IPv6 at Hetzner&amp;nbsp;Cloud&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="networking"/></entry><entry><title>FreeBSD 15.0 on the ThinkPad T480 — Efficient, Stable, and 8 Hours on Battery</title><link href="https://hofstede.it/freebsd-150-on-the-thinkpad-t480-efficient-stable-and-8-hours-on-battery/" rel="alternate"/><published>2025-11-09T00:00:00+01:00</published><updated>2025-11-09T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-11-09:/freebsd-150-on-the-thinkpad-t480-efficient-stable-and-8-hours-on-battery/</id><summary type="html">&lt;p&gt;Achieving near-Linux battery life on the T480 while staying purely &lt;span class="caps"&gt;BSD&lt;/span&gt;.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;h2 id="introduction"&gt;Introduction&lt;/h2&gt;
&lt;p&gt;The ThinkPad T480 has long had a reputation for being one of the most practical and robust laptops for engineers and developers. It’s built like a tank, easy to maintain, and works brilliantly with open-source operating&amp;nbsp;systems.&lt;/p&gt;
&lt;p&gt;Over the last few months, I’ve been tuning and optimizing &lt;strong&gt;FreeBSD&lt;/strong&gt; on this machine with a very clear goal in mind:&lt;br&gt;
&lt;strong&gt;reach the best possible power efficiency without sacrificing performance or&amp;nbsp;usability.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;After a good amount of trial and error, firmware testing, and sysctl tuning, I finally reached a sweet spot. A setup that reliably gives me &lt;strong&gt;6–8 hours of real battery life&lt;/strong&gt; under normal&amp;nbsp;workloads.&lt;/p&gt;
&lt;p&gt;This post documents that configuration for anyone in the &lt;span class="caps"&gt;BSD&lt;/span&gt; community who wants to make their ThinkPad a genuinely mobile FreeBSD&amp;nbsp;workstation.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Screenshot of FreeBSD desktop" src="https://hofstede.it/images/2025-11-09-t480-freebsd-efficiency.webp" title="FreeBSD 15.0 Desktop"&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="hardware-overview"&gt;Hardware&amp;nbsp;Overview&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Model:&lt;/strong&gt; Lenovo ThinkPad&amp;nbsp;T480&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt;:&lt;/strong&gt; Intel Core i7-8550U (4C/8T, Kaby&amp;nbsp;Lake-R)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;GPU&lt;/span&gt;:&lt;/strong&gt; Intel &lt;span class="caps"&gt;UHD&lt;/span&gt; Graphics 620&amp;nbsp;(integrated)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Display:&lt;/strong&gt; 1920×1080 (&lt;span class="caps"&gt;IPS&lt;/span&gt;,&amp;nbsp;14&amp;#8221;)  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Wireless:&lt;/strong&gt; Intel &lt;span class="caps"&gt;AX210&lt;/span&gt;&amp;nbsp;(&lt;code&gt;iwlwifi&lt;/code&gt;)  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Storage:&lt;/strong&gt; NVMe &lt;span class="caps"&gt;SSD&lt;/span&gt; (&lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;nbsp;root)  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Memory:&lt;/strong&gt; 32 GiB &lt;span class="caps"&gt;DDR4&lt;/span&gt;  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Battery:&lt;/strong&gt; 72 Wh&amp;nbsp;extended&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The combination of efficient hardware and modern drivers from the FreeBSD 15 branch provides an excellent base to work&amp;nbsp;with.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="drivers-and-firmware"&gt;Drivers and&amp;nbsp;Firmware&lt;/h2&gt;
&lt;p&gt;I’m using the latest driver sets available from the ports&amp;nbsp;tree:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;drm-latest-kmod-6.9.1500068
gpu-firmware-intel-kmod-kabylake-20230625.1500501
wifi-firmware-iwlwifi-kmod-ax210-20241017.1500501_2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;These make the Intel &lt;span class="caps"&gt;GPU&lt;/span&gt; and Wi-Fi adapters fully functional with modern capabilities: &lt;span class="caps"&gt;RC6&lt;/span&gt;, &lt;span class="caps"&gt;PSR&lt;/span&gt;, &lt;span class="caps"&gt;FBC&lt;/span&gt;, and full 802.11ac/ax&amp;nbsp;performance.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="system-configuration"&gt;System&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;FreeBSD’s configuration model is simple and transparent, which makes power tuning straightforward once you understand the&amp;nbsp;fundamentals.&lt;/p&gt;
&lt;h3 id="etcrcconf"&gt;/etc/rc.conf&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;hostname=&amp;quot;christop-bsd&amp;quot;

# Wireless configuration
wlans_iwlwifi0=&amp;quot;wlan0&amp;quot;
ifconfig_wlan0=&amp;quot;WPA DHCP&amp;quot;
create_args_wlan0=&amp;quot;wlanmode sta regdomain ETSI country DE mode 11na&amp;quot;
ifconfig_wlan0_ipv6=&amp;quot;inet6 accept_rtadv&amp;quot;

# Networking bridge (for bhyve)
cloned_interfaces=&amp;quot;bridge0&amp;quot;
ifconfig_bridge0=&amp;quot;inet 192.168.87.1 netmask 255.255.255.0&amp;quot;

# Power management
powerd_enable=&amp;quot;YES&amp;quot;
powerd_flags=&amp;quot;-a hiadaptive -b adaptive -m 400 -M 4000&amp;quot;
performance_cx_lowest=&amp;quot;Cmax&amp;quot;
economy_cx_lowest=&amp;quot;Cmax&amp;quot;

# Driver autoloads
kld_list=&amp;quot;i915kms&amp;quot;

# Services
sshd_enable=&amp;quot;YES&amp;quot;
zfs_enable=&amp;quot;YES&amp;quot;
seatd_enable=&amp;quot;YES&amp;quot;
dbus_enable=&amp;quot;YES&amp;quot;
sddm_enable=&amp;quot;YES&amp;quot;
webcamd_enable=&amp;quot;YES&amp;quot;
pf_enable=&amp;quot;YES&amp;quot;
pflog_enable=&amp;quot;YES&amp;quot;
dnsmasq_enable=&amp;quot;YES&amp;quot;
pcscd_enable=&amp;quot;YES&amp;quot;
gateway_enable=&amp;quot;YES&amp;quot;
clear_tmp_enable=&amp;quot;YES&amp;quot;

# bhyve
vm_enable=&amp;quot;YES&amp;quot;
vm_dir=&amp;quot;zfs:zroot/bhyve&amp;quot;

# Misc
moused_nondefault_enable=&amp;quot;NO&amp;quot;
dumpdev=&amp;quot;NO&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The key piece here&amp;nbsp;is &lt;code&gt;powerd&lt;/code&gt;, with different adaptive modes for &lt;span class="caps"&gt;AC&lt;/span&gt; and battery, letting the &lt;span class="caps"&gt;CPU&lt;/span&gt; drop down to deep C-states efficiently.&lt;br&gt;
&lt;span class="caps"&gt;XFCE&lt;/span&gt; and lightweight services complete the power-slim system&amp;nbsp;profile.&lt;/p&gt;
&lt;hr&gt;
&lt;h3 id="bootloaderconf"&gt;/boot/loader.conf&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Kernel features
kern.geom.label.disk_ident.enable=&amp;quot;0&amp;quot;
kern.geom.label.gptid.enable=&amp;quot;0&amp;quot;

# Essential modules
zfs_load=&amp;quot;YES&amp;quot;
cryptodev_load=&amp;quot;YES&amp;quot;
cuse_load=&amp;quot;YES&amp;quot;
cpuctl_load=&amp;quot;YES&amp;quot;
coretemp_load=&amp;quot;YES&amp;quot;
nvme_load=&amp;quot;YES&amp;quot;

# Graphics and power tuning
i915kms_load=&amp;quot;YES&amp;quot;
drm.i915.enable_rc6=7
drm.i915.enable_fbc=1
drm.i915.lvds_downclock=1
drm.i915.enable_psr=1
compat.linuxkpi.i915_disable_power_well=&amp;quot;0&amp;quot;

# NVMe and PCI power tweaks
hw.nvme.use_nvd=&amp;quot;0&amp;quot;
hw.pci.do_power_nodriver=3
hw.pci.do_power_suspend=1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;These settings allow the &lt;span class="caps"&gt;GPU&lt;/span&gt; and &lt;span class="caps"&gt;PCI&lt;/span&gt; devices to enter the lowest power states reliably.&lt;br&gt;
Together&amp;nbsp;with &lt;code&gt;powerd&lt;/code&gt;, they make a significant difference in idle and light load power&amp;nbsp;draw.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="desktop-environment"&gt;Desktop&amp;nbsp;Environment&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Desktop:&lt;/strong&gt; &lt;span class="caps"&gt;XFCE&lt;/span&gt;&amp;nbsp;4.20  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Window Manager:&lt;/strong&gt; Xfwm4&amp;nbsp;(X11)  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Theme:&lt;/strong&gt;&amp;nbsp;Greybird-Dark  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Icons:&lt;/strong&gt;&amp;nbsp;elementary-xfce  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Fonts:&lt;/strong&gt; Noto Sans (11&amp;nbsp;pt)  &lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Terminal font:&lt;/strong&gt; UbuntuMono Nerd Font (11&amp;nbsp;pt)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;span class="caps"&gt;XFCE&lt;/span&gt; is lightweight, fast, and power-friendly - no unnecessary daemons, no heavy compositing. It feels modern and stays responsive even under&amp;nbsp;load.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="real-world-results"&gt;Real-World&amp;nbsp;Results&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Scenario&lt;/th&gt;
&lt;th&gt;Power Draw&lt;/th&gt;
&lt;th&gt;Notes&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Idle (Wi-Fi on, 40 % brightness)&lt;/td&gt;
&lt;td&gt;4–5 W&lt;/td&gt;
&lt;td&gt;ultra-low power idle&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Video playback (720p)&lt;/td&gt;
&lt;td&gt;6–7 W&lt;/td&gt;
&lt;td&gt;stable playback via Intel &lt;span class="caps"&gt;GPU&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Moderate development workload&lt;/td&gt;
&lt;td&gt;10–12 W&lt;/td&gt;
&lt;td&gt;perfect balance&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;make buildkernel&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;18–22 W&lt;/td&gt;
&lt;td&gt;all cores fully utilized&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;With the 72 Wh extended battery, real-world battery time ranges from &lt;strong&gt;6 to 8 hours&lt;/strong&gt; depending on activity, brightness, and Wi-Fi&amp;nbsp;intensity.&lt;/p&gt;
&lt;p&gt;Suspend and resume are fully functional and reliable, one of the last major milestones that used to be tricky on FreeBSD&amp;nbsp;laptops.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="expert-notes-additional-tweaks"&gt;Expert Notes &lt;span class="amp"&gt;&amp;amp;&lt;/span&gt; Additional&amp;nbsp;Tweaks&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Enable &lt;span class="caps"&gt;ZFS&lt;/span&gt; compression&amp;nbsp;(&lt;code&gt;lz4&lt;/code&gt;) and&amp;nbsp;disable &lt;code&gt;atime&lt;/code&gt; to cut down write&amp;nbsp;I/O.  &lt;/li&gt;
&lt;li&gt;Keep firmware packages current&amp;nbsp;using &lt;code&gt;pkg upgrade&lt;/code&gt;.  &lt;/li&gt;
&lt;li&gt;Consider&amp;nbsp;running &lt;code&gt;powerdxx&lt;/code&gt; if you want even finer &lt;span class="caps"&gt;CPU&lt;/span&gt; scaling — though&amp;nbsp;base &lt;code&gt;powerd&lt;/code&gt; does a great job&amp;nbsp;here.&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;FreeBSD 15.0 marks a huge step forward for laptop support.&lt;br&gt;
On the ThinkPad T480, everything from suspend to Wi-Fi to graphics just works, and with the right configuration, you can enjoy &lt;strong&gt;Linux-level power efficiency&lt;/strong&gt; combined with &lt;strong&gt;FreeBSD’s rock-solid stability&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;It’s a joy to use this machine daily now: clean, simple, and efficient. True to FreeBSD’s&amp;nbsp;spirit.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;Thanks to everyone in the &lt;span class="caps"&gt;BSD&lt;/span&gt; community pushing kernel and driver work - this level of polish on a laptop would’ve been unthinkable just a few releases&amp;nbsp;ago.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;Comments and feedback welcome via Mastodon: &lt;a href="https://mastodon.bsd.cafe/@larvitz"&gt;@larvitz@bsd.cafe&lt;/a&gt;&lt;/em&gt;&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="thinkpad"/><category term="power"/><category term="xfce"/><category term="zfs"/></entry><entry><title>Simple Temperature Monitoring on FreeBSD</title><link href="https://hofstede.it/simple-temperature-monitoring-on-freebsd/" rel="alternate"/><published>2025-10-09T00:00:00+02:00</published><updated>2025-10-09T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-10-09:/simple-temperature-monitoring-on-freebsd/</id><summary type="html">&lt;p&gt;Using sysctl, coretemp, and shell scripting to monitor &lt;span class="caps"&gt;CPU&lt;/span&gt; and disk temperatures on FreeBSD - a lightweight approach without heavy monitoring&amp;nbsp;frameworks.&lt;/p&gt;</summary><content type="html">&lt;p&gt;If you&amp;#8217;re running FreeBSD 14.x and want to keep an eye on your system temperatures without installing third-party packages, you&amp;#8217;re in luck.&amp;nbsp;FreeBSD&amp;#8217;s &lt;code&gt;sysctl&lt;/code&gt; interface provides everything you&amp;nbsp;need.&lt;/p&gt;
&lt;h2 id="the-built-in-approach"&gt;The Built-in&amp;nbsp;Approach&lt;/h2&gt;
&lt;p&gt;Modern FreeBSD systems expose temperature data through the sysctl tree. You can query all temperature sensors with a simple&amp;nbsp;command:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysctl&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;temperature
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This will show &lt;span class="caps"&gt;CPU&lt;/span&gt; core temperatures, &lt;span class="caps"&gt;ACPI&lt;/span&gt; thermal zones, and &lt;span class="caps"&gt;PCH&lt;/span&gt; (Platform Controller Hub) temperatures if&amp;nbsp;available.&lt;/p&gt;
&lt;h2 id="a-practical-monitoring-script"&gt;A Practical Monitoring&amp;nbsp;Script&lt;/h2&gt;
&lt;p&gt;Rather than repeatedly typing sysctl commands, here&amp;#8217;s a lightweight monitoring script that refreshes every two&amp;nbsp;seconds:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
kldload&lt;span class="w"&gt; &lt;/span&gt;coretemp
&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;true&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
clear
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;======================================&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;   Temperature Monitor - &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;+%H:%M:%S&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;======================================&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;CPU Cores:&amp;quot;&lt;/span&gt;
sysctl&lt;span class="w"&gt; &lt;/span&gt;dev.cpu&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;temperature&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort&lt;span class="w"&gt; &lt;/span&gt;-V
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;System:&amp;quot;&lt;/span&gt;
sysctl&lt;span class="w"&gt; &lt;/span&gt;hw.acpi.thermal.tz0.temperature&lt;span class="w"&gt; &lt;/span&gt;hw.acpi.thermal.tz1.temperature&lt;span class="w"&gt; &lt;/span&gt;dev.pchtherm.0.temperature&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Press Ctrl+C to exit&amp;quot;&lt;/span&gt;
sleep&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Save this&amp;nbsp;as &lt;code&gt;tempmon.sh&lt;/code&gt;, make it executable&amp;nbsp;with &lt;code&gt;chmod +x tempmon.sh&lt;/code&gt;, and run it whenever you need real-time temperature&amp;nbsp;monitoring.&lt;/p&gt;
&lt;h2 id="quick-access-with-an-alias"&gt;Quick Access with an&amp;nbsp;Alias&lt;/h2&gt;
&lt;p&gt;For instant temperature checks, add this alias to your shell&amp;nbsp;configuration:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# For sh/bash (~/.shrc or ~/.bashrc)&lt;/span&gt;
&lt;span class="nb"&gt;alias&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;temps&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;sysctl dev.cpu | grep temperature | sort -V; echo &amp;quot;&amp;quot;; sysctl hw.acpi.thermal.tz0.temperature dev.pchtherm.0.temperature&amp;#39;&lt;/span&gt;
&lt;span class="c1"&gt;# For csh/tcsh (~/.cshrc)&lt;/span&gt;
&lt;span class="nb"&gt;alias&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;temps&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;sysctl dev.cpu | grep temperature | sort -V; echo &amp;quot;&amp;quot;; sysctl hw.acpi.thermal.tz0.temperature dev.pchtherm.0.temperature&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now &lt;code&gt;temps&lt;/code&gt; gives you an instant snapshot of your system&amp;#8217;s thermal&amp;nbsp;state.&lt;/p&gt;
&lt;h2 id="understanding-the-output"&gt;Understanding the&amp;nbsp;Output&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;dev.cpu.X.temperature&lt;/code&gt; values represent individual &lt;span class="caps"&gt;CPU&lt;/span&gt; core temperatures,&amp;nbsp;while &lt;code&gt;hw.acpi.thermal.tzX.temperature&lt;/code&gt; shows &lt;span class="caps"&gt;ACPI&lt;/span&gt; thermal zone readings (typically overall system/motherboard temperatures).&amp;nbsp;The &lt;code&gt;dev.pchtherm.0.temperature&lt;/code&gt; reports the Platform Controller Hub temperature when available.
For most systems under normal load, &lt;span class="caps"&gt;CPU&lt;/span&gt; temperatures between 40-60°C are typical. Sustained temperatures above 80°C warrant&amp;nbsp;investigation.&lt;/p&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="sysadmin"/></entry><entry><title>FreeBSD Cheat Sheet for Linux Admins</title><link href="https://hofstede.it/freebsd-cheat-sheet-for-linux-admins/" rel="alternate"/><published>2025-08-29T00:00:00+02:00</published><updated>2025-08-29T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-08-29:/freebsd-cheat-sheet-for-linux-admins/</id><summary type="html">&lt;p&gt;A side-by-side reference of FreeBSD and Linux commands covering hardware, disks, networking, packages, services, users, and system management - for Linux admins getting started with&amp;nbsp;FreeBSD.&lt;/p&gt;</summary><content type="html">&lt;h2 id="hardware-information"&gt;Hardware&amp;nbsp;Information&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;List &lt;span class="caps"&gt;PCI&lt;/span&gt; devices&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pciconf -lv&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;lspci -v&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List &lt;span class="caps"&gt;USB&lt;/span&gt; devices&lt;/td&gt;
&lt;td&gt;&lt;code&gt;usbconfig&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;lsusb&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show &lt;span class="caps"&gt;CPU&lt;/span&gt; info&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sysctl hw.model&lt;/code&gt; or &lt;code&gt;dmesg \| grep CPU&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;cat /proc/cpuinfo&lt;/code&gt; or &lt;code&gt;lscpu&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show memory info&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sysctl hw.physmem&lt;/code&gt; or &lt;code&gt;top&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;free -h&lt;/code&gt; or &lt;code&gt;cat /proc/meminfo&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show kernel modules&lt;/td&gt;
&lt;td&gt;&lt;code&gt;kldstat&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;lsmod&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Load kernel module&lt;/td&gt;
&lt;td&gt;&lt;code&gt;kldload module_name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;modprobe module_name&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unload kernel module&lt;/td&gt;
&lt;td&gt;&lt;code&gt;kldunload module_name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;modprobe -r module_name&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="disk-and-storage-management"&gt;Disk and Storage&amp;nbsp;Management&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;List all disks&lt;/td&gt;
&lt;td&gt;&lt;code&gt;geom disk list&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;lsblk&lt;/code&gt; or &lt;code&gt;fdisk -l&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show disk partitions&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gpart show&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;fdisk -l&lt;/code&gt; or &lt;code&gt;parted -l&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show disk usage&lt;/td&gt;
&lt;td&gt;&lt;code&gt;df -h&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;df -h&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show mounted filesystems&lt;/td&gt;
&lt;td&gt;&lt;code&gt;mount&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;mount&lt;/code&gt; or &lt;code&gt;findmnt&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Check filesystem&lt;/td&gt;
&lt;td&gt;&lt;code&gt;fsck&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;fsck&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show &lt;span class="caps"&gt;SMART&lt;/span&gt; data&lt;/td&gt;
&lt;td&gt;&lt;code&gt;smartctl -a /dev/ada0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;smartctl -a /dev/sda&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List &lt;span class="caps"&gt;ZFS&lt;/span&gt; pools&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool list&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool list&lt;/code&gt; (if &lt;span class="caps"&gt;ZFS&lt;/span&gt; installed)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show &lt;span class="caps"&gt;ZFS&lt;/span&gt; datasets&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list&lt;/code&gt; (if &lt;span class="caps"&gt;ZFS&lt;/span&gt; installed)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="network-commands"&gt;Network&amp;nbsp;Commands&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Show network interfaces&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ifconfig&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ip addr&lt;/code&gt; or &lt;code&gt;ifconfig&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Configure interface&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ifconfig em0 inet 192.168.1.10&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ip addr add 192.168.1.10/24 dev eth0&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show routing table&lt;/td&gt;
&lt;td&gt;&lt;code&gt;netstat -rn&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ip route&lt;/code&gt; or &lt;code&gt;route -n&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Add static route&lt;/td&gt;
&lt;td&gt;&lt;code&gt;route add default 192.168.1.1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ip route add default via 192.168.1.1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show &lt;span class="caps"&gt;ARP&lt;/span&gt; table&lt;/td&gt;
&lt;td&gt;&lt;code&gt;arp -a&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;arp -a&lt;/code&gt; or &lt;code&gt;ip neigh&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Clear &lt;span class="caps"&gt;ARP&lt;/span&gt; entry&lt;/td&gt;
&lt;td&gt;&lt;code&gt;arp -d hostname&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;arp -d hostname&lt;/code&gt; or &lt;code&gt;ip neigh del&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show network statistics&lt;/td&gt;
&lt;td&gt;&lt;code&gt;netstat -s&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;netstat -s&lt;/code&gt; or &lt;code&gt;ss -s&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show listening ports&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sockstat -l&lt;/code&gt; or &lt;code&gt;netstat -an&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ss -tuln&lt;/code&gt; or &lt;code&gt;netstat -tuln&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show all open ports/connections&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sockstat&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ss -tuan&lt;/code&gt; or &lt;code&gt;netstat -tuan&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Packet capture&lt;/td&gt;
&lt;td&gt;&lt;code&gt;tcpdump -i em0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;tcpdump -i eth0&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="process-management"&gt;Process&amp;nbsp;Management&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;List all processes&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ps aux&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ps aux&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Process tree&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pstree&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pstree&lt;/code&gt; or &lt;code&gt;ps axjf&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Real-time process view&lt;/td&gt;
&lt;td&gt;&lt;code&gt;top&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;top&lt;/code&gt; or &lt;code&gt;htop&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kill process&lt;/td&gt;
&lt;td&gt;&lt;code&gt;kill PID&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;kill PID&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show open files by process&lt;/td&gt;
&lt;td&gt;&lt;code&gt;fstat&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;lsof&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show process using port&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sockstat -p 80&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;lsof -i :80&lt;/code&gt; or &lt;code&gt;ss -tulpn \| grep :80&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="package-management"&gt;Package&amp;nbsp;Management&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux (varies by distro)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Install package&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pkg install package_name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;apt install&lt;/code&gt; / &lt;code&gt;yum install&lt;/code&gt; / &lt;code&gt;dnf install&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Remove package&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pkg delete package_name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;apt remove&lt;/code&gt; / &lt;code&gt;yum remove&lt;/code&gt; / &lt;code&gt;dnf remove&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Update package list&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pkg update&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;apt update&lt;/code&gt; / &lt;code&gt;yum check-update&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Upgrade packages&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pkg upgrade&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;apt upgrade&lt;/code&gt; / &lt;code&gt;yum upgrade&lt;/code&gt; / &lt;code&gt;dnf upgrade&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Search packages&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pkg search keyword&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;apt search&lt;/code&gt; / &lt;code&gt;yum search&lt;/code&gt; / &lt;code&gt;dnf search&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show package info&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pkg info package_name&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;apt show&lt;/code&gt; / &lt;code&gt;yum info&lt;/code&gt; / &lt;code&gt;dnf info&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List installed packages&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pkg info&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;dpkg -l&lt;/code&gt; / &lt;code&gt;rpm -qa&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="system-services"&gt;System&amp;nbsp;Services&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux (systemd)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Start service&lt;/td&gt;
&lt;td&gt;&lt;code&gt;service servicename start&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl start servicename&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stop service&lt;/td&gt;
&lt;td&gt;&lt;code&gt;service servicename stop&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl stop servicename&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Restart service&lt;/td&gt;
&lt;td&gt;&lt;code&gt;service servicename restart&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl restart servicename&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Service status&lt;/td&gt;
&lt;td&gt;&lt;code&gt;service servicename status&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl status servicename&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enable at boot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sysrc servicename_enable="YES"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl enable servicename&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Disable at boot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sysrc servicename_enable="NO"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl disable servicename&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List all services&lt;/td&gt;
&lt;td&gt;&lt;code&gt;service -e&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl list-units --type=service&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="firewall"&gt;Firewall&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD (pf/ipfw)&lt;/th&gt;
&lt;th&gt;Linux (iptables/nftables)&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Show rules (pf)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pfctl -sr&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;iptables -L -n -v&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show rules (ipfw)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ipfw list&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;nft list ruleset&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enable firewall (pf)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pfctl -e&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl start firewalld&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Disable firewall (pf)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pfctl -d&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl stop firewalld&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reload rules (pf)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pfctl -f /etc/pf.conf&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;iptables-restore &amp;lt; /etc/iptables/rules.v4&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="system-information"&gt;System&amp;nbsp;Information&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;System uptime&lt;/td&gt;
&lt;td&gt;&lt;code&gt;uptime&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;uptime&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Kernel version&lt;/td&gt;
&lt;td&gt;&lt;code&gt;uname -a&lt;/code&gt; or &lt;code&gt;freebsd-version&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;uname -a&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show all sysctl variables&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sysctl -a&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sysctl -a&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show system messages&lt;/td&gt;
&lt;td&gt;&lt;code&gt;dmesg&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;dmesg&lt;/code&gt; or &lt;code&gt;journalctl -k&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show system logs&lt;/td&gt;
&lt;td&gt;&lt;code&gt;tail /var/log/messages&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;journalctl&lt;/code&gt; or &lt;code&gt;tail /var/log/syslog&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="user-management"&gt;User&amp;nbsp;Management&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Add user&lt;/td&gt;
&lt;td&gt;&lt;code&gt;adduser&lt;/code&gt; or &lt;code&gt;pw useradd username&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;useradd username&lt;/code&gt; or &lt;code&gt;adduser username&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Delete user&lt;/td&gt;
&lt;td&gt;&lt;code&gt;rmuser&lt;/code&gt; or &lt;code&gt;pw userdel username&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;userdel username&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Modify user&lt;/td&gt;
&lt;td&gt;&lt;code&gt;pw usermod username&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;usermod username&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Change password&lt;/td&gt;
&lt;td&gt;&lt;code&gt;passwd username&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;passwd username&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show logged-in users&lt;/td&gt;
&lt;td&gt;&lt;code&gt;who&lt;/code&gt; or &lt;code&gt;w&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;who&lt;/code&gt; or &lt;code&gt;w&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="file-systems"&gt;File&amp;nbsp;Systems&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;FreeBSD&lt;/th&gt;
&lt;th&gt;Linux&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Mount filesystem&lt;/td&gt;
&lt;td&gt;&lt;code&gt;mount /dev/ada0p2 /mnt&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;mount /dev/sda2 /mnt&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unmount filesystem&lt;/td&gt;
&lt;td&gt;&lt;code&gt;umount /mnt&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;umount /mnt&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create &lt;span class="caps"&gt;UFS&lt;/span&gt; filesystem&lt;/td&gt;
&lt;td&gt;&lt;code&gt;newfs /dev/ada0p2&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;N/A (&lt;span class="caps"&gt;UFS&lt;/span&gt; not common)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create ext4 filesystem&lt;/td&gt;
&lt;td&gt;N/A&lt;/td&gt;
&lt;td&gt;&lt;code&gt;mkfs.ext4 /dev/sda2&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Check disk space&lt;/td&gt;
&lt;td&gt;&lt;code&gt;du -sh /path&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;du -sh /path&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="zfs-commands-same-on-freebsd-and-linux-with-zfs"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; Commands (Same on FreeBSD and Linux with &lt;span class="caps"&gt;ZFS&lt;/span&gt;)&lt;/h2&gt;
&lt;h3 id="pool-operations"&gt;Pool&amp;nbsp;Operations&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;List all pools&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool list&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool list&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show pool status&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool status&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool status tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show pool history&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool history&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool history tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show pool I/O statistics&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool iostat&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool iostat -v 2&lt;/code&gt; (every 2 seconds)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create simple pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool create&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool create tank /dev/ada1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create mirror pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool create&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool create tank mirror /dev/ada1 /dev/ada2&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create &lt;span class="caps"&gt;RAIDZ&lt;/span&gt; pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool create&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool create tank raidz /dev/ada1 /dev/ada2 /dev/ada3&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Add disk to pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool add&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool add tank /dev/ada4&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replace disk in pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool replace&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool replace tank /dev/ada1 /dev/ada4&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Remove device from pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool remove&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool remove tank /dev/ada4&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Scrub pool (check integrity)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool scrub&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool scrub tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stop scrub&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool scrub -s&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool scrub -s tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Clear pool errors&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool clear&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool clear tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Export pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool export&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool export tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Import pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool import&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool import tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List importable pools&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool import&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool import&lt;/code&gt; (without pool name)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Upgrade pool&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool upgrade&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool upgrade tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set pool property&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool set&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool set autoreplace=on tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Get pool properties&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool get&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool get all tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="dataset-filesystem-operations"&gt;Dataset (Filesystem)&amp;nbsp;Operations&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;List all datasets&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List with specific properties&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -o name,used,avail,mountpoint&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List snapshots&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -t snapshot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -t snapshot&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create dataset&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs create&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs create tank/home/user&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Destroy dataset&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs destroy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs destroy tank/old_data&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Destroy dataset and children&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs destroy -r&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs destroy -r tank/old_data&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set dataset property&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set compression=lz4 tank/home&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Get dataset properties&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs get&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs get all tank/home&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set quota&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set quota=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set quota=10G tank/home/user&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set reservation&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set reservation=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set reservation=5G tank/database&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Mount dataset&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs mount&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs mount tank/home&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unmount dataset&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs unmount&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs unmount tank/home&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show mounted &lt;span class="caps"&gt;ZFS&lt;/span&gt; filesystems&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs mount&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs mount&lt;/code&gt; (without arguments)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="snapshot-operations"&gt;Snapshot&amp;nbsp;Operations&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Create snapshot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs snapshot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs snapshot tank/home@backup-2024&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create recursive snapshot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs snapshot -r&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs snapshot -r tank/home@daily-2024&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List snapshots&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -t snapshot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -t snapshot -r tank/home&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rollback to snapshot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs rollback&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs rollback tank/home@backup-2024&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Destroy snapshot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs destroy&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs destroy tank/home@old-backup&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rename snapshot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs rename&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs rename tank/home@old tank/home@archived&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Clone snapshot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs clone&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs clone tank/home@backup tank/home_clone&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show snapshot disk usage&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -o space&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -r -o space tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="sendreceive-replication"&gt;Send/Receive&amp;nbsp;(Replication)&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Send snapshot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs send&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs send tank/home@backup &amp;gt; /backup/home.zfs&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Send incremental&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs send -i&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs send -i @snap1 tank/home@snap2 &amp;gt; incremental.zfs&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Receive snapshot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs receive&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs receive tank/home_restore &amp;lt; /backup/home.zfs&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Send over &lt;span class="caps"&gt;SSH&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs send \| ssh&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs send tank/home@backup \| ssh user@host zfs receive tank/backup&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dry-run receive&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs receive -n&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs receive -n tank/test &amp;lt; backup.zfs&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Send with progress&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs send -v&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs send -v tank/home@backup &amp;gt; backup.zfs&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="common-property-settings"&gt;Common Property&amp;nbsp;Settings&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Enable compression&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set compression=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set compression=lz4 tank/data&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enable deduplication&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set dedup=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set dedup=on tank/backup&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set record size&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set recordsize=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set recordsize=1M tank/media&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enable encryption&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs create -o encryption=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs create -o encryption=on -o keyformat=passphrase tank/secure&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set access time&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set atime=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set atime=off tank/database&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set sync behavior&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set sync=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set sync=disabled tank/temp&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Enable case insensitivity&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs set casesensitivity=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs create -o casesensitivity=insensitive tank/windows&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="monitoring-and-maintenance"&gt;Monitoring and&amp;nbsp;Maintenance&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Show pool I/O stats&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool iostat&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool iostat -v tank 2&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show &lt;span class="caps"&gt;ARC&lt;/span&gt; stats (FreeBSD)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sysctl kstat.zfs.misc.arcstats&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sysctl kstat.zfs.misc.arcstats.size&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show &lt;span class="caps"&gt;ARC&lt;/span&gt; stats (Linux)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;arc_summary&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;arc_summary&lt;/code&gt; or &lt;code&gt;cat /proc/spl/kstat/zfs/arcstats&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Check pool health&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool status -x&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool status -x&lt;/code&gt; (only shows problems)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show pool events&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool events&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool events -v&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Estimated scrub time&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool status&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zpool status&lt;/code&gt; (during scrub)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show compression ratio&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs get compressratio&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs get compressratio tank/data&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show space usage by type&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -o space&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;zfs list -r -o space tank&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="useful-one-liners"&gt;Useful&amp;nbsp;One-Liners&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Find largest datasets&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;name,used&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;used

&lt;span class="c1"&gt;# Show all snapshots sorted by creation&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;name,creation&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;creation

&lt;span class="c1"&gt;# Calculate total snapshot space&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;used&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{sum+=$1} END {print sum/1024/1024/1024 &amp;quot; GB&amp;quot;}&amp;#39;&lt;/span&gt;

&lt;span class="c1"&gt;# Show datasets with compression disabled&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;compression&lt;span class="w"&gt; &lt;/span&gt;tank&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;lz4\|gzip\|zle&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Monitor pool I/O in real-time&lt;/span&gt;
zpool&lt;span class="w"&gt; &lt;/span&gt;iostat&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;

&lt;span class="c1"&gt;# Show properties that differ from defaults&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;get&lt;span class="w"&gt; &lt;/span&gt;all&lt;span class="w"&gt; &lt;/span&gt;tank&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;default

&lt;span class="c1"&gt;# Quick backup to remote system&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;snapshot&lt;span class="w"&gt; &lt;/span&gt;tank/important@&lt;span class="k"&gt;$(&lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;+%Y%m%d&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
zfs&lt;span class="w"&gt; &lt;/span&gt;send&lt;span class="w"&gt; &lt;/span&gt;tank/important@&lt;span class="k"&gt;$(&lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;+%Y%m%d&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;backup-server&lt;span class="w"&gt; &lt;/span&gt;zfs&lt;span class="w"&gt; &lt;/span&gt;receive&lt;span class="w"&gt; &lt;/span&gt;tank/backup-2024
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="quick-tips"&gt;Quick&amp;nbsp;Tips&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Device naming&lt;/strong&gt;: FreeBSD uses different naming conventions (ada0 for &lt;span class="caps"&gt;SATA&lt;/span&gt;, da0 for &lt;span class="caps"&gt;SCSI&lt;/span&gt;/&lt;span class="caps"&gt;USB&lt;/span&gt;) vs Linux (sda, sdb,&amp;nbsp;etc.)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Network interfaces&lt;/strong&gt;: FreeBSD names interfaces by driver (em0, re0, bge0) while Linux traditionally used eth0, eth1 (now often uses predictable names like&amp;nbsp;enp0s3)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Configuration files&lt;/strong&gt;: FreeBSD centralizes many configs&amp;nbsp;in &lt;code&gt;/etc/rc.conf&lt;/code&gt; while Linux distributes them across various&amp;nbsp;files&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Manual pages&lt;/strong&gt;: Both&amp;nbsp;use &lt;code&gt;man command&lt;/code&gt;, but FreeBSD&amp;#8217;s man pages are often more&amp;nbsp;comprehensive&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ports vs Packages&lt;/strong&gt;: FreeBSD has both ports (source) and packages (binary), while Linux typically uses one package manager per&amp;nbsp;distribution&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt;&lt;/strong&gt;: While native to FreeBSD, &lt;span class="caps"&gt;ZFS&lt;/span&gt; on Linux (ZoL) has reached feature parity. Commands are identical, but kernel module loading differs&amp;nbsp;(&lt;code&gt;kldload zfs&lt;/code&gt; on FreeBSD&amp;nbsp;vs &lt;code&gt;modprobe zfs&lt;/code&gt; on&amp;nbsp;Linux)&lt;/li&gt;
&lt;/ul&gt;</content><category term="FreeBSD"/><category term="freebsd"/><category term="linux"/><category term="sysadmin"/></entry></feed>