<?xml version="1.0" encoding="utf-8"?>
<feed xmlns="http://www.w3.org/2005/Atom"><title>Larvitz Blog - Linux</title><link href="https://hofstede.it/" rel="alternate"/><link href="https://hofstede.it/feeds/linux.atom.xml" rel="self"/><id>https://hofstede.it/</id><updated>2026-11-17T00:00:00+01:00</updated><subtitle>FreeBSD, Linux, all things cleanly engineered</subtitle><entry><title>One Old JavaCard, Two Lives: Kerberos PKINIT and SSH From a Drawer Find</title><link href="https://hofstede.it/one-old-javacard-two-lives-kerberos-pkinit-and-ssh-from-a-drawer-find/" rel="alternate"/><published>2026-11-17T00:00:00+01:00</published><updated>2026-11-17T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-11-17:/one-old-javacard-two-lives-kerberos-pkinit-and-ssh-from-a-drawer-find/</id><summary type="html">&lt;p&gt;An old &lt;span class="caps"&gt;NXP&lt;/span&gt; JavaCard was about to become e-waste. Instead it got an applet loaded with GlobalPlatformPro, became a &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 token, received a certificate from my IdM &lt;span class="caps"&gt;CA&lt;/span&gt; and fetched a Kerberos ticket via &lt;span class="caps"&gt;PKINIT&lt;/span&gt;. Ninety minutes later the same card was wiped, turned into an OpenPGP card and logged in to a FreeBSD server over &lt;span class="caps"&gt;SSH&lt;/span&gt;. This is how both lives worked, and the six places where they did not work on the first&amp;nbsp;try.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A blank white smart card in a small white ACS USB reader with a green LED, lying next to a ThinkPad keyboard on a desk with a FreeBSD mousepad" src="https://hofstede.it/images/2026-11-17-javacard-pkinit-and-openpgp.webp" title="The card itself, in its second life: blank white plastic in an ACR39U reader."&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-card-is-a-small-computer"&gt;A Card Is a Small&amp;nbsp;Computer&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#first-life-a-pkcs15-token-for-kerberos"&gt;First Life: A &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 Token for Kerberos&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#stumbling-block-1-gp-install-says-6a80"&gt;Stumbling Block 1: gp &amp;#8212;install Says&amp;nbsp;6A80&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-first-init-and-a-reason-to-start-over"&gt;A First Init, and a Reason to Start&amp;nbsp;Over&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#stumbling-block-2-deleting-the-instance-says-6985"&gt;Stumbling Block 2: Deleting the Instance Says&amp;nbsp;6985&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#stumbling-block-3-the-opensc-profile"&gt;Stumbling Block 3: The OpenSC&amp;nbsp;Profile&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-idm-side"&gt;The IdM&amp;nbsp;Side&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-ticket"&gt;The&amp;nbsp;Ticket&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#second-life-an-openpgp-card-for-ssh"&gt;Second Life: An OpenPGP Card for &lt;span class="caps"&gt;SSH&lt;/span&gt;&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#swapping-the-applet"&gt;Swapping the&amp;nbsp;Applet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-aid-is-the-identity"&gt;The &lt;span class="caps"&gt;AID&lt;/span&gt; Is the&amp;nbsp;Identity&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#stumbling-block-4-gnupg-says-the-card-is-not-there"&gt;Stumbling Block 4: GnuPG Says the Card Is Not&amp;nbsp;There&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#keys"&gt;Keys&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#stumbling-block-5-the-agent-has-no-identities"&gt;Stumbling Block 5: The Agent Has No&amp;nbsp;Identities&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#stumbling-block-6-agent-refused-operation"&gt;Stumbling Block 6: Agent Refused&amp;nbsp;Operation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-login"&gt;The&amp;nbsp;Login&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#stumbling-blocks-at-a-glance"&gt;Stumbling Blocks at a&amp;nbsp;Glance&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-i-did-not-test"&gt;What I Did Not&amp;nbsp;Test&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;There was a smart card in my drawer that I had not touched in years. An &lt;span class="caps"&gt;NXP&lt;/span&gt; &lt;span class="caps"&gt;J3A081&lt;/span&gt;, a JavaCard running &lt;span class="caps"&gt;JCOP&lt;/span&gt; 2.4.1, white plastic with a gold contact pad and nothing printed on it. It came as a free extra with a rather large order of SmartCard-HSMs, and I had never found a use for it. I was sorting out old hardware, and not for the first time I was about to throw it&amp;nbsp;out.&lt;/p&gt;
&lt;p&gt;Then I wondered what it would take to make it useful. My laptop logs in to my Identity Management realm with a commercial &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 card, and my &lt;span class="caps"&gt;SSH&lt;/span&gt; keys live on a Nitrokey. Both of those are things a JavaCard can be, in principle. So the question for the evening was: can a blank card of that age become an enterprise login token, with nothing but open source&amp;nbsp;tools?&lt;/p&gt;
&lt;p&gt;It can. At 20:30 it fetched a Kerberos ticket for a test user. At 22:08 the same card, wiped and reloaded, logged in to a FreeBSD server over &lt;span class="caps"&gt;SSH&lt;/span&gt; as an OpenPGP card. This article walks through both lives, including the six places where it did not work on the first&amp;nbsp;try.&lt;/p&gt;
&lt;p&gt;All hostnames, realm names and paths in this post are examples. The card, the tools and the error codes are&amp;nbsp;real.&lt;/p&gt;
&lt;h2 id="a-card-is-a-small-computer"&gt;A Card Is a Small&amp;nbsp;Computer&lt;/h2&gt;
&lt;p&gt;A JavaCard is not a &amp;#8220;Kerberos card&amp;#8221; or an &amp;#8220;OpenPGP card&amp;#8221;. It is a tiny computer with a few dozen kilobytes of persistent memory, a crypto coprocessor and a stripped-down Java virtual machine. What it does depends entirely on the &lt;strong&gt;applet&lt;/strong&gt; that is installed on&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;Three terms are enough to follow the&amp;nbsp;rest:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;CAP&lt;/span&gt; file.&lt;/strong&gt; The compiled applet, the JavaCard equivalent of a &lt;span class="caps"&gt;JAR&lt;/span&gt;. It contains a package and one or more applet&amp;nbsp;classes.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;AID&lt;/span&gt;.&lt;/strong&gt; The application identifier, a byte string of 5 to 16 bytes. Packages, applets and installed instances each have one. A host program selects an applet by its &lt;span class="caps"&gt;AID&lt;/span&gt; before it talks to&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;GlobalPlatform.&lt;/strong&gt; The standard for managing what is on the card. A card manager on the card accepts authenticated commands to load packages, install instances and delete them. Authentication uses a set of symmetric keys that the card issuer&amp;nbsp;chooses.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The tool for the GlobalPlatform side is &lt;a href="https://github.com/martinpaljak/GlobalPlatformPro"&gt;GlobalPlatformPro&lt;/a&gt;, a&amp;nbsp;single &lt;code&gt;gp.jar&lt;/code&gt;. The first command only looks at the&amp;nbsp;card:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;java&lt;span class="w"&gt; &lt;/span&gt;-jar&lt;span class="w"&gt; &lt;/span&gt;gp.jar&lt;span class="w"&gt; &lt;/span&gt;--info&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Alcor Link AK9563 02 00&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;-r&lt;/code&gt; picks the reader, in my case the one built into the laptop. With Java&amp;nbsp;25, &lt;code&gt;--enable-native-access=ALL-UNNAMED&lt;/code&gt; in front&amp;nbsp;of &lt;code&gt;-jar&lt;/code&gt; silences a warning about native access. In the rest of this&amp;nbsp;post, &lt;code&gt;gp&lt;/code&gt; stands for that whole Java command line with the reader&amp;nbsp;option.&lt;/p&gt;
&lt;p&gt;My card accepted the default test keys that GlobalPlatformPro tries when you do not&amp;nbsp;pass &lt;code&gt;--key&lt;/code&gt;. That is common for development cards and it is the only reason this worked at all. If you find a card with unknown keys, be careful. The card counts failed authentication attempts, and after a small number of them it locks its card manager for good. There is no reset for&amp;nbsp;that.&lt;/p&gt;
&lt;p&gt;The software on the laptop, for reference: Fedora 45, GlobalPlatformPro 26.06.04, OpenSC 0.27.1, OpenSSL 4.0.3 with pkcs11-provider 1.3.0, &lt;span class="caps"&gt;MIT&lt;/span&gt; Kerberos 1.22.2 and OpenSSH&amp;nbsp;10.5.&lt;/p&gt;
&lt;h2 id="first-life-a-pkcs15-token-for-kerberos"&gt;First Life: A &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 Token for&amp;nbsp;Kerberos&lt;/h2&gt;
&lt;p&gt;The applet for the first life is &lt;a href="https://github.com/philipWendland/IsoApplet"&gt;IsoApplet&lt;/a&gt; by Philip Wendland, version 0.6.1. It implements an &lt;span class="caps"&gt;ISO&lt;/span&gt; 7816 file system with on-card key generation, and OpenSC has a driver for it. Together they turn the card into a regular &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 token. The 0.6 line needs Java Card 2.2.2 or newer, which this card just about provides. Pick the right release: the current IsoApplet 1.x on the main branch targets Java Card 3.0.4 and needs extended APDUs. For cards of this generation, the legacy&amp;nbsp;branch &lt;code&gt;main-javacard-v2.2.2&lt;/code&gt; and its last release, 0.6.1 from 2019, are the ones that fit. Its &lt;span class="caps"&gt;README&lt;/span&gt; also warns that &lt;span class="caps"&gt;ECDSA&lt;/span&gt; is hardly usable there, so &lt;span class="caps"&gt;RSA&lt;/span&gt; it&amp;nbsp;is.&lt;/p&gt;
&lt;h3 id="stumbling-block-1-gp-install-says-6a80"&gt;Stumbling Block&amp;nbsp;1: &lt;code&gt;gp --install&lt;/code&gt; Says&amp;nbsp;6A80&lt;/h3&gt;
&lt;p&gt;The obvious command loads the &lt;span class="caps"&gt;CAP&lt;/span&gt; file and creates an instance in one&amp;nbsp;go:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gp&lt;span class="w"&gt; &lt;/span&gt;--install&lt;span class="w"&gt; &lt;/span&gt;IsoApplet.cap
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It failed. The load went through, and the card answered the following &lt;span class="caps"&gt;INSTALL&lt;/span&gt; command with status&amp;nbsp;word &lt;code&gt;6A80&lt;/code&gt;, &amp;#8220;incorrect data&amp;#8221;. Splitting the step&amp;nbsp;with &lt;code&gt;--package&lt;/code&gt;, &lt;code&gt;--applet&lt;/code&gt; and &lt;code&gt;--create&lt;/code&gt; gave the same&amp;nbsp;result.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;gp -d&lt;/code&gt; prints every &lt;span class="caps"&gt;APDU&lt;/span&gt;, and the trace of&amp;nbsp;the &lt;code&gt;--create&lt;/code&gt; attempt shows the problem directly. Shortened, with the secure-channel &lt;span class="caps"&gt;MAC&lt;/span&gt; left&amp;nbsp;out:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;A&amp;gt;&amp;gt; 84E60C00 36 0BF276A288BCFBA69D34F310 0CF276A288BCFBA69D34F31001 0CF276A288BCFBA69D34F31001 03000000 02C900 00
A&amp;lt;&amp;lt; 6A80
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;INSTALL&lt;/span&gt; command carries a privileges field,&amp;nbsp;here &lt;code&gt;03 000000&lt;/code&gt;: a length of three, then three bytes. GlobalPlatform 2.1.1, which is what this card implements, defines that field as exactly one byte. Newer versions of the specification allow three. Current GlobalPlatformPro sends three, and the card wants one. The command I built by hand is identical except for that field, and the card accepted&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;GlobalPlatformPro can send raw APDUs over its authenticated channel&amp;nbsp;with &lt;code&gt;-s&lt;/code&gt;. So the package gets loaded without installing, and the &lt;span class="caps"&gt;INSTALL&lt;/span&gt; is typed out&amp;nbsp;manually:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gp&lt;span class="w"&gt; &lt;/span&gt;--load&lt;span class="w"&gt; &lt;/span&gt;IsoApplet.cap
gp&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;80E60C002C0BF276A288BCFBA69D34F3100CF276A288BCFBA69D34F310010CF276A288BCFBA69D34F31001010002C90000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That hex string is less scary than it&amp;nbsp;looks:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bytes&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;80 E6 0C 00&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Class, instruction &lt;span class="caps"&gt;INSTALL&lt;/span&gt;,&amp;nbsp;P1 &lt;code&gt;0C&lt;/code&gt; means &amp;#8220;for install and make selectable&amp;#8221;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;2C&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Length of the data field, 44 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;0B F276A288BCFBA69D34F310&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Package &lt;span class="caps"&gt;AID&lt;/span&gt;, 11 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;0C F276A288BCFBA69D34F31001&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Applet &lt;span class="caps"&gt;AID&lt;/span&gt; inside the package, 12 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;0C F276A288BCFBA69D34F31001&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;AID&lt;/span&gt; of the new instance, 12 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;01 00&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Privileges: one byte, no special privileges&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;02 C9 00&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Install parameters, an&amp;nbsp;empty &lt;code&gt;C9&lt;/code&gt; element&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;00&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Length of the install token, none&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Every field is a length byte followed by its content. After&amp;nbsp;that, &lt;code&gt;gp --list&lt;/code&gt; showed the package and a selectable&amp;nbsp;instance.&lt;/p&gt;
&lt;h3 id="a-first-init-and-a-reason-to-start-over"&gt;A First Init, and a Reason to Start&amp;nbsp;Over&lt;/h3&gt;
&lt;p&gt;With the applet in place, OpenSC takes&amp;nbsp;over:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pkcs15-init&lt;span class="w"&gt; &lt;/span&gt;--create-pkcs15
pkcs15-init&lt;span class="w"&gt; &lt;/span&gt;--generate-key&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;rsa/2048&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--auth-id&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FF&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--label&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;test&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--id&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That worked immediately. It also produced a token called &amp;#8220;JavaCard isoApplet&amp;#8221; from the manufacturer &amp;#8220;unknown&amp;#8221;. If I program my own card, I want my own name on it. Label, manufacturer and serial number are written when the &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 structure is created, so the card had to go back to&amp;nbsp;empty.&lt;/p&gt;
&lt;h3 id="stumbling-block-2-deleting-the-instance-says-6985"&gt;Stumbling Block 2: Deleting the Instance Says&amp;nbsp;6985&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gp&lt;span class="w"&gt; &lt;/span&gt;--delete&lt;span class="w"&gt; &lt;/span&gt;F276A288BCFBA69D34F31001
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Status&amp;nbsp;word &lt;code&gt;6985&lt;/code&gt;, &amp;#8220;conditions of use not&amp;nbsp;satisfied&amp;#8221;. &lt;code&gt;gp&lt;/code&gt; even suggests that some app might still be active, so I closed every program that had OpenSC&amp;#8217;s &lt;span class="caps"&gt;PKCS&lt;/span&gt;#11 module loaded. No change.&amp;nbsp;The &lt;code&gt;gp -d&lt;/code&gt; trace shows a plain &lt;span class="caps"&gt;DELETE&lt;/span&gt; with nothing but the instance &lt;span class="caps"&gt;AID&lt;/span&gt;, so this is not another format problem. This card refuses to delete the applet instance on its own, and I did not find out why. What it does accept is deleting the package together with everything that depends on&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;gp&lt;span class="w"&gt; &lt;/span&gt;--delete&lt;span class="w"&gt; &lt;/span&gt;F276A288BCFBA69D34F310&lt;span class="w"&gt; &lt;/span&gt;-f
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note the shorter &lt;span class="caps"&gt;AID&lt;/span&gt;. That is the package, not the instance,&amp;nbsp;and &lt;code&gt;-f&lt;/code&gt; forces the removal of its instances as well. All data in the applet is gone with it, including the test&amp;nbsp;key.&lt;/p&gt;
&lt;p&gt;So the reliable recipe for this card has three&amp;nbsp;parts: &lt;code&gt;--load&lt;/code&gt; to get the package on, a raw &lt;span class="caps"&gt;INSTALL&lt;/span&gt; with a one-byte privileges field to create the instance,&amp;nbsp;and &lt;code&gt;--delete &amp;lt;package AID&amp;gt; -f&lt;/code&gt; to get rid of it. It carried me through the rest of the&amp;nbsp;evening.&lt;/p&gt;
&lt;h3 id="stumbling-block-3-the-opensc-profile"&gt;Stumbling Block 3: The OpenSC&amp;nbsp;Profile&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;pkcs15-init&lt;/code&gt; takes the token&amp;#8217;s label, manufacturer and &lt;span class="caps"&gt;PIN&lt;/span&gt; policy from profile files&amp;nbsp;in &lt;code&gt;/usr/share/opensc/&lt;/code&gt;. I did not want to edit files that belong to an &lt;span class="caps"&gt;RPM&lt;/span&gt;, so I gave OpenSC a private&amp;nbsp;copy:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;~/javacard/opensc-profile
cp&lt;span class="w"&gt; &lt;/span&gt;/usr/share/opensc/pkcs15.profile&lt;span class="w"&gt; &lt;/span&gt;/usr/share/opensc/isoApplet.profile&lt;span class="w"&gt; &lt;/span&gt;~/javacard/opensc-profile/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Both files are&amp;nbsp;needed. &lt;code&gt;isoApplet.profile&lt;/code&gt; includes &lt;code&gt;pkcs15.profile&lt;/code&gt;, and OpenSC looks for it in the same directory. A small configuration file points OpenSC at the&amp;nbsp;copy:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;~/javacard/opensc-custom.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;app default {
    profile_dir = /home/alice/javacard/opensc-profile;
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In the&amp;nbsp;copied &lt;code&gt;isoApplet.profile&lt;/code&gt;, I changed two&amp;nbsp;lines:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="gd"&gt;-   label          =&amp;quot;JavaCard isoApplet&amp;quot;;&lt;/span&gt;
&lt;span class="gd"&gt;-   manufacturer   = &amp;quot;unknown&amp;quot;;&lt;/span&gt;
&lt;span class="gi"&gt;+   label          =&amp;quot;Personal IdM SmartCard&amp;quot;;&lt;/span&gt;
&lt;span class="gi"&gt;+   manufacturer   = &amp;quot;Larvitz PKI Enterprises&amp;quot;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then the initialization again, this time with the private profile and a serial number of my&amp;nbsp;choice:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;env&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;OPENSC_CONF&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;/javacard/opensc-custom.conf&lt;span class="w"&gt; &lt;/span&gt;pkcs15-init&lt;span class="w"&gt; &lt;/span&gt;--create-pkcs15&lt;span class="w"&gt; &lt;/span&gt;--serial&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;232323424242&lt;/span&gt;
pkcs15-init&lt;span class="w"&gt; &lt;/span&gt;--generate-key&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;rsa/2048&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--auth-id&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FF&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--label&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;l0ls3c&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--id&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;1&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;env&lt;/code&gt; in front is there because my shell is fish, which does not&amp;nbsp;accept &lt;code&gt;VAR=value command&lt;/code&gt;. In bash you can drop&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;--create-pkcs15&lt;/code&gt; asks for a user &lt;span class="caps"&gt;PIN&lt;/span&gt; and a &lt;span class="caps"&gt;PUK&lt;/span&gt;. IsoApplet insists on a &lt;span class="caps"&gt;PUK&lt;/span&gt; of at least 16 characters and answers anything shorter&amp;nbsp;with &lt;code&gt;PIN too short (min 16 characters)&lt;/code&gt;. The &lt;span class="caps"&gt;PUK&lt;/span&gt; cannot be skipped&amp;nbsp;either.&lt;/p&gt;
&lt;p&gt;One detail differs from other &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 cards I have used. On IsoApplet, the user &lt;span class="caps"&gt;PIN&lt;/span&gt; object has the &lt;span class="caps"&gt;ID&lt;/span&gt; &lt;code&gt;ff&lt;/code&gt;,&amp;nbsp;not &lt;code&gt;01&lt;/code&gt;.&amp;nbsp;Every &lt;code&gt;--auth-id&lt;/code&gt; has to say so,&amp;nbsp;otherwise &lt;code&gt;pkcs15-init&lt;/code&gt; looks for a &lt;span class="caps"&gt;PIN&lt;/span&gt; that does not&amp;nbsp;exist.&lt;/p&gt;
&lt;p&gt;The&amp;nbsp;result:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;label&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Personal&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;IdM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SmartCard&lt;/span&gt;
&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;manufacturer&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Larvitz&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PKI&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Enterprises&lt;/span&gt;
&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;model&lt;/span&gt;&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PKCS&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="mi"&gt;15&lt;/span&gt;
&lt;span class="n"&gt;serial&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;num&lt;/span&gt;&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;232323424242&lt;/span&gt;

&lt;span class="n"&gt;PIN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;User&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PIN&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;ID&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ff&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Reference&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ascii&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;numeric&lt;/span&gt;
&lt;span class="n"&gt;Private&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;RSA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l0ls3c&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;ID&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mo"&gt;01&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2048&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Usage&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sign&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                          &lt;/span&gt;&lt;span class="n"&gt;Access&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Flags&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sensitive&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;alwaysSensitive&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;neverExtract&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The last line is the point of the whole&amp;nbsp;exercise. &lt;code&gt;local&lt;/code&gt; means the key was generated on the&amp;nbsp;card. &lt;code&gt;neverExtract&lt;/code&gt; means it cannot leave it. No copy of this private key ever existed on the&amp;nbsp;laptop.&lt;/p&gt;
&lt;h3 id="the-idm-side"&gt;The IdM&amp;nbsp;Side&lt;/h3&gt;
&lt;p&gt;The card needs someone to be. With an admin ticket on one of the IdM servers, I created a test&amp;nbsp;user:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;user-add&lt;span class="w"&gt; &lt;/span&gt;john&lt;span class="w"&gt; &lt;/span&gt;--first&lt;span class="o"&gt;=&lt;/span&gt;John&lt;span class="w"&gt; &lt;/span&gt;--last&lt;span class="o"&gt;=&lt;/span&gt;Testuser&lt;span class="w"&gt; &lt;/span&gt;--cn&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;John Testuser&amp;quot;&lt;/span&gt;
ipa&lt;span class="w"&gt; &lt;/span&gt;user-add-certmapdata&lt;span class="w"&gt; &lt;/span&gt;john&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--issuer&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;CN=Certificate Authority,O=IPA.EXAMPLE.COM&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--subject&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;CN=john,O=IPA.EXAMPLE.COM&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;john&lt;/code&gt; has no password and no Kerberos keys. The card is his only way&amp;nbsp;in.&lt;/p&gt;
&lt;p&gt;The second command is the one that is easy to forget. My certificate mapping rules match on issuer and subject, and without mapping data on the user, the &lt;span class="caps"&gt;KDC&lt;/span&gt; rejects a perfectly valid certificate. I learned that a few days earlier while provisioning my regular card, so this time it went in&amp;nbsp;first.&lt;/p&gt;
&lt;p&gt;Next, a certificate signing request. The private key is on the card, so OpenSSL has to sign the request through &lt;span class="caps"&gt;PKCS&lt;/span&gt;#11. On Fedora 45 with OpenSSL 4 that is the job of the pkcs11&amp;nbsp;provider:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;env&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PKCS11_PROVIDER_MODULE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/usr/lib64/pkcs11/opensc-pkcs11.so&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;openssl&lt;span class="w"&gt; &lt;/span&gt;req&lt;span class="w"&gt; &lt;/span&gt;-new&lt;span class="w"&gt; &lt;/span&gt;-provider&lt;span class="w"&gt; &lt;/span&gt;pkcs11&lt;span class="w"&gt; &lt;/span&gt;-provider&lt;span class="w"&gt; &lt;/span&gt;default&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-key&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pkcs11:token=Personal%20IdM%20SmartCard;id=%01;type=private&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-subj&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/CN=john&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-out&lt;span class="w"&gt; &lt;/span&gt;john.csr
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;OpenSSL asks for the card &lt;span class="caps"&gt;PIN&lt;/span&gt; and writes the request. The IdM &lt;span class="caps"&gt;CA&lt;/span&gt; turns it into a&amp;nbsp;certificate:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;cert-request&lt;span class="w"&gt; &lt;/span&gt;john.csr&lt;span class="w"&gt; &lt;/span&gt;--principal&lt;span class="w"&gt; &lt;/span&gt;john&lt;span class="w"&gt; &lt;/span&gt;--certificate-out&lt;span class="w"&gt; &lt;/span&gt;john.pem
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Before writing anything to the card, I compared the public key of the &lt;span class="caps"&gt;CSR&lt;/span&gt;, of the issued certificate and of&amp;nbsp;key &lt;code&gt;01&lt;/code&gt; on the card. All three were identical. That was not paranoia. A few minutes earlier I had been unsure which directory the &lt;span class="caps"&gt;CSR&lt;/span&gt; had ended up in, and a certificate for the wrong key is a quiet kind of&amp;nbsp;failure.&lt;/p&gt;
&lt;p&gt;Then the certificate goes next to its&amp;nbsp;key:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;env&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;OPENSC_CONF&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$HOME&lt;/span&gt;/javacard/opensc-custom.conf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;pkcs15-init&lt;span class="w"&gt; &lt;/span&gt;--store-certificate&lt;span class="w"&gt; &lt;/span&gt;john.pem&lt;span class="w"&gt; &lt;/span&gt;--auth-id&lt;span class="w"&gt; &lt;/span&gt;ff&lt;span class="w"&gt; &lt;/span&gt;--id&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;01&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;pem
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="the-ticket"&gt;The&amp;nbsp;Ticket&lt;/h3&gt;
&lt;p&gt;A separate credential cache keeps my own tickets out of the&amp;nbsp;experiment:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;❯&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;env&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;KRB5CCNAME&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;FILE&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;krb5cc_john_test&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kinit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;X&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;X509_user_identity&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;PKCS11:token=Personal IdM SmartCard&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;john&lt;/span&gt;&lt;span class="nv"&gt;@IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;COM&lt;/span&gt;
&lt;span class="n"&gt;Personal&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;IdM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SmartCard&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="nl"&gt;PIN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;❯&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;klist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;c&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;FILE&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;tmp&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;krb5cc_john_test&lt;/span&gt;
&lt;span class="k"&gt;Default&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;principal&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;john&lt;/span&gt;&lt;span class="nv"&gt;@IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;COM&lt;/span&gt;

&lt;span class="n"&gt;Valid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;starting&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="n"&gt;Expires&lt;/span&gt;&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="n"&gt;Service&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;principal&lt;/span&gt;
&lt;span class="mf"&gt;17.11.2026&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;41&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="mf"&gt;18.11.2026&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;37&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;krbtgt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;COM&lt;/span&gt;&lt;span class="nv"&gt;@IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;COM&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;renew&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;until&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;24.11.2026&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;30&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;36&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;First attempt, about four seconds including typing the &lt;span class="caps"&gt;PIN&lt;/span&gt;. A ticket-granting ticket for a user without a password, authenticated by a card that had been on its way to the bin that&amp;nbsp;morning.&lt;/p&gt;
&lt;p&gt;A few&amp;nbsp;observations:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Signing is enough.&lt;/strong&gt; The key on the card has the&amp;nbsp;usage &lt;code&gt;sign&lt;/code&gt; and nothing else. &lt;span class="caps"&gt;PKINIT&lt;/span&gt; uses Diffie-Hellman for the key exchange here, and the card only has to sign the&amp;nbsp;request.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Nothing in the chain cares who made the card.&lt;/strong&gt; The &lt;span class="caps"&gt;KDC&lt;/span&gt; sees a certificate from its own &lt;span class="caps"&gt;CA&lt;/span&gt; and a valid signature. Anything OpenSC can present as a &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 token with an &lt;span class="caps"&gt;RSA&lt;/span&gt; key works, as long as the certificate and the mapping data are&amp;nbsp;right.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The ticket came from home.&lt;/strong&gt; The &lt;span class="caps"&gt;KDC&lt;/span&gt; that answered was the &lt;a href="https://hofstede.it/an-idm-replica-at-home-kerberos-that-survives-a-dead-tunnel/"&gt;replica I installed at home&lt;/a&gt; a few days ago. It has no &lt;span class="caps"&gt;CA&lt;/span&gt;. The certificate was issued by a server in the data center, and the home replica validated it with the &lt;span class="caps"&gt;CA&lt;/span&gt; certificate it already had. That split works exactly as&amp;nbsp;designed.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="second-life-an-openpgp-card-for-ssh"&gt;Second Life: An OpenPGP Card for &lt;span class="caps"&gt;SSH&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;With a ticket in the cache, the first question was answered. The second one followed naturally. My daily &lt;span class="caps"&gt;SSH&lt;/span&gt; key lives on a Nitrokey and is served by the &lt;a href="https://hofstede.it/hardware-keys-without-gnupg-the-rust-openpgp-card-stack-now-in-freebsd-ports/"&gt;Rust OpenPGP card stack&lt;/a&gt;. Would that stack accept this card as&amp;nbsp;well?&lt;/p&gt;
&lt;h3 id="swapping-the-applet"&gt;Swapping the&amp;nbsp;Applet&lt;/h3&gt;
&lt;p&gt;The same three-part recipe as before, with a different &lt;span class="caps"&gt;CAP&lt;/span&gt; file. The shell history has a comment at this point, and I am leaving it&amp;nbsp;in:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Sorry John :)&lt;/span&gt;
gp&lt;span class="w"&gt; &lt;/span&gt;--delete&lt;span class="w"&gt; &lt;/span&gt;F276A288BCFBA69D34F310&lt;span class="w"&gt; &lt;/span&gt;-f
gp&lt;span class="w"&gt; &lt;/span&gt;--load&lt;span class="w"&gt; &lt;/span&gt;gpg.cap
gp&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;80E60C003A10D276000124000200000000000000000010D276000124010200000000000000000010D276000124010200FFFF232342420000010003C9010000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The applet is &lt;a href="https://github.com/FluffyKaon/OpenPGP-Card"&gt;OpenPGP-Card&lt;/a&gt; by FluffyKaon, an open source implementation of version 2.0 of the OpenPGP card specification for JavaCard. The repository has not changed since 2015, and it ships an install script written for &lt;span class="caps"&gt;JCOP&lt;/span&gt; cards in GlobalPlatform 2.1.1 mode, so it is a good match for a card of this generation. The &lt;span class="caps"&gt;INSTALL&lt;/span&gt; command has the same structure as the first&amp;nbsp;one:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Bytes&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;80 E6 0C 00 3A&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;INSTALL&lt;/span&gt; for install and make selectable, 58 data bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;10 D2760001240002000000000000000000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Package &lt;span class="caps"&gt;AID&lt;/span&gt;, 16 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;10 D2760001240102000000000000000000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Applet &lt;span class="caps"&gt;AID&lt;/span&gt;, 16 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;10 D276000124010200FFFF232342420000&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Instance &lt;span class="caps"&gt;AID&lt;/span&gt;, 16 bytes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;01 00&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Privileges, one byte&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;03 C9 01 00&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Install parameters&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;00&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Length of the install token, none&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="the-aid-is-the-identity"&gt;The &lt;span class="caps"&gt;AID&lt;/span&gt; Is the&amp;nbsp;Identity&lt;/h3&gt;
&lt;p&gt;The instance &lt;span class="caps"&gt;AID&lt;/span&gt; in that command is worth a closer look, because on an OpenPGP card it is not just an&amp;nbsp;address:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;D2 76 00 01 24 01   02 00   FF FF   23 23 42 42   00 00
└─ OpenPGP app ─┘   └ver┘   └mfr┘   └─ serial ─┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The first six bytes say &amp;#8220;this is an OpenPGP application&amp;#8221;. Then follow the specification version, a manufacturer &lt;span class="caps"&gt;ID&lt;/span&gt; and a serial number. The card ident that tools&amp;nbsp;like &lt;code&gt;oct&lt;/code&gt; show is built from those last two&amp;nbsp;fields.&lt;/p&gt;
&lt;p&gt;Which means the manufacturer and the serial number of an OpenPGP card are whatever the person typing the &lt;span class="caps"&gt;INSTALL&lt;/span&gt; command wants them to be. I&amp;nbsp;picked &lt;code&gt;FFFF&lt;/code&gt;, which is the range reserved for test cards, and a serial number I can&amp;nbsp;remember:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~ ❯ oct list
Available OpenPGP cards:
FFFF:23234242
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;oct info&lt;/code&gt; reads the rest of the application data without complaint: version 2.0, manufacturer &amp;#8220;Testcard&amp;#8221;, key import, command chaining, at most 255 bytes per&amp;nbsp;command.&lt;/p&gt;
&lt;p&gt;For a card you buy, the vendor does this step and you never see it. Doing it by hand takes some of the magic out of those&amp;nbsp;idents.&lt;/p&gt;
&lt;h3 id="stumbling-block-4-gnupg-says-the-card-is-not-there"&gt;Stumbling Block 4: GnuPG Says the Card Is Not&amp;nbsp;There&lt;/h3&gt;
&lt;p&gt;The Rust tools were happy. GnuPG was&amp;nbsp;not:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~ ❯ gpg --card-status
gpg: OpenPGP card not available: No such file or directory
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A branded OpenPGP card in the same reader worked, so the reader and&amp;nbsp;my &lt;code&gt;scdaemon.conf&lt;/code&gt; were not the problem.&amp;nbsp;Restarting &lt;code&gt;scdaemon&lt;/code&gt; changed nothing, and asking it for the serial number directly worked fine. The failure came later. A debug log&amp;nbsp;of &lt;code&gt;scdaemon&lt;/code&gt; during &lt;code&gt;LEARN&lt;/code&gt;, which is&amp;nbsp;what &lt;code&gt;--card-status&lt;/code&gt; triggers, ends like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;send&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;apdu&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;c&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;i&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;47&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;p1&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;81&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;p2&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;lc&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;le&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;em&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;response&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;sw&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="nv"&gt;A82&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nv"&gt;datalen&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;
&lt;span class="nv"&gt;reading&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;public&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;failed&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;No&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;such&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;file&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;or&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;directory&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;While learning a&amp;nbsp;card, &lt;code&gt;scdaemon&lt;/code&gt; reads the public key of every key slot. On a card without keys, this applet&amp;nbsp;answers &lt;code&gt;6A82&lt;/code&gt;, &amp;#8220;file not found&amp;#8221;, and GnuPG takes that as a fatal error for the whole&amp;nbsp;card. &lt;code&gt;oct&lt;/code&gt; treats an empty slot as an empty slot. The fix is simply to put keys on the card&amp;nbsp;first.&lt;/p&gt;
&lt;h3 id="keys"&gt;Keys&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;oct&lt;/code&gt; generates all three keys on the&amp;nbsp;card:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;oct&lt;span class="w"&gt; &lt;/span&gt;admin&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;FFFF:23234242&lt;span class="w"&gt; &lt;/span&gt;generate&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;test.pub&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Test &amp;lt;test@example.org&amp;gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;rsa2048
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~ ❯ oct status --card FFFF:23234242
Signature key:       a968bee8297bbca56504d4360be76a1d645622e1   RSA 2048   created 2026-11-17 20:54:00 UTC
Decryption key:      558b9967f8391e73759dff5728e4004ef71fdd5c   RSA 2048   created 2026-11-17 20:54:17 UTC
Authentication key:  49538308792d7ec8ae84ca1fe81f681dee915448   RSA 2048   created 2026-11-17 20:54:34 UTC
Signatures made: 3
Remaining PIN attempts: User: 3, Admin: 3, Reset Code: 0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Roughly 17 seconds per &lt;span class="caps"&gt;RSA&lt;/span&gt; 2048 key. That is the speed of prime number search on a smart card &lt;span class="caps"&gt;CPU&lt;/span&gt; from that era, and a good reminder of how little hardware is doing this&amp;nbsp;work. &lt;code&gt;oct ssh&lt;/code&gt; prints the &lt;span class="caps"&gt;SSH&lt;/span&gt; public key of the authentication&amp;nbsp;slot.&lt;/p&gt;
&lt;p&gt;With keys in all three slots, GnuPG reads the card as&amp;nbsp;well:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;❯&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;gpg&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="nx"&gt;card&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;status&lt;/span&gt;
&lt;span class="nx"&gt;Reader&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;.........&lt;/span&gt;&lt;span class="p"&gt;..:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Alcor&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Link&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;AK9563&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;02&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;
&lt;span class="nx"&gt;Application&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ID&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;D276000124010200FFFF232342420000&lt;/span&gt;
&lt;span class="nx"&gt;Application&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;.:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;OpenPGP&lt;/span&gt;
&lt;span class="nx"&gt;Version&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;.........&lt;/span&gt;&lt;span class="p"&gt;.:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m m-Double"&gt;2.0&lt;/span&gt;
&lt;span class="nx"&gt;Manufacturer&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;..:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;test&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;card&lt;/span&gt;
&lt;span class="nx"&gt;Serial&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;number&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;.:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;23234242&lt;/span&gt;
&lt;span class="nx"&gt;Key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;attributes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rsa2048&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rsa2048&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rsa2048&lt;/span&gt;
&lt;span class="nx"&gt;Signature&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;.:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;A968&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;BEE8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;297&lt;/span&gt;&lt;span class="nx"&gt;B&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;BCA5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6504&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nx"&gt;D436&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="nx"&gt;BE7&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6&lt;/span&gt;&lt;span class="nx"&gt;A1D&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;6456&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="nx"&gt;E1&lt;/span&gt;
&lt;span class="nx"&gt;Encryption&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;.:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;558&lt;/span&gt;&lt;span class="nx"&gt;B&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9967&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;F839&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="nx"&gt;E73&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;759&lt;/span&gt;&lt;span class="nx"&gt;D&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nx"&gt;FF57&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;28&lt;/span&gt;&lt;span class="nx"&gt;E4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;004&lt;/span&gt;&lt;span class="nx"&gt;E&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;F71F&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;DD5C&lt;/span&gt;
&lt;span class="nx"&gt;Authentication&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;key&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4953&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;8308&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;792&lt;/span&gt;&lt;span class="nx"&gt;D&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt;&lt;span class="nx"&gt;EC8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;AE84&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nx"&gt;CA1F&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;E81F&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;681&lt;/span&gt;&lt;span class="nx"&gt;D&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;EE91&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5448&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="stumbling-block-5-the-agent-has-no-identities"&gt;Stumbling Block 5: The Agent Has No&amp;nbsp;Identities&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~ ❯ echo $SSH_AUTH_SOCK
/run/user/1000/openpgp-card/ssh-agent.sock

~ ❯ ssh-add -L
The agent has no identities.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The card was&amp;nbsp;fine. &lt;code&gt;oct list&lt;/code&gt;, &lt;code&gt;oct status&lt;/code&gt; and &lt;code&gt;oct ssh&lt;/code&gt; all worked. Only the &lt;span class="caps"&gt;SSH&lt;/span&gt; agent could not see&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;This one I did to myself. In January I wrote about &lt;a href="https://hofstede.it/card-wars-hiding-smartcard-readers-from-eager-rust-agents-with-ld_preload/"&gt;hiding a smart card reader from the agent with LD_PRELOAD&lt;/a&gt;. The agent&amp;#8217;s systemd unit still has that&amp;nbsp;drop-in:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;LD_PRELOAD=%h/.local/lib/libhackyreaderfix.so&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The library hides the laptop&amp;#8217;s built-in Alcor reader from the agent, because my &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 card normally lives there and the agent used to abort when it met a card without an OpenPGP applet. And the built-in Alcor reader is exactly where the freshly minted OpenPGP JavaCard was sitting. My own workaround from ten months ago was filtering out my own&amp;nbsp;experiment.&lt;/p&gt;
&lt;p&gt;The proper fix exists upstream by now. I sent pull requests so that the Rust stack skips cards without an OpenPGP applet instead of failing. They are released in openpgp-card 0.7.3 and openpgp-card-ssh-agent 0.3.9, and the maintainers fixed the same thing in openpgp-card-tools 0.11.16. My laptop still runs 0.3.7 of the agent, so the workaround is still active and can go with the next package rebuild. For that evening, the pragmatic fix was a different reader. I have enough of them. In an external &lt;span class="caps"&gt;ACS&lt;/span&gt; &lt;span class="caps"&gt;ACR39U&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~ ❯ ssh-add -L
ssh-rsa AAAAB3NzaC1yc2E...q3h FFFF:23234242
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="stumbling-block-6-agent-refused-operation"&gt;Stumbling Block 6: Agent Refused&amp;nbsp;Operation&lt;/h3&gt;
&lt;p&gt;With the public key&amp;nbsp;in &lt;code&gt;authorized_keys&lt;/code&gt; on a FreeBSD test host, the first login&amp;nbsp;attempt:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;sign_and_send_pubkey&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;signing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;failed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;RSA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FFFF:23234242&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;refused&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;operation&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The agent&amp;#8217;s journal explains&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ERROR openpgp_card_ssh_agent] No User PIN for card FFFF:23234242 found via openpgp_card_state
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;openpgp-card-ssh-agent&lt;/code&gt; never prompts for a &lt;span class="caps"&gt;PIN&lt;/span&gt; in the middle of a signature. It looks the &lt;span class="caps"&gt;PIN&lt;/span&gt; up in its state store, and for a card that was born ten minutes ago there is nothing to find. The way to store one is the standard smart card option&amp;nbsp;of &lt;code&gt;ssh-add&lt;/code&gt;, with the card ident as the&amp;nbsp;argument:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;❯&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;ssh&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;add&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nv"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;FFFF&lt;/span&gt;:&lt;span class="mi"&gt;23234242&lt;/span&gt;
&lt;span class="nv"&gt;Enter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;passphrase&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PKCS&lt;/span&gt;&lt;span class="sc"&gt;#11&lt;/span&gt;:
&lt;span class="nv"&gt;Card&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;added&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;FFFF&lt;/span&gt;:&lt;span class="mi"&gt;23234242&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The prompt says &amp;#8220;&lt;span class="caps"&gt;PKCS&lt;/span&gt;#11&amp;#8221; because that text is hard-coded in OpenSSH. What it wants is the OpenPGP user &lt;span class="caps"&gt;PIN&lt;/span&gt;. In the interest of honesty: my first attempt was answered&amp;nbsp;with &lt;code&gt;Could not add card&lt;/code&gt;, and the cause was a mistyped &lt;span class="caps"&gt;PIN&lt;/span&gt;. The agent verifies the &lt;span class="caps"&gt;PIN&lt;/span&gt; against the card before storing it, which cost one of three tries. A correct &lt;span class="caps"&gt;PIN&lt;/span&gt; resets the&amp;nbsp;counter.&lt;/p&gt;
&lt;h3 id="the-login"&gt;The&amp;nbsp;Login&lt;/h3&gt;
&lt;p&gt;The relevant lines&amp;nbsp;of &lt;code&gt;ssh -vvv&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;debug1&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Will&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;attempt&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;FFFF&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;23234242&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;RSA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SHA256&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;c6Rum6kzcDSeS01b4DTr&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;SUVAsAaGe8qrEhhmNlqUww&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;
&lt;span class="n"&gt;debug1&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Offering&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kd"&gt;public&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;FFFF&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;23234242&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;RSA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SHA256&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;c6Rum6kzcDSeS01b4DTr&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;SUVAsAaGe8qrEhhmNlqUww&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;
&lt;span class="n"&gt;debug1&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;accepts&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;key&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;FFFF&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;23234242&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;RSA&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SHA256&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;c6Rum6kzcDSeS01b4DTr&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;SUVAsAaGe8qrEhhmNlqUww&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;agent&lt;/span&gt;
&lt;span class="n"&gt;debug3&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sign_and_send_pubkey&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;signing&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;using&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rsa&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;sha2&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;256&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SHA256&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="n"&gt;c6Rum6kzcDSeS01b4DTr&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="n"&gt;SUVAsAaGe8qrEhhmNlqUww&lt;/span&gt;
&lt;span class="n"&gt;Authenticated&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bsd01&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;example&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="na"&gt;com&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;([&lt;/span&gt;&lt;span class="mf"&gt;192.0&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="mf"&gt;2.20&lt;/span&gt;&lt;span class="o"&gt;]:&lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;using&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;publickey&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;
&lt;span class="o"&gt;...&lt;/span&gt;
&lt;span class="n"&gt;FreeBSD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;15.1&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;RELEASE&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;p3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="n"&gt;GENERIC&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A shell on FreeBSD 15.1, authenticated by a signature that was computed inside a card I had nearly thrown&amp;nbsp;out.&lt;/p&gt;
&lt;p&gt;Two things in the same log look like problems and are&amp;nbsp;not:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ssh_agent_bind_hostkey: agent refused operation&lt;/code&gt;.&lt;/strong&gt; Newer OpenSSH clients send a session-bind extension to the agent. This agent does not implement it, says so, and &lt;span class="caps"&gt;SSH&lt;/span&gt; carries&amp;nbsp;on.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Connection multiplexing hides the card.&lt;/strong&gt; My &lt;span class="caps"&gt;SSH&lt;/span&gt; config keeps master connections open for hours. A second login reuses the existing connection and never asks the card for anything. For repeated tests, close the master first&amp;nbsp;with &lt;code&gt;ssh -O exit &amp;lt;host&amp;gt;&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="stumbling-blocks-at-a-glance"&gt;Stumbling Blocks at a&amp;nbsp;Glance&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;#&lt;/th&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;Cause&lt;/th&gt;
&lt;th&gt;Fix&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gp --install&lt;/code&gt; fails&amp;nbsp;with &lt;code&gt;6A80&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;GlobalPlatformPro sends a three-byte privileges field, the &lt;span class="caps"&gt;GP&lt;/span&gt; 2.1.1 card wants one byte&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gp --load&lt;/code&gt;, then a raw &lt;span class="caps"&gt;INSTALL&lt;/span&gt; with one-byte privileges&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;2&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gp --delete &amp;lt;instance AID&amp;gt;&lt;/code&gt; fails&amp;nbsp;with &lt;code&gt;6985&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;This card will not remove an instance on its own&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gp --delete &amp;lt;package AID&amp;gt; -f&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;td&gt;Token has a generic label&lt;/td&gt;
&lt;td&gt;Label and manufacturer come from the OpenSC profile&lt;/td&gt;
&lt;td&gt;Private profile copy&amp;nbsp;via &lt;code&gt;OPENSC_CONF&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;--auth-id ff&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gpg --card-status&lt;/code&gt;: card not available&lt;/td&gt;
&lt;td&gt;&lt;code&gt;scdaemon&lt;/code&gt; reads every key slot, the empty applet&amp;nbsp;answers &lt;code&gt;6A82&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Generate keys first&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ssh-add -L&lt;/code&gt; shows no identities&lt;/td&gt;
&lt;td&gt;My own LD_PRELOAD workaround hides the reader&lt;/td&gt;
&lt;td&gt;Another reader, or a current agent without the workaround&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;6&lt;/td&gt;
&lt;td&gt;&lt;code&gt;agent refused operation&lt;/code&gt; when signing&lt;/td&gt;
&lt;td&gt;The agent has no stored &lt;span class="caps"&gt;PIN&lt;/span&gt; and never prompts&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ssh-add -s &amp;lt;card ident&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;There is one more that did not happen: a valid certificate that the &lt;span class="caps"&gt;KDC&lt;/span&gt; rejects because the user has no certificate mapping&amp;nbsp;data. &lt;code&gt;ipa user-add-certmapdata&lt;/code&gt; prevents&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="what-i-did-not-test"&gt;What I Did Not&amp;nbsp;Test&lt;/h2&gt;
&lt;p&gt;The evening proved two things and it is worth being precise about the&amp;nbsp;rest.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The two applets were never on the card at the same time.&lt;/strong&gt; IsoApplet was deleted before the OpenPGP applet was loaded. Whether the &lt;span class="caps"&gt;J3A081&lt;/span&gt; has room for both is an open question, and an interesting one, because a single card that does &lt;span class="caps"&gt;PKINIT&lt;/span&gt; and &lt;span class="caps"&gt;SSH&lt;/span&gt; would be a nice thing to&amp;nbsp;have.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Only &lt;code&gt;kinit&lt;/code&gt;.&lt;/strong&gt; I did not log in to a host&amp;nbsp;as &lt;code&gt;john&lt;/code&gt;, neither through &lt;span class="caps"&gt;SSH&lt;/span&gt; with &lt;span class="caps"&gt;GSSAPI&lt;/span&gt; nor through &lt;span class="caps"&gt;SSSD&lt;/span&gt; on a&amp;nbsp;console.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Only &lt;span class="caps"&gt;SSH&lt;/span&gt; authentication.&lt;/strong&gt; No Git signing and no decryption with the OpenPGP&amp;nbsp;keys.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Only &lt;span class="caps"&gt;RSA&lt;/span&gt; 2048.&lt;/strong&gt; No elliptic curve keys on either&amp;nbsp;applet.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;GnuPG only looked.&lt;/strong&gt; Once the card had&amp;nbsp;keys, &lt;code&gt;gpg --card-status&lt;/code&gt; read it, but every operation that used a key went through the Rust stack&amp;nbsp;with &lt;code&gt;oct&lt;/code&gt; and &lt;code&gt;openpgp-card-ssh-agent&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No benchmarks.&lt;/strong&gt; &amp;#8220;About four seconds&amp;#8221; is a shell prompt timer, not a&amp;nbsp;measurement.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Two caveats belong here as well. The card accepted GlobalPlatform&amp;#8217;s well-known default keys, so anyone who gets hold of it can replace its applets. They cannot read the private keys out, but they can wipe them. For a card that is meant to be used seriously, changing those keys is the first step, and it is also the step with the most potential to brick the card. And &lt;span class="caps"&gt;RSA&lt;/span&gt; 2048 on a chip of this age is fine for an experiment. My daily keys stay on much younger&amp;nbsp;hardware.&lt;/p&gt;
&lt;p&gt;As&amp;nbsp;for &lt;code&gt;john&lt;/code&gt;: his private key was destroyed together with the IsoApplet, so the certificate that is still valid in IdM until 2028 has nothing left to prove possession with. Revoking it and deleting the user is the cleanup that&amp;nbsp;remains.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The same piece of plastic was an enterprise login token at 20:30 and an &lt;span class="caps"&gt;SSH&lt;/span&gt; key at 22:08. Nothing about the hardware changed in between. Only the applet&amp;nbsp;did.&lt;/p&gt;
&lt;p&gt;That is the part I found worth writing down. Smart cards tend to be treated as fixed-function devices, a &amp;#8220;&lt;span class="caps"&gt;PIV&lt;/span&gt; card&amp;#8221; or an &amp;#8220;OpenPGP card&amp;#8221;, bought in that shape and used in that shape. Underneath, many of them are general-purpose computers with a loader, and the function is a file of a few kilobytes. Everything else in the chain was standard software talking to standard interfaces: OpenSC, the OpenSSL &lt;span class="caps"&gt;PKCS&lt;/span&gt;#11 provider, &lt;span class="caps"&gt;MIT&lt;/span&gt; Kerberos and IdM on one side, the Rust OpenPGP card tools and OpenSSH on the other. None of them asked where the card came&amp;nbsp;from.&lt;/p&gt;
&lt;p&gt;The stumbling blocks were all at the edges. A management tool that is many years younger than the card. A profile file in a system directory. A workaround I had written myself and forgotten. A client that treats an empty key slot as a broken card. None of them had anything to do with cryptography, and each was solved in minutes once the error code was taken&amp;nbsp;literally.&lt;/p&gt;
&lt;p&gt;The card is not going to the recycling yard. As I write this, it still answers&amp;nbsp;to &lt;code&gt;FFFF:23234242&lt;/code&gt;.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/martinpaljak/GlobalPlatformPro"&gt;GlobalPlatformPro&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/philipWendland/IsoApplet"&gt;IsoApplet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/OpenSC/OpenSC"&gt;OpenSC&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/FluffyKaon/OpenPGP-Card"&gt;OpenPGP-Card applet by&amp;nbsp;FluffyKaon&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://codeberg.org/openpgp-card"&gt;OpenPGP Card&amp;nbsp;project&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/an-idm-replica-at-home-kerberos-that-survives-a-dead-tunnel/"&gt;An IdM Replica at Home: Kerberos That Survives a Dead&amp;nbsp;Tunnel&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/hardware-keys-without-gnupg-the-rust-openpgp-card-stack-now-in-freebsd-ports/"&gt;Hardware Keys Without GnuPG: The Rust OpenPGP Card Stack, Now in FreeBSD&amp;nbsp;Ports&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/card-wars-hiding-smartcard-readers-from-eager-rust-agents-with-ld_preload/"&gt;Card Wars: Hiding Smartcard Readers from Eager Rust Agents with&amp;nbsp;LD_PRELOAD&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="javacard"/><category term="smartcard"/><category term="globalplatform"/><category term="opensc"/><category term="pkcs15"/><category term="freeipa"/><category term="idm"/><category term="kerberos"/><category term="pkinit"/><category term="openpgp"/><category term="ssh"/><category term="fedora"/><category term="security"/></entry><entry><title>Git Beyond the Basics: Worktrees, Cherry-Picks and Signed Commits</title><link href="https://hofstede.it/git-beyond-the-basics-worktrees-cherry-picks-and-signed-commits/" rel="alternate"/><published>2026-11-16T00:00:00+01:00</published><updated>2026-11-16T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-11-16:/git-beyond-the-basics-worktrees-cherry-picks-and-signed-commits/</id><summary type="html">&lt;p&gt;Almost every Git tutorial stops after add, commit, push and a merge conflict. The tools that make Git pleasant in daily work get far less attention: worktrees for parallel feature branches, cherry-picks for backports, fixup commits and autosquash for clean history, the reflog for recovering from mistakes, and signed commits that prove who made them. This article starts with the mental model and the basics, then works through the parts I actually use, including signing with an OpenPGP smartcard through&amp;nbsp;oct-git.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Isometric illustration of three glowing commit graphs in cyan, orange and white branching and running in parallel between stacks of editor windows, next to a hardware security key under a shield with a padlock" src="https://hofstede.it/images/2026-11-16-git-beyond-the-basics.webp" title="Parallel branches, one history, and a key that signs it."&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-mental-model"&gt;The Mental&amp;nbsp;Model&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-basics-briefly"&gt;The Basics,&amp;nbsp;Briefly&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-configuration-worth-having"&gt;A Configuration Worth Having&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#different-identities-per-directory"&gt;Different Identities per&amp;nbsp;Directory&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#merge-or-rebase"&gt;Merge or Rebase&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#rerere-resolve-each-conflict-once"&gt;rerere: Resolve Each Conflict&amp;nbsp;Once&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#recovering-from-mistakes"&gt;Recovering From Mistakes&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-reflog"&gt;The&amp;nbsp;Reflog&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#undoing-a-stacked-rebase"&gt;Undoing a Stacked&amp;nbsp;Rebase&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-the-reflog-doesnt-know"&gt;What the Reflog Doesn&amp;#8217;t&amp;nbsp;Know&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-three-kinds-of-reset"&gt;The Three Kinds of&amp;nbsp;Reset&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#cleaning-up-before-you-share"&gt;Cleaning Up Before You Share&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#fixup-commits-and-autosquash"&gt;Fixup Commits and&amp;nbsp;Autosquash&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#stacked-branches-with-update-refs"&gt;Stacked Branches With&amp;nbsp;&amp;#8212;update-refs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#comparing-two-versions-of-a-branch"&gt;Comparing Two Versions of a&amp;nbsp;Branch&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#cherry-picking"&gt;Cherry-Picking&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#ranges-and-conflicts"&gt;Ranges and&amp;nbsp;Conflicts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#when-not-to-cherry-pick"&gt;When Not to&amp;nbsp;Cherry-Pick&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#worktrees-several-branches-at-once"&gt;Worktrees: Several Branches at Once&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#reviewing-pull-requests-in-a-worktree"&gt;Reviewing Pull Requests in a&amp;nbsp;Worktree&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-is-shared-and-what-isnt"&gt;What Is Shared and What&amp;nbsp;Isn&amp;#8217;t&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#how-i-lay-them-out"&gt;How I Lay Them&amp;nbsp;Out&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#finding-things-in-history"&gt;Finding Things in&amp;nbsp;History&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#signing-commits"&gt;Signing Commits&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#what-a-signature-proves-and-what-it-doesnt"&gt;What a Signature Proves, and What It&amp;nbsp;Doesn&amp;#8217;t&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#three-signature-formats"&gt;Three Signature&amp;nbsp;Formats&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#openpgp-on-a-smartcard-with-oct-git"&gt;OpenPGP on a Smartcard With&amp;nbsp;oct-git&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rewriting-history-drops-signatures"&gt;Rewriting History Drops&amp;nbsp;Signatures&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#checking-signatures"&gt;Checking&amp;nbsp;Signatures&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#lazygit-the-same-ideas-fewer-keystrokes"&gt;lazygit: The Same Ideas, Fewer&amp;nbsp;Keystrokes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#housekeeping"&gt;Housekeeping&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-this-article-doesnt-cover"&gt;What This Article Doesn&amp;#8217;t&amp;nbsp;Cover&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-command-card-i-actually-use"&gt;The Command Card I Actually&amp;nbsp;Use&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#further-reading"&gt;Further&amp;nbsp;Reading&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;Git is one of those tools everybody uses and few people enjoy. There is no shortage of tutorials, but they all seem to cover the same twenty&amp;nbsp;minutes: &lt;code&gt;init&lt;/code&gt;, &lt;code&gt;add&lt;/code&gt;, &lt;code&gt;commit&lt;/code&gt;, &lt;code&gt;push&lt;/code&gt;, a branch, a merge conflict, done. After that you are on your own, and &amp;#8220;on your own&amp;#8221; usually means a Stack Overflow answer from 2014 that tells you&amp;nbsp;to &lt;code&gt;git stash&lt;/code&gt; your way out of&amp;nbsp;everything.&lt;/p&gt;
&lt;p&gt;The part that is missing is the middle. Not Git internals, not plumbing commands, just the dozen features that turn Git from something you survive into something that works for you. This article starts with the basics, because the advanced parts only make sense on top of a correct mental model. Then it gradually gets deeper. Skip ahead if you already know where&amp;nbsp;your &lt;code&gt;HEAD&lt;/code&gt; is.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The short version:&lt;/strong&gt; commits are snapshots, branches are movable labels, and almost nothing you have committed is lost. Once that clicks, worktrees, cherry-picks and rebases stop being scary. Sign what you publish, and remember that rewriting history drops signatures unless you ask Git to sign&amp;nbsp;again.&lt;/p&gt;
&lt;p&gt;Everything in this article was checked against Git 2.56.1 on Fedora 45. The examples all come from one small test repository, a toy &lt;span class="caps"&gt;INI&lt;/span&gt; parser&amp;nbsp;called &lt;code&gt;inigo&lt;/code&gt;, so the commit IDs in one section match the ones in the next. Nothing here needs a bleeding-edge Git, but a few features have minimum&amp;nbsp;versions: &lt;code&gt;--force-if-includes&lt;/code&gt; needs Git 2.30, &lt;span class="caps"&gt;SSH&lt;/span&gt; signing&amp;nbsp;2.34, &lt;code&gt;zdiff3&lt;/code&gt; 2.35, &lt;code&gt;push.autoSetupRemote&lt;/code&gt; 2.37&amp;nbsp;and &lt;code&gt;--update-refs&lt;/code&gt; 2.38. If you are on an older enterprise distribution,&amp;nbsp;check &lt;code&gt;git --version&lt;/code&gt; first.&lt;/p&gt;
&lt;h2 id="the-mental-model"&gt;The Mental&amp;nbsp;Model&lt;/h2&gt;
&lt;p&gt;Most Git confusion comes from one wrong picture: that Git stores changes. It doesn&amp;#8217;t, at least not conceptually. A commit is a snapshot of the whole tree, plus metadata (author, committer, date, message) and a pointer to its parent commit or commits. Diffs are computed when you ask for&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;From that, everything else&amp;nbsp;follows:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Commits form a graph. Each commit points back to its parent, and a merge commit has two parents. The history is a directed acyclic graph, not a&amp;nbsp;list.&lt;/li&gt;
&lt;li&gt;A commit&amp;#8217;s &lt;span class="caps"&gt;ID&lt;/span&gt; is a hash of its content and its parents. Change anything, the message, the author date, a parent, and you get a different commit with a different &lt;span class="caps"&gt;ID&lt;/span&gt;. This is why &amp;#8220;rewriting history&amp;#8221; really means &amp;#8220;creating new commits and moving labels to&amp;nbsp;them&amp;#8221;.&lt;/li&gt;
&lt;li&gt;A branch is a label pointing at one commit. That is all. Creating a branch costs almost nothing: Git just stores a commit &lt;span class="caps"&gt;ID&lt;/span&gt; under a name. When you commit on a branch, the label moves forward to the new&amp;nbsp;commit.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;HEAD&lt;/code&gt; is where you are. Usually it points at a branch, which points at a commit. If it points directly at a commit, you are in &amp;#8220;detached &lt;span class="caps"&gt;HEAD&lt;/span&gt;&amp;#8221; state, which sounds alarming and&amp;nbsp;isn&amp;#8217;t.&lt;/li&gt;
&lt;li&gt;The index (staging area) is the next commit being&amp;nbsp;prepared. &lt;code&gt;git add&lt;/code&gt; copies content into&amp;nbsp;it, &lt;code&gt;git commit&lt;/code&gt; turns it into a&amp;nbsp;snapshot.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;So there are three places a file can differ: your working directory, the index and the last commit&amp;nbsp;(&lt;code&gt;HEAD&lt;/code&gt;). &lt;code&gt;git diff&lt;/code&gt; compares working directory to&amp;nbsp;index, &lt;code&gt;git diff --staged&lt;/code&gt; compares index&amp;nbsp;to &lt;code&gt;HEAD&lt;/code&gt;. Once you have that triangle in your head, most commands are just ways of moving content between its&amp;nbsp;corners.&lt;/p&gt;
&lt;h2 id="the-basics-briefly"&gt;The Basics,&amp;nbsp;Briefly&lt;/h2&gt;
&lt;p&gt;If you have used Git before, skim this. If you haven&amp;#8217;t, this is the minimum&amp;nbsp;set:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;clone&lt;span class="w"&gt; &lt;/span&gt;https://git.example.org/me/inigo.git
&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;inigo

git&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# what changed, what is staged&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;diff&lt;span class="w"&gt;                   &lt;/span&gt;&lt;span class="c1"&gt;# unstaged changes&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# stage changes hunk by hunk&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;diff&lt;span class="w"&gt; &lt;/span&gt;--staged&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="c1"&gt;# what will be committed&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;commit&lt;span class="w"&gt;                 &lt;/span&gt;&lt;span class="c1"&gt;# opens the editor for a message&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;--oneline&lt;span class="w"&gt; &lt;/span&gt;--graph&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# history as a graph&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;push
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two habits from that list are worth adopting from day&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;Use &lt;code&gt;git add -p&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;git add .&lt;/code&gt;. It walks through every change and asks whether to stage it. It&amp;#8217;s the cheapest code review you will ever get, and it is where you notice the&amp;nbsp;debug &lt;code&gt;print&lt;/code&gt; and the commented-out block before they end up in a&amp;nbsp;commit.&lt;/p&gt;
&lt;p&gt;Use &lt;code&gt;switch&lt;/code&gt; and &lt;code&gt;restore&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;checkout&lt;/code&gt;. &lt;code&gt;git checkout&lt;/code&gt; does two unrelated jobs: switching branches and overwriting files. That&amp;#8217;s how people lose work by typing a filename where they meant a branch. The newer commands split those&amp;nbsp;jobs:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Old&lt;/th&gt;
&lt;th&gt;New&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Switch to a branch&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git checkout main&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git switch main&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create and switch&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git checkout -b feature&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git switch -c feature&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Go back to the previous branch&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git checkout -&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git switch -&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Discard working copy changes&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git checkout -- file&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git restore file&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unstage a file&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git reset HEAD file&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git restore --staged file&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;code&gt;git checkout&lt;/code&gt; still works and will keep working. I just find the new commands harder to&amp;nbsp;misuse.&lt;/p&gt;
&lt;h2 id="a-configuration-worth-having"&gt;A Configuration Worth&amp;nbsp;Having&lt;/h2&gt;
&lt;p&gt;Git&amp;#8217;s defaults are conservative and mostly reflect decisions made for the Linux kernel workflow in 2005. A handful of settings make daily work noticeably better. This is roughly what I set on a new&amp;nbsp;machine:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;init.defaultBranch&lt;span class="w"&gt; &lt;/span&gt;main
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;pull.ff&lt;span class="w"&gt; &lt;/span&gt;only
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;push.autoSetupRemote&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;fetch.prune&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;merge.conflictStyle&lt;span class="w"&gt; &lt;/span&gt;zdiff3
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;diff.algorithm&lt;span class="w"&gt; &lt;/span&gt;histogram
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;rerere.enabled&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;rebase.autoSquash&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;rebase.updateRefs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;branch.sort&lt;span class="w"&gt; &lt;/span&gt;-committerdate
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;What each one&amp;nbsp;does:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Setting&lt;/th&gt;
&lt;th&gt;Effect&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;pull.ff only&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git pull&lt;/code&gt; only fast-forwards. If your branch and the remote have diverged, it stops and lets you decide between merge and rebase, instead of silently creating a merge commit.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;push.autoSetupRemote&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The&amp;nbsp;first &lt;code&gt;git push&lt;/code&gt; on a new branch sets the upstream. No&amp;nbsp;more &lt;code&gt;--set-upstream origin feature&lt;/code&gt; error message.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;fetch.prune&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Remote-tracking branches that were deleted on the server disappear locally on fetch.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;merge.conflictStyle zdiff3&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Conflict markers also show the common ancestor. You see what both sides changed, not just what they ended up with. This one setting made conflicts much less confusing for me.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;diff.algorithm histogram&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Often produces more readable diffs than the default, especially when code was moved around.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rerere.enabled&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Git remembers how you resolved a conflict and reapplies the resolution next time. More on that in the next section.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rebase.autoSquash&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git rebase -i&lt;/code&gt; automatically&amp;nbsp;moves &lt;code&gt;fixup!&lt;/code&gt; commits to the right place.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rebase.updateRefs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;When rebasing a stack of branches, Git moves every branch label in the stack, not only the one you are on.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;branch.sort -committerdate&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git branch&lt;/code&gt; lists recently used branches first.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="different-identities-per-directory"&gt;Different Identities per&amp;nbsp;Directory&lt;/h3&gt;
&lt;p&gt;If you use the same machine for work and private projects, you need two identities. The clumsy way is to&amp;nbsp;remember &lt;code&gt;git config user.email&lt;/code&gt; in every new clone. The better way is conditional&amp;nbsp;includes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# ~/.gitconfig&lt;/span&gt;
&lt;span class="k"&gt;[user]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Christian Hofstede-Kuhn&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;email&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;private@example.org&lt;/span&gt;

&lt;span class="k"&gt;[includeIf &amp;quot;gitdir:~/work/&amp;quot;]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;path&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;~/.gitconfig-work&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# ~/.gitconfig-work&lt;/span&gt;
&lt;span class="k"&gt;[user]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;email&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;me@work.example.com&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every repository&amp;nbsp;under &lt;code&gt;~/work/&lt;/code&gt; picks up the work address automatically. The trailing slash matters: it means &amp;#8220;this directory and everything below it&amp;#8221;. The same trick works for different signing keys, which becomes relevant&amp;nbsp;later.&lt;/p&gt;
&lt;h2 id="merge-or-rebase"&gt;Merge or&amp;nbsp;Rebase&lt;/h2&gt;
&lt;p&gt;This is the oldest argument in Git, and I don&amp;#8217;t think there is one right answer. Here is what each does and when I use&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;Merge creates a new commit with two parents. History stays exactly as it happened, including every &amp;#8220;wip&amp;#8221; and &amp;#8220;fix typo&amp;#8221; commit. Nothing is rewritten, so nothing that anybody else has can&amp;nbsp;break.&lt;/p&gt;
&lt;p&gt;Rebase takes your commits and replays them on top of another commit, creating new commits with new IDs. History becomes linear and easier to read. The old commits still exist for a while, as the next section shows, but your branch no longer points at&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;My rule is simple: rebase what is still mine, merge what is shared. Before I open a pull request, I rebase my feature branch onto the&amp;nbsp;current &lt;code&gt;main&lt;/code&gt; and clean it up. Once others have built on top of a branch, I don&amp;#8217;t rewrite it any more. The &lt;span class="caps"&gt;PR&lt;/span&gt; itself is then merged on the&amp;nbsp;forge.&lt;/p&gt;
&lt;p&gt;When you do have to force-push a rebased branch, never use a&amp;nbsp;plain &lt;code&gt;--force&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;push&lt;span class="w"&gt; &lt;/span&gt;--force-with-lease&lt;span class="w"&gt; &lt;/span&gt;--force-if-includes
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;--force-with-lease&lt;/code&gt; refuses to overwrite the remote branch if it changed since you last fetched it, so you don&amp;#8217;t destroy a colleague&amp;#8217;s&amp;nbsp;push. &lt;code&gt;--force-if-includes&lt;/code&gt; closes a gap in that check: a&amp;nbsp;background &lt;code&gt;git fetch&lt;/code&gt; from your editor updates your remote-tracking refs, and&amp;nbsp;then &lt;code&gt;--force-with-lease&lt;/code&gt; alone would happily overwrite commits you never actually looked at. Together they are&amp;nbsp;what &lt;code&gt;--force&lt;/code&gt; should have been. I have an alias for&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;alias.pushf&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;push --force-with-lease --force-if-includes&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="rerere-resolve-each-conflict-once"&gt;rerere: Resolve Each Conflict&amp;nbsp;Once&lt;/h3&gt;
&lt;p&gt;Merges and rebases both produce conflicts, and a long-lived branch tends to produce the same conflict again and&amp;nbsp;again. &lt;code&gt;rerere&lt;/code&gt; stands for &amp;#8220;reuse recorded resolution&amp;#8221;.&amp;nbsp;With &lt;code&gt;rerere.enabled&lt;/code&gt; set, Git saves the conflict and your resolution whenever you resolve one.&amp;nbsp;In &lt;code&gt;inigo&lt;/code&gt;, two branches reworded the &lt;span class="caps"&gt;README&lt;/span&gt; title differently. The first merge conflicts as usual, and Git records&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;CONFLICT (content): Merge conflict in README.md
Recorded preimage for &amp;#39;README.md&amp;#39;
Automatic merge failed; fix conflicts and then commit the result.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After I resolve and commit it, Git&amp;nbsp;prints &lt;code&gt;Recorded resolution for 'README.md'&lt;/code&gt;. When I throw that merge away and merge again, the same conflict gets my old resolution&amp;nbsp;automatically:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;CONFLICT (content): Merge conflict in README.md
Resolved &amp;#39;README.md&amp;#39; using previous resolution.
Automatic merge failed; fix conflicts and then commit the result.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This matters most when you rebase a long-lived branch repeatedly, or when you test-merge a branch, throw the merge away and do it again later. Git still stops and marks the merge as conflicted, so you review the result&amp;nbsp;and &lt;code&gt;git add&lt;/code&gt; it yourself. It just saves you the&amp;nbsp;typing.&lt;/p&gt;
&lt;h2 id="recovering-from-mistakes"&gt;Recovering From&amp;nbsp;Mistakes&lt;/h2&gt;
&lt;p&gt;Before the sections that rewrite history, the safety net. Rebases, resets and amends create new commits and move labels. The old commits stay in the object database until garbage collection removes them, and that takes a while. Once something has been committed, it is very hard to lose it by&amp;nbsp;accident.&lt;/p&gt;
&lt;h3 id="the-reflog"&gt;The&amp;nbsp;Reflog&lt;/h3&gt;
&lt;p&gt;Git records every movement&amp;nbsp;of &lt;code&gt;HEAD&lt;/code&gt; and of every branch in the reflog. That is your undo&amp;nbsp;history. &lt;code&gt;inigo&lt;/code&gt; has&amp;nbsp;a &lt;code&gt;parser&lt;/code&gt; branch and&amp;nbsp;a &lt;code&gt;cli&lt;/code&gt; branch stacked on top of it, and here is the reflog right after rebasing them in the next&amp;nbsp;section:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ git reflog
c4137b3 HEAD@{0}: rebase (finish): returning to refs/heads/cli
c4137b3 HEAD@{1}: rebase (pick): cli: add --config option
9b6891a HEAD@{2}: rebase (pick): parser: support include files
70e4cd2 HEAD@{3}: rebase (fixup): parser: parse sections and keys
43cfa17 HEAD@{4}: rebase (pick): parser: parse sections and keys
ecf2e5a HEAD@{5}: rebase (start): checkout main
c8be862 HEAD@{6}: checkout: moving from main to cli
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Botched the rebase? Find the entry from just&amp;nbsp;before &lt;code&gt;rebase (start)&lt;/code&gt; and reset to&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;reset&lt;span class="w"&gt; &lt;/span&gt;--hard&lt;span class="w"&gt; &lt;/span&gt;HEAD@&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="m"&gt;6&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Branches have their own reflog too, and that is usually the easier&amp;nbsp;handle. &lt;code&gt;cli@{1}&lt;/code&gt; is&amp;nbsp;where &lt;code&gt;cli&lt;/code&gt; pointed before its last change. A rebase only moves the branch once, when it finishes, so after a&amp;nbsp;rebase &lt;code&gt;cli@{1}&lt;/code&gt; is the old version of the&amp;nbsp;branch:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;reset&lt;span class="w"&gt; &lt;/span&gt;--hard&lt;span class="w"&gt; &lt;/span&gt;cli@&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You may have&amp;nbsp;seen &lt;code&gt;ORIG_HEAD&lt;/code&gt; recommended for this. Rebase, merge and reset do store the previous position there, but&amp;nbsp;every &lt;code&gt;git reset&lt;/code&gt; overwrites it. If you ran a reset during the rebase, for example to split a commit in&amp;nbsp;an &lt;code&gt;edit&lt;/code&gt; step, &lt;code&gt;ORIG_HEAD&lt;/code&gt; points somewhere in the middle of the rebase, not at the old branch. I tested that,&amp;nbsp;and &lt;code&gt;cli@{1}&lt;/code&gt; was still correct&amp;nbsp;while &lt;code&gt;ORIG_HEAD&lt;/code&gt; was not. The branch reflog is the form I&amp;nbsp;trust.&lt;/p&gt;
&lt;h3 id="undoing-a-stacked-rebase"&gt;Undoing a Stacked&amp;nbsp;Rebase&lt;/h3&gt;
&lt;p&gt;One catch if you follow my configuration and&amp;nbsp;set &lt;code&gt;rebase.updateRefs&lt;/code&gt;. A rebase of a stack moves several branches, and resetting the one you are on restores only that one.&amp;nbsp;In &lt;code&gt;inigo&lt;/code&gt;, &lt;code&gt;cli&lt;/code&gt; is stacked&amp;nbsp;on &lt;code&gt;parser&lt;/code&gt;. To undo the rebase, both need to go&amp;nbsp;back:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;reset&lt;span class="w"&gt; &lt;/span&gt;--hard&lt;span class="w"&gt; &lt;/span&gt;cli@&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;branch&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;parser&lt;span class="w"&gt; &lt;/span&gt;parser@&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The output of the rebase lists every ref it updated, so you know which branches to&amp;nbsp;restore.&lt;/p&gt;
&lt;h3 id="what-the-reflog-doesnt-know"&gt;What the Reflog Doesn&amp;#8217;t&amp;nbsp;Know&lt;/h3&gt;
&lt;p&gt;The reflog is local and not unlimited. By default, entries are kept for 90 days. Entries for commits that are no longer reachable from the current tip of that ref, such as the old commits after a rebase, expire after 30 days. That is plenty of time to notice a mistake, but it&amp;#8217;s not a&amp;nbsp;backup.&lt;/p&gt;
&lt;p&gt;And it only knows about commits. &lt;strong&gt;Changes you never committed or stashed are not in the&amp;nbsp;reflog.&lt;/strong&gt; &lt;code&gt;git reset --hard&lt;/code&gt; and &lt;code&gt;git restore&lt;/code&gt; on uncommitted work really do destroy&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="the-three-kinds-of-reset"&gt;The Three Kinds of&amp;nbsp;Reset&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;git reset&lt;/code&gt; moves the current branch to another commit. The flag decides what happens to the index and your&amp;nbsp;files:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;th&gt;Branch moves&lt;/th&gt;
&lt;th&gt;Index&lt;/th&gt;
&lt;th&gt;Working files&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;git reset --soft &amp;lt;c&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;unchanged&lt;/td&gt;
&lt;td&gt;unchanged&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;git reset --mixed &amp;lt;c&amp;gt;&lt;/code&gt; (default)&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;reset&amp;nbsp;to &lt;code&gt;&amp;lt;c&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;unchanged&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;git reset --hard &amp;lt;c&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;yes&lt;/td&gt;
&lt;td&gt;reset&amp;nbsp;to &lt;code&gt;&amp;lt;c&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;reset&amp;nbsp;to &lt;code&gt;&amp;lt;c&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;code&gt;git reset --soft HEAD~3&lt;/code&gt; is a quick way to squash the last three commits into one: the branch moves back, everything stays staged, and you commit&amp;nbsp;again.&lt;/p&gt;
&lt;p&gt;For changes that are already pushed and shared, don&amp;#8217;t reset at all.&amp;nbsp;Use &lt;code&gt;git revert&lt;/code&gt;, which creates a new commit that undoes an old one. History stays intact and nobody else&amp;#8217;s clone&amp;nbsp;breaks.&lt;/p&gt;
&lt;h2 id="cleaning-up-before-you-share"&gt;Cleaning Up Before You&amp;nbsp;Share&lt;/h2&gt;
&lt;p&gt;Nobody writes a feature in a perfect sequence of commits. You implement something, find a bug in a commit three steps back, fix it, rename a function, fix the rename. The history ends up as a diary. Reviewers don&amp;#8217;t need your diary. They need a sequence of changes that each make sense on their&amp;nbsp;own.&lt;/p&gt;
&lt;p&gt;Interactive rebase is the tool for that, and fixup commits make it&amp;nbsp;painless.&lt;/p&gt;
&lt;h3 id="fixup-commits-and-autosquash"&gt;Fixup Commits and&amp;nbsp;Autosquash&lt;/h3&gt;
&lt;p&gt;In &lt;code&gt;inigo&lt;/code&gt; I have two&amp;nbsp;branches. &lt;code&gt;parser&lt;/code&gt; adds the &lt;span class="caps"&gt;INI&lt;/span&gt; parser in two commits,&amp;nbsp;and &lt;code&gt;cli&lt;/code&gt; builds on top of it. While working&amp;nbsp;on &lt;code&gt;cli&lt;/code&gt;, I noticed that the first parser commit&amp;nbsp;splits &lt;code&gt;key = value&lt;/code&gt; on&amp;nbsp;every &lt;code&gt;=&lt;/code&gt; and doesn&amp;#8217;t strip whitespace. Instead of a &amp;#8220;fix parser bug&amp;#8221; commit, I tell Git which commit the fix belongs&amp;nbsp;to:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;-p
git&lt;span class="w"&gt; &lt;/span&gt;commit&lt;span class="w"&gt; &lt;/span&gt;--fixup&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;:/^parser: parse sections&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;:/text&lt;/code&gt; selects the most recent commit whose message matches that regular expression, which saves looking up the &lt;span class="caps"&gt;ID&lt;/span&gt;.&amp;nbsp;The &lt;code&gt;^&lt;/code&gt; anchor matters: without it, a second fixup for the same commit would match the&amp;nbsp;first &lt;code&gt;fixup! parser: parse sections and keys&lt;/code&gt;, because that is the more recent commit containing the text. Git creates a commit&amp;nbsp;named &lt;code&gt;fixup! parser: parse sections and keys&lt;/code&gt;.&amp;nbsp;Meanwhile &lt;code&gt;main&lt;/code&gt; gained a &lt;span class="caps"&gt;CI&lt;/span&gt; commit, so the branch is out of date as&amp;nbsp;well:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;c8be862 (HEAD -&amp;gt; cli) fixup! parser: parse sections and keys
1d549e5 cli: add --config option
984a6c3 (parser) parser: support include files
0417e83 parser: parse sections and keys
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;One command fixes&amp;nbsp;both:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;rebase&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;main
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With &lt;code&gt;rebase.autoSquash&lt;/code&gt; set, the editor opens with the fixup commit already moved right after its target and&amp;nbsp;marked &lt;code&gt;fixup&lt;/code&gt;. Save, close, and the fix is folded into the original commit, on top of the&amp;nbsp;current &lt;code&gt;main&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;c4137b3 (HEAD -&amp;gt; cli) cli: add --config option
9b6891a (parser) parser: support include files
70e4cd2 parser: parse sections and keys
ecf2e5a (main) ci: run tests on FreeBSD
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every commit has a new &lt;span class="caps"&gt;ID&lt;/span&gt;. These are new commits, as described in the mental&amp;nbsp;model.&lt;/p&gt;
&lt;p&gt;There are&amp;nbsp;variations. &lt;code&gt;git commit --fixup=amend:&amp;lt;commit&amp;gt;&lt;/code&gt; also replaces the target&amp;#8217;s commit message,&amp;nbsp;and &lt;code&gt;git commit --squash=&amp;lt;commit&amp;gt;&lt;/code&gt; keeps both messages for you to&amp;nbsp;merge.&lt;/p&gt;
&lt;h3 id="stacked-branches-with-update-refs"&gt;Stacked Branches&amp;nbsp;With &lt;code&gt;--update-refs&lt;/code&gt;&lt;/h3&gt;
&lt;p&gt;Look at that output&amp;nbsp;again: &lt;code&gt;parser&lt;/code&gt; moved too, even though I rebased while&amp;nbsp;on &lt;code&gt;cli&lt;/code&gt;.&amp;nbsp;Before &lt;code&gt;--update-refs&lt;/code&gt;,&amp;nbsp;rebasing &lt;code&gt;cli&lt;/code&gt; would have&amp;nbsp;left &lt;code&gt;parser&lt;/code&gt; pointing at the old commits, and you had to fix up the labels by hand.&amp;nbsp;With &lt;code&gt;rebase.updateRefs&lt;/code&gt; set, Git moves every local branch that points into the rebased range. I then force-push both branches, and both pull requests are up to&amp;nbsp;date.&lt;/p&gt;
&lt;p&gt;Two side effects are worth knowing, and I tested both. First, &amp;#8220;every local branch&amp;#8221; includes&amp;nbsp;a &lt;code&gt;backup&lt;/code&gt; branch you created just before the rebase as a safety net. It moves along with the others, so it no longer points at the old commits. The reflog is the better safety net anyway. Second, a branch that is checked out in another worktree is not moved. The todo list shows it as a&amp;nbsp;comment:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Ref refs/heads/parser checked out at &amp;#39;/home/me/src/inigo-parser&amp;#39;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is reasonable, because moving a branch under another worktree would leave that worktree in a confusing state. But it means you rebase that branch&amp;nbsp;separately.&lt;/p&gt;
&lt;h3 id="comparing-two-versions-of-a-branch"&gt;Comparing Two Versions of a&amp;nbsp;Branch&lt;/h3&gt;
&lt;p&gt;After a rebase, a reviewer will want to know what actually changed since the last&amp;nbsp;version. &lt;code&gt;git diff&lt;/code&gt; between the old and new branch tip is useless, because it includes everything that changed&amp;nbsp;on &lt;code&gt;main&lt;/code&gt; in&amp;nbsp;between. &lt;code&gt;git range-diff&lt;/code&gt; compares the two series commit by&amp;nbsp;commit:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;range-diff&lt;span class="w"&gt; &lt;/span&gt;main&lt;span class="w"&gt; &lt;/span&gt;cli@&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;cli
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;nbsp;compares &lt;code&gt;main..cli@{1}&lt;/code&gt;, the old version from the branch reflog,&amp;nbsp;with &lt;code&gt;main..cli&lt;/code&gt;, the new one. For the rebase&amp;nbsp;above:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;1:  0417e83 ! 1:  70e4cd2 parser: parse sections and keys
    @@ parser.py
     +            section = line[1:-1]
     +            result[section] = {}
     +        elif &amp;quot;=&amp;quot; in line:
    -+            key, value = line.split(&amp;quot;=&amp;quot;)
    ++            key, value = (s.strip() for s in line.split(&amp;quot;=&amp;quot;, 1))
     +            result[section][key] = value
          return result
2:  984a6c3 ! 2:  9b6891a parser: support include files
    @@ Commit message

      ## parser.py ##
     @@ parser.py: def parse(text):
    -             key, value = line.split(&amp;quot;=&amp;quot;)
    +             key, value = (s.strip() for s in line.split(&amp;quot;=&amp;quot;, 1))
                  result[section][key] = value
          return result
     +
3:  1d549e5 = 3:  c4137b3 cli: add --config option
4:  c8be862 &amp;lt; -:  ------- fixup! parser: parse sections and keys
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It takes a moment to read, because it is a diff of diffs. The first column&amp;nbsp;of &lt;code&gt;-&lt;/code&gt; and &lt;code&gt;+&lt;/code&gt; says what changed between the two versions of the patch. The second column is the patch&amp;nbsp;itself. &lt;code&gt;!&lt;/code&gt; marks a modified commit: the first one now contains the fix. The second commit&amp;#8217;s own change is the same, but its context lines changed, so it shows up as modified&amp;nbsp;too. &lt;code&gt;=&lt;/code&gt; means unchanged apart from its parent,&amp;nbsp;and &lt;code&gt;&amp;lt;&lt;/code&gt; marks a commit that only exists in the old series, here the fixup that was folded&amp;nbsp;in.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;CI&lt;/span&gt; commit&amp;nbsp;from &lt;code&gt;main&lt;/code&gt; doesn&amp;#8217;t appear at all, which is the point. I&amp;nbsp;run &lt;code&gt;range-diff&lt;/code&gt; before every force-push to make sure the rebase did what I wanted, and nothing&amp;nbsp;else.&lt;/p&gt;
&lt;h2 id="cherry-picking"&gt;Cherry-Picking&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;git cherry-pick&lt;/code&gt; takes the change introduced by one commit and applies it as a new commit on the current branch. That is the entire feature, and it&amp;#8217;s more useful than it&amp;nbsp;sounds.&lt;/p&gt;
&lt;p&gt;The classic case is a&amp;nbsp;backport. &lt;code&gt;inigo&lt;/code&gt; has&amp;nbsp;a &lt;code&gt;release-1.0&lt;/code&gt; branch.&amp;nbsp;On &lt;code&gt;main&lt;/code&gt;, I fixed the parser to skip blank lines and comments, and that fix also belongs in the release branch, but the documentation work&amp;nbsp;on &lt;code&gt;main&lt;/code&gt; doesn&amp;#8217;t. First, which commits are&amp;nbsp;on &lt;code&gt;main&lt;/code&gt; but not on the release&amp;nbsp;branch?&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ git cherry -v release-1.0 main
+ 75f6c7c6c6418f610cf645165ecf4f819d21cef6 parser: skip blank lines and comments
+ 9e8db67b8e3b86546481f3c9d8ee42e9402e806b docs: describe include files
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then pick the&amp;nbsp;fix:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;switch&lt;span class="w"&gt; &lt;/span&gt;release-1.0
git&lt;span class="w"&gt; &lt;/span&gt;cherry-pick&lt;span class="w"&gt; &lt;/span&gt;-x&lt;span class="w"&gt; &lt;/span&gt;75f6c7c
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Always&amp;nbsp;use &lt;code&gt;-x&lt;/code&gt; for backports.&lt;/strong&gt; It appends a line to the commit message that records where the change came&amp;nbsp;from:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;parser: skip blank lines and comments

(cherry picked from commit 75f6c7c6c6418f610cf645165ecf4f819d21cef6)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Six months later, when somebody asks&amp;nbsp;whether &lt;code&gt;release-1.0&lt;/code&gt; has the fix for that &lt;span class="caps"&gt;CVE&lt;/span&gt;,&amp;nbsp;a &lt;code&gt;git log --grep&lt;/code&gt; answers the&amp;nbsp;question.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;git cherry&lt;/code&gt; compares patches, not commit IDs, so after the pick it recognizes the fix as already present, even though the commit&amp;nbsp;on &lt;code&gt;release-1.0&lt;/code&gt; has a different &lt;span class="caps"&gt;ID&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ git cherry -v release-1.0 main
- 75f6c7c6c6418f610cf645165ecf4f819d21cef6 parser: skip blank lines and comments
+ 9e8db67b8e3b86546481f3c9d8ee42e9402e806b docs: describe include files
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="ranges-and-conflicts"&gt;Ranges and&amp;nbsp;Conflicts&lt;/h3&gt;
&lt;p&gt;You can pick several commits at&amp;nbsp;once:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;cherry-pick&lt;span class="w"&gt; &lt;/span&gt;A..B&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="c1"&gt;# everything after A, up to and including B&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;cherry-pick&lt;span class="w"&gt; &lt;/span&gt;A^..B&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# including A itself&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The first form&amp;nbsp;excludes &lt;code&gt;A&lt;/code&gt;, which trips people up every time. When a pick conflicts, Git stops in the middle of the sequence, exactly like a rebase. Resolve the&amp;nbsp;conflict, &lt;code&gt;git add&lt;/code&gt; the files, and&amp;nbsp;then:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;cherry-pick&lt;span class="w"&gt; &lt;/span&gt;--continue&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# carry on with the next commit&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;cherry-pick&lt;span class="w"&gt; &lt;/span&gt;--skip&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="c1"&gt;# drop this commit and carry on&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;cherry-pick&lt;span class="w"&gt; &lt;/span&gt;--abort&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="c1"&gt;# go back to where you started&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="when-not-to-cherry-pick"&gt;When Not to&amp;nbsp;Cherry-Pick&lt;/h3&gt;
&lt;p&gt;A cherry-pick creates a new commit with the same change and a different &lt;span class="caps"&gt;ID&lt;/span&gt;. If both branches are later merged, Git usually copes, but you end up with the same change twice in history, and conflicts around it get more&amp;nbsp;confusing.&lt;/p&gt;
&lt;p&gt;So cherry-picking is a tool for branches that will never be merged into each other: release branches, stable branches, a downstream fork. For two feature branches that will both end up&amp;nbsp;in &lt;code&gt;main&lt;/code&gt;, the better approach is to put the shared change into its own branch, merge it first, and rebase the others on top. Where a project has long-lived maintenance branches, some prefer to commit the fix to the oldest branch and merge it forward instead. Both work. What doesn&amp;#8217;t work well is cherry-picking back and forth between branches that later&amp;nbsp;meet.&lt;/p&gt;
&lt;p&gt;Cherry-picking a merge commit&amp;nbsp;needs &lt;code&gt;-m 1&lt;/code&gt; to tell Git which parent to diff against. If you find yourself doing that regularly, it is probably a sign the branching model is fighting&amp;nbsp;you.&lt;/p&gt;
&lt;h2 id="worktrees-several-branches-at-once"&gt;Worktrees: Several Branches at&amp;nbsp;Once&lt;/h2&gt;
&lt;p&gt;This is the feature I wish somebody had shown me ten years&amp;nbsp;earlier.&lt;/p&gt;
&lt;p&gt;The usual situation: you are in the middle of a feature, with half-finished changes and a build directory full of artifacts. A colleague asks you to review a pull request, or production needs a hotfix. Your options without worktrees are all bad. You stash your changes, switch branch, rebuild everything, do the other thing, switch back, rebuild again, pop the stash and hope it applies cleanly. Or you clone the repository a second time and keep two copies in sync by&amp;nbsp;hand.&lt;/p&gt;
&lt;p&gt;A worktree is an additional working directory attached to the same repository. Each worktree has its own checked-out branch, its own index and its own files. They all share the same object database, refs and configuration. Creating one takes a second and no network access.&amp;nbsp;Say &lt;code&gt;release-1.0&lt;/code&gt; of &lt;code&gt;inigo&lt;/code&gt; needs a hotfix while I&amp;#8217;m working&amp;nbsp;on &lt;code&gt;main&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;worktree&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;../inigo-hotfix&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;hotfix&lt;span class="w"&gt; &lt;/span&gt;release-1.0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That creates the&amp;nbsp;directory &lt;code&gt;../inigo-hotfix&lt;/code&gt;, a new&amp;nbsp;branch &lt;code&gt;hotfix&lt;/code&gt; starting&amp;nbsp;at &lt;code&gt;release-1.0&lt;/code&gt;, and checks it out there. My original directory is untouched, including the uncommitted changes and the build&amp;nbsp;artifacts. &lt;code&gt;cd&lt;/code&gt; over, fix, commit, push, and go&amp;nbsp;back.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ git worktree list
/home/me/src/inigo         9409eec [main]
/home/me/src/inigo-hotfix  d4bd468 [hotfix]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;When you are&amp;nbsp;done:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;worktree&lt;span class="w"&gt; &lt;/span&gt;remove&lt;span class="w"&gt; &lt;/span&gt;../inigo-hotfix
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;remove&lt;/code&gt; refuses if the worktree has uncommitted changes, which is the right default. If you deleted a worktree directory&amp;nbsp;with &lt;code&gt;rm -rf&lt;/code&gt;, &lt;code&gt;git worktree prune&lt;/code&gt; cleans up the leftover administrative&amp;nbsp;data.&lt;/p&gt;
&lt;h3 id="reviewing-pull-requests-in-a-worktree"&gt;Reviewing Pull Requests in a&amp;nbsp;Worktree&lt;/h3&gt;
&lt;p&gt;Forgejo, Gitea and GitHub all expose pull requests as refs&amp;nbsp;under &lt;code&gt;refs/pull/&amp;lt;number&amp;gt;/head&lt;/code&gt;. That makes a review worktree a&amp;nbsp;two-liner:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;fetch&lt;span class="w"&gt; &lt;/span&gt;origin&lt;span class="w"&gt; &lt;/span&gt;pull/104/head:pr-104
git&lt;span class="w"&gt; &lt;/span&gt;worktree&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;../inigo-pr-104&lt;span class="w"&gt; &lt;/span&gt;pr-104
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now I can build and run the &lt;span class="caps"&gt;PR&lt;/span&gt; next to my own work, without touching it. If I only want to look, I skip the local branch and use a detached worktree. That needs a plain fetch first, which leaves the &lt;span class="caps"&gt;PR&lt;/span&gt; head&amp;nbsp;in &lt;code&gt;FETCH_HEAD&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;fetch&lt;span class="w"&gt; &lt;/span&gt;origin&lt;span class="w"&gt; &lt;/span&gt;pull/104/head
git&lt;span class="w"&gt; &lt;/span&gt;worktree&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;--detach&lt;span class="w"&gt; &lt;/span&gt;../inigo-pr-104&lt;span class="w"&gt; &lt;/span&gt;FETCH_HEAD
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="what-is-shared-and-what-isnt"&gt;What Is Shared and What&amp;nbsp;Isn&amp;#8217;t&lt;/h3&gt;
&lt;p&gt;This table is the part most worktree introductions skip, and it explains every surprise I have had with&amp;nbsp;them:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Shared between worktrees&lt;/th&gt;
&lt;th&gt;Separate per worktree&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Commits and all other objects&lt;/td&gt;
&lt;td&gt;&lt;code&gt;HEAD&lt;/code&gt;, the checked-out branch&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Branches, tags, remote refs&lt;/td&gt;
&lt;td&gt;The index (staging area)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Repository configuration&amp;nbsp;(&lt;code&gt;.git/config&lt;/code&gt;)&lt;/td&gt;
&lt;td&gt;The working files&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Stashes&lt;/td&gt;
&lt;td&gt;Untracked and ignored files, build output&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hooks&lt;/td&gt;
&lt;td&gt;Bisect and rebase state&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The most visible consequence: &lt;strong&gt;a branch can only be checked out in one worktree at a time.&lt;/strong&gt; Git refuses otherwise, and tells you&amp;nbsp;where:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;fatal: &amp;#39;hotfix&amp;#39; is already used by worktree at &amp;#39;/home/me/src/inigo-hotfix&amp;#39;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is a feature. Two working directories committing to the same branch would be chaos. It is also&amp;nbsp;why &lt;code&gt;--update-refs&lt;/code&gt; leaves such a branch alone. The other&amp;nbsp;consequences:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;A commit in one worktree is immediately visible in all others. No fetch, no push. You can cherry-pick from the hotfix worktree into the main worktree right after&amp;nbsp;committing.&lt;/li&gt;
&lt;li&gt;The stash is shared.&amp;nbsp;A &lt;code&gt;git stash&lt;/code&gt; in one worktree shows up&amp;nbsp;in &lt;code&gt;git stash list&lt;/code&gt; in all of them. That surprised me the first time. With worktrees I rarely stash at all any more, which is half the&amp;nbsp;point.&lt;/li&gt;
&lt;li&gt;Build artifacts and dependencies are not shared. Each worktree needs its&amp;nbsp;own &lt;code&gt;node_modules&lt;/code&gt;, &lt;code&gt;target/&lt;/code&gt; or virtualenv. That costs disk space and an initial build, but it is also why switching contexts no longer invalidates your&amp;nbsp;build.&lt;/li&gt;
&lt;li&gt;Configuration is shared by default. If one worktree needs different settings,&amp;nbsp;enable &lt;code&gt;extensions.worktreeConfig&lt;/code&gt; and&amp;nbsp;use &lt;code&gt;git config --worktree&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="how-i-lay-them-out"&gt;How I Lay Them&amp;nbsp;Out&lt;/h3&gt;
&lt;p&gt;I keep worktrees as siblings of the main checkout, named after the repository and the&amp;nbsp;branch: &lt;code&gt;inigo&lt;/code&gt;, &lt;code&gt;inigo-hotfix&lt;/code&gt;, &lt;code&gt;inigo-pr-104&lt;/code&gt;. Some people prefer a bare repository with every branch as a worktree inside it. That works too, but sibling directories keep the main checkout a normal repository that every tool&amp;nbsp;understands.&lt;/p&gt;
&lt;p&gt;For a worktree on a removable disk or a network share, lock it, so&amp;nbsp;that &lt;code&gt;prune&lt;/code&gt; doesn&amp;#8217;t clean it up while the disk is not&amp;nbsp;mounted:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;worktree&lt;span class="w"&gt; &lt;/span&gt;lock&lt;span class="w"&gt; &lt;/span&gt;--reason&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;on usb disk&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;../inigo-usb
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="finding-things-in-history"&gt;Finding Things in&amp;nbsp;History&lt;/h2&gt;
&lt;p&gt;Git&amp;#8217;s history is a database, and the query tools are better than most people&amp;nbsp;know.&lt;/p&gt;
&lt;p&gt;Who changed this line, and&amp;nbsp;why? &lt;code&gt;git blame&lt;/code&gt; is the obvious answer, but the useful flags are less&amp;nbsp;known. &lt;code&gt;-w&lt;/code&gt; ignores whitespace changes,&amp;nbsp;and &lt;code&gt;-C&lt;/code&gt; follows lines that were moved or copied from other files. For mass reformatting commits that pollute every blame, put their IDs in a file and tell Git to skip&amp;nbsp;them:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;blame.ignoreRevsFile&lt;span class="w"&gt; &lt;/span&gt;.git-blame-ignore-revs
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;When did this string appear or disappear? The pickaxe search finds commits that changed the number of occurrences of a&amp;nbsp;string:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;split(&amp;quot;=&amp;quot;, 1)&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--oneline
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;How did this function&amp;nbsp;evolve? &lt;code&gt;git log -L&lt;/code&gt; shows the history of a line range or a&amp;nbsp;function:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;-L&lt;span class="w"&gt; &lt;/span&gt;:parse:parser.py
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Which commit broke&amp;nbsp;it? &lt;code&gt;git bisect&lt;/code&gt; does a binary search through history. Some time after the&amp;nbsp;backport, &lt;code&gt;inigo&lt;/code&gt; stopped handling values that contain&amp;nbsp;an &lt;code&gt;=&lt;/code&gt;,&amp;nbsp;like &lt;code&gt;url = a=b&lt;/code&gt;. A test script that exits 0 for good and non-zero for bad lets bisect run&amp;nbsp;unattended:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="c1"&gt;# test-equals.sh&lt;/span&gt;
python3&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;import parser; assert parser.parse(&amp;quot;[s]\nurl = a=b&amp;quot;)[&amp;quot;s&amp;quot;][&amp;quot;url&amp;quot;] == &amp;quot;a=b&amp;quot;&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;bisect&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;main&lt;span class="w"&gt; &lt;/span&gt;release-1.0&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# bad, then good&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;bisect&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;../test-equals.sh
git&lt;span class="w"&gt; &lt;/span&gt;bisect&lt;span class="w"&gt; &lt;/span&gt;reset
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Bisect checks out commits, runs the script and narrows the range. The interesting part of the&amp;nbsp;output:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Bisecting: 0 revisions left to test after this (roughly 0 steps)
[9368001702033b723bdf2466d1d38f6afa4a2cee] parser: tidy up
running &amp;#39;../test-equals.sh&amp;#39;
9368001702033b723bdf2466d1d38f6afa4a2cee is the first &amp;#39;bad&amp;#39; commit
commit 9368001702033b723bdf2466d1d38f6afa4a2cee
[author and date lines]

    parser: tidy up

 parser.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
bisect found first &amp;#39;bad&amp;#39; commit
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Tidy up&amp;#8221; quietly went back to splitting on&amp;nbsp;every &lt;code&gt;=&lt;/code&gt;. Commits with honest, specific messages make this output much more&amp;nbsp;useful.&lt;/p&gt;
&lt;p&gt;The exit code convention has one more value worth knowing. Exit code 125 tells bisect to skip the current commit, which is what&amp;nbsp;makes &lt;code&gt;bisect run&lt;/code&gt; usable when some commits in the range don&amp;#8217;t build: have the script return 125 when the build fails. Codes from 1 to 127, apart from 125, mean bad, and anything above 127 aborts the&amp;nbsp;run.&lt;/p&gt;
&lt;h2 id="signing-commits"&gt;Signing&amp;nbsp;Commits&lt;/h2&gt;
&lt;p&gt;On my Forgejo instance, the branch protection&amp;nbsp;for &lt;code&gt;main&lt;/code&gt; and my article branches rejects unsigned commits. That is a deliberate choice, and to me signing is the most underexplained topic in&amp;nbsp;Git.&lt;/p&gt;
&lt;h3 id="what-a-signature-proves-and-what-it-doesnt"&gt;What a Signature Proves, and What It&amp;nbsp;Doesn&amp;#8217;t&lt;/h3&gt;
&lt;p&gt;The author and committer fields of a commit are just text. Anybody can&amp;nbsp;run &lt;code&gt;git config user.name "Linus Torvalds"&lt;/code&gt; and commit as him. A forge that shows the author&amp;#8217;s avatar next to that commit is not lying, exactly, but it is showing you something nobody&amp;nbsp;checked.&lt;/p&gt;
&lt;p&gt;A signature proves that the commit was created by someone holding a particular private key. Since the commit &lt;span class="caps"&gt;ID&lt;/span&gt; covers the whole content and the complete history, a signature on a commit also protects everything that commit points&amp;nbsp;to.&lt;/p&gt;
&lt;p&gt;What it does not&amp;nbsp;prove:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;That the code is correct or&amp;nbsp;safe.&lt;/li&gt;
&lt;li&gt;That the key belongs to the person you think it does. That is a separate question, answered by how you got the public&amp;nbsp;key.&lt;/li&gt;
&lt;li&gt;That the author field is accurate. The signature covers the author field, but nothing stops a key holder from writing someone else&amp;#8217;s name into&amp;nbsp;it.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;So signing is about provenance. Combined with a forge that refuses unsigned pushes, it means a leaked access token or &lt;span class="caps"&gt;SSH&lt;/span&gt; push key is not enough to push code to my repositories. The attacker would also need my signing key, and in my case that key lives on a hardware token that requires a &lt;span class="caps"&gt;PIN&lt;/span&gt; and a&amp;nbsp;touch.&lt;/p&gt;
&lt;p&gt;A stolen account password is a different story. Somebody who can log in to the forge can register their own signing key with my email address, and their commits then show as verified. Signing only helps against that if the account itself is protected with a second&amp;nbsp;factor.&lt;/p&gt;
&lt;h3 id="three-signature-formats"&gt;Three Signature&amp;nbsp;Formats&lt;/h3&gt;
&lt;p&gt;Git supports three formats, selected&amp;nbsp;with &lt;code&gt;gpg.format&lt;/code&gt;:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;code&gt;gpg.format&lt;/code&gt;&lt;/th&gt;
&lt;th&gt;Program&lt;/th&gt;
&lt;th&gt;Key&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;openpgp&lt;/code&gt; (default)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gpg&lt;/code&gt;, or a compatible replacement such&amp;nbsp;as &lt;code&gt;oct-git&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;OpenPGP key, possibly on a smartcard&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ssh&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ssh-keygen&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Any &lt;span class="caps"&gt;SSH&lt;/span&gt; key, including &lt;span class="caps"&gt;FIDO2&lt;/span&gt; keys&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;x509&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gpgsm&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;X.509 certificate, for corporate &lt;span class="caps"&gt;PKI&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;If you have no signing setup at all yet, &lt;span class="caps"&gt;SSH&lt;/span&gt; signing is the easiest start, because you probably already have an &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;key:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;gpg.format&lt;span class="w"&gt; &lt;/span&gt;ssh
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;user.signingkey&lt;span class="w"&gt; &lt;/span&gt;~/.ssh/id_ed25519.pub
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For verification, &lt;span class="caps"&gt;SSH&lt;/span&gt; has no keyring or web of trust, so Git needs a file that maps identities to&amp;nbsp;keys:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;~/.config/git
&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;me@example.org &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;~/.ssh/id_ed25519.pub&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;~/.config/git/allowed_signers
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;gpg.ssh.allowedSignersFile&lt;span class="w"&gt; &lt;/span&gt;~/.config/git/allowed_signers
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ git commit -S -m &amp;quot;signed&amp;quot;
$ git verify-commit HEAD
Good &amp;quot;git&amp;quot; signature for me@example.org with ED25519 key SHA256:GPiOP4A1Oi7Chq0pqtPVhZSycN7gbFBAJeD/J7PoiAY
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="openpgp-on-a-smartcard-with-oct-git"&gt;OpenPGP on a Smartcard With&amp;nbsp;oct-git&lt;/h3&gt;
&lt;p&gt;I sign with OpenPGP, on a hardware token. But not with&amp;nbsp;GnuPG.&lt;/p&gt;
&lt;p&gt;Git doesn&amp;#8217;t actually need GnuPG. It calls whatever&amp;nbsp;program &lt;code&gt;gpg.program&lt;/code&gt; points to, passes the data on standard input and expects a signature in GnuPG&amp;#8217;s output format. &lt;a href="https://codeberg.org/openpgp-card/oct-git"&gt;oct-git&lt;/a&gt; from the OpenPGP Card project implements exactly that subset and nothing else, and talks directly to the OpenPGP card.&amp;nbsp;No &lt;code&gt;gpg-agent&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;scdaemon&lt;/code&gt;, no keyring on&amp;nbsp;disk.&lt;/p&gt;
&lt;p&gt;I maintain the packages for it, together with the other two tools of the&amp;nbsp;project, &lt;code&gt;oct&lt;/code&gt; and &lt;code&gt;openpgp-card-ssh-agent&lt;/code&gt;. On FreeBSD it is in the ports tree as &lt;a href="https://www.freshports.org/security/openpgp-card-tools-git/"&gt;&lt;code&gt;security/openpgp-card-tools-git&lt;/code&gt;&lt;/a&gt;. For Fedora 44, 45 and Rawhide, I build them in my &lt;span class="caps"&gt;COPR&lt;/span&gt; repository &lt;a href="https://copr.fedorainfracloud.org/coprs/larvitz/openpgp-card/"&gt;larvitz/openpgp-card&lt;/a&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Fedora&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;copr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;larvitz/openpgp-card
sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools-git

&lt;span class="c1"&gt;# FreeBSD&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-tools-git
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The FreeBSD ports are new, so depending on your package&amp;nbsp;branch &lt;code&gt;pkg&lt;/code&gt; may not offer them yet. In that case, build them from the ports&amp;nbsp;tree.&lt;/p&gt;
&lt;p&gt;Then point Git at&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;gpg.program&lt;span class="w"&gt; &lt;/span&gt;oct-git
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;user.signingkey&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;fingerprint&amp;gt;
oct-git&lt;span class="w"&gt; &lt;/span&gt;--store-card-pin
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The fingerprint is the signing key on the card, as shown&amp;nbsp;by &lt;code&gt;oct status&lt;/code&gt;, in hex without spaces. My &lt;a href="https://hofstede.it/hardware-keys-without-gnupg-the-rust-openpgp-card-stack-now-in-freebsd-ports/"&gt;article on the Rust OpenPGP card stack&lt;/a&gt; covers the card setup, &lt;span class="caps"&gt;PIN&lt;/span&gt; storage, touch confirmation and the FreeBSD pinentry wrapper in detail, so I won&amp;#8217;t repeat that here. Two Git-specific traps are worth knowing,&amp;nbsp;though.&lt;/p&gt;
&lt;p&gt;Set &lt;code&gt;user.signingkey&lt;/code&gt; to a fingerprint. If it is unset, Git passes your committer name and email to the signing program as the key to use. GnuPG looks that up in its&amp;nbsp;keyring. &lt;code&gt;oct-git&lt;/code&gt; expects a fingerprint and fails with an unhelpful parse&amp;nbsp;error:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Odd number of digits
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That cost me some head-scratching the first time. Setting the key explicitly, globally or per repository, fixes&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;GnuPG&amp;#8217;s &lt;code&gt;scdaemon&lt;/code&gt; can hold the card exclusively. If GnuPG is also installed and something triggered&amp;nbsp;it, &lt;code&gt;oct-git&lt;/code&gt; may report that no card was found while the token is plugged in and blinking&amp;nbsp;happily. &lt;code&gt;gpgconf --kill scdaemon&lt;/code&gt; releases it. The card stack article has the permanent&amp;nbsp;fix.&lt;/p&gt;
&lt;h3 id="rewriting-history-drops-signatures"&gt;Rewriting History Drops&amp;nbsp;Signatures&lt;/h3&gt;
&lt;p&gt;This is the part that bites people who sign only now and then, which includes me. I&amp;nbsp;keep &lt;code&gt;commit.gpgsign&lt;/code&gt; off and sign explicitly&amp;nbsp;with &lt;code&gt;git commit -S&lt;/code&gt;, because not every scratch repository needs a &lt;span class="caps"&gt;PIN&lt;/span&gt;&amp;nbsp;prompt.&lt;/p&gt;
&lt;p&gt;But a rebase, an amend or a cherry-pick creates new commits, and &lt;strong&gt;new commits are only signed if Git is told to sign them.&lt;/strong&gt; I tested&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;orig: G
after plain rebase: N
after rebase -S: G
cherry-pick: N
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;G&lt;/code&gt; means a good&amp;nbsp;signature, &lt;code&gt;N&lt;/code&gt; means no signature. The original commit was signed. A&amp;nbsp;plain &lt;code&gt;git rebase&lt;/code&gt; produced an unsigned copy, and my forge would have rejected the push. The same is true&amp;nbsp;for &lt;code&gt;cherry-pick&lt;/code&gt;. The fix is to&amp;nbsp;pass &lt;code&gt;-S&lt;/code&gt; to those commands as well&amp;nbsp;(&lt;code&gt;git rebase -S&lt;/code&gt;, &lt;code&gt;git cherry-pick -S&lt;/code&gt;, &lt;code&gt;git commit --amend -S&lt;/code&gt;), or to enable signing per repository where it&amp;nbsp;matters:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;commit.gpgsign&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;tag.gpgsign&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With a smartcard and touch confirmation, a rebase of ten commits means ten touches. That&amp;#8217;s a reasonable price, and it is a nice reminder that you are rewriting ten&amp;nbsp;commits.&lt;/p&gt;
&lt;h3 id="checking-signatures"&gt;Checking&amp;nbsp;Signatures&lt;/h3&gt;
&lt;p&gt;To see signatures in the&amp;nbsp;log:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;--show-signature&lt;span class="w"&gt; &lt;/span&gt;-1
git&lt;span class="w"&gt; &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%h %G? %GS %s&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;%G?&lt;/code&gt; placeholder is the useful one for&amp;nbsp;scripts. &lt;code&gt;G&lt;/code&gt; is&amp;nbsp;good, &lt;code&gt;N&lt;/code&gt; is&amp;nbsp;unsigned, &lt;code&gt;B&lt;/code&gt; is&amp;nbsp;bad, &lt;code&gt;E&lt;/code&gt; means the signature could not be checked (often a missing public key),&amp;nbsp;and &lt;code&gt;U&lt;/code&gt; is a good signature from a key with unknown&amp;nbsp;trust. &lt;code&gt;git verify-commit&lt;/code&gt; and &lt;code&gt;git verify-tag&lt;/code&gt; exit non-zero if verification fails, so they work in &lt;span class="caps"&gt;CI&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;Sign tags too. A signed release tag is how users can check&amp;nbsp;that &lt;code&gt;v1.0&lt;/code&gt; is really the commit you&amp;nbsp;released:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;tag&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;v1.0&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Release 1.0&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;release-1.0
git&lt;span class="w"&gt; &lt;/span&gt;verify-tag&lt;span class="w"&gt; &lt;/span&gt;v1.0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On the forge side, upload your public key to your account. Forgejo, Gitea, GitLab and GitHub then show a verified badge on signed commits, as long as the committer email matches an address on the account. Forgejo&amp;#8217;s branch protection has a &amp;#8220;require signed commits&amp;#8221; option, which is what rejects my unsigned&amp;nbsp;pushes.&lt;/p&gt;
&lt;h2 id="lazygit-the-same-ideas-fewer-keystrokes"&gt;lazygit: The Same Ideas, Fewer&amp;nbsp;Keystrokes&lt;/h2&gt;
&lt;p&gt;Everything above works on the command line, and I think it is worth knowing it there first. But for the more complex operations, I regularly reach for &lt;a href="https://github.com/jesseduffield/lazygit"&gt;lazygit&lt;/a&gt;, a terminal &lt;span class="caps"&gt;UI&lt;/span&gt; for Git written in Go. It doesn&amp;#8217;t replace understanding Git. It makes the operations from this article fast enough that I actually use them, instead of telling myself I&amp;#8217;ll clean up the history&amp;nbsp;later.&lt;/p&gt;
&lt;p&gt;The interesting part is how directly lazygit maps to the concepts&amp;nbsp;above:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concept from this article&lt;/th&gt;
&lt;th&gt;In lazygit&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;git add -p&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Open a file, stage single lines or hunks&amp;nbsp;with &lt;code&gt;&amp;lt;space&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fixup commits and autosquash&lt;/td&gt;
&lt;td&gt;&lt;code&gt;F&lt;/code&gt; on a commit creates&amp;nbsp;a &lt;code&gt;fixup!&lt;/code&gt; commit for&amp;nbsp;it, &lt;code&gt;S&lt;/code&gt; applies them&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Finding the commit to fix up&lt;/td&gt;
&lt;td&gt;&lt;code&gt;&amp;lt;ctrl+f&amp;gt;&lt;/code&gt; in the files view finds the commit your current changes build on&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Interactive rebase&lt;/td&gt;
&lt;td&gt;&lt;code&gt;i&lt;/code&gt; starts one,&amp;nbsp;then &lt;code&gt;s&lt;/code&gt; squash, &lt;code&gt;f&lt;/code&gt; fixup, &lt;code&gt;d&lt;/code&gt; drop, &lt;code&gt;e&lt;/code&gt; edit, directly on the commit list&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cherry-pick&lt;/td&gt;
&lt;td&gt;&lt;code&gt;C&lt;/code&gt; copies&amp;nbsp;commits, &lt;code&gt;V&lt;/code&gt; pastes them onto the checked-out branch&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Worktrees&lt;/td&gt;
&lt;td&gt;&lt;code&gt;w&lt;/code&gt; on a branch creates a worktree for it and switches to it&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reflog and undo&lt;/td&gt;
&lt;td&gt;&lt;code&gt;z&lt;/code&gt; undoes the last Git operation by looking at the reflog&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Bisect&lt;/td&gt;
&lt;td&gt;&lt;code&gt;b&lt;/code&gt; on a commit opens the bisect options&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The keys are from lazygit&amp;#8217;s own keybinding documentation.&amp;nbsp;Press &lt;code&gt;?&lt;/code&gt; in any panel to see what is available&amp;nbsp;there.&lt;/p&gt;
&lt;p&gt;Two features go beyond what I would comfortably do by hand. Custom patches let you pick individual hunks out of an old commit and move them into a new commit, into the index or out of the commit entirely. On the command line, that is an interactive rebase&amp;nbsp;with &lt;code&gt;edit&lt;/code&gt;, a partial&amp;nbsp;reset, &lt;code&gt;add -p&lt;/code&gt; and a&amp;nbsp;careful &lt;code&gt;rebase --continue&lt;/code&gt;. Amending an older commit&amp;nbsp;with &lt;code&gt;A&lt;/code&gt; is a rebase in the background, too, and lazygit handles it for&amp;nbsp;you.&lt;/p&gt;
&lt;p&gt;The undo comes with the same caveat as the reflog. Lazygit&amp;#8217;s documentation says it plainly: undo only considers commits, not changes in your working tree. Discarded uncommitted changes are gone in lazygit just as they are&amp;nbsp;with &lt;code&gt;git restore&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Lazygit drives the&amp;nbsp;normal &lt;code&gt;git&lt;/code&gt; binary, so your configuration applies,&amp;nbsp;including &lt;code&gt;gpg.program&lt;/code&gt;. That also means the signing caveat from above applies: commits that lazygit rewrites during a rebase are only signed if signing is enabled for the repository, for example&amp;nbsp;with &lt;code&gt;commit.gpgsign&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;On FreeBSD it is&amp;nbsp;a &lt;code&gt;pkg install lazygit&lt;/code&gt; away. For Fedora, the project&amp;#8217;s &lt;span class="caps"&gt;README&lt;/span&gt; points to the &lt;a href="https://copr.fedorainfracloud.org/coprs/dejan/lazygit/"&gt;dejan/lazygit&lt;/a&gt; &lt;span class="caps"&gt;COPR&lt;/span&gt;&amp;nbsp;repository:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Fedora&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;copr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;dejan/lazygit
sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;lazygit

&lt;span class="c1"&gt;# FreeBSD&lt;/span&gt;
pkg&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;lazygit
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then&amp;nbsp;run &lt;code&gt;lazygit&lt;/code&gt; inside any&amp;nbsp;repository.&lt;/p&gt;
&lt;h2 id="housekeeping"&gt;Housekeeping&lt;/h2&gt;
&lt;p&gt;Large repositories slow down over time as loose objects pile up. Git cleans up opportunistically, but you can also hand that job to a&amp;nbsp;scheduler:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;maintenance&lt;span class="w"&gt; &lt;/span&gt;start
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This registers the repository and schedules background tasks, such as prefetching from remotes and repacking objects. Git picks the platform&amp;#8217;s scheduler: systemd user timers on Linux where available, cron on the BSDs and on Linux without systemd, launchd on macOS and the task scheduler on&amp;nbsp;Windows. &lt;code&gt;git maintenance stop&lt;/code&gt; undoes it. For the occasional manual&amp;nbsp;cleanup, &lt;code&gt;git gc&lt;/code&gt; is still&amp;nbsp;there.&lt;/p&gt;
&lt;h2 id="what-this-article-doesnt-cover"&gt;What This Article Doesn&amp;#8217;t&amp;nbsp;Cover&lt;/h2&gt;
&lt;p&gt;Git is big, and I deliberately left out several&amp;nbsp;things:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Submodules and subtrees. Both solve &amp;#8220;repository inside a repository&amp;#8221;, both come with sharp edges, and both deserve their own&amp;nbsp;article.&lt;/li&gt;
&lt;li&gt;Large files. Git &lt;span class="caps"&gt;LFS&lt;/span&gt;, partial clones and sparse checkouts matter for monorepos and game assets. I have&amp;nbsp;neither.&lt;/li&gt;
&lt;li&gt;Rewriting entire histories. Removing a leaked secret from every commit is a job&amp;nbsp;for &lt;code&gt;git filter-repo&lt;/code&gt;, and the more important step is rotating the secret, because anybody who cloned already has&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;Hooks. Useful for local checks, but they are not distributed with the repository and are easily bypassed. Enforcement belongs on the server or in &lt;span class="caps"&gt;CI&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;Plumbing&amp;nbsp;commands. &lt;code&gt;cat-file&lt;/code&gt;, &lt;code&gt;hash-object&lt;/code&gt; and friends are fascinating if you want to see how the object store works. Pro Git&amp;#8217;s internals chapter does that better than I&amp;nbsp;could.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="the-command-card-i-actually-use"&gt;The Command Card I Actually&amp;nbsp;Use&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Stage changes interactively&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git add -p&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Unstage a file&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git restore --staged &amp;lt;file&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Safe force-push&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git push --force-with-lease --force-if-includes&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Find a lost commit&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git reflog&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Undo the last rebase of a branch&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git reset --hard &amp;lt;branch&amp;gt;@{1}&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Restore another branch moved&amp;nbsp;by &lt;code&gt;--update-refs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git branch -f &amp;lt;other&amp;gt; &amp;lt;other&amp;gt;@{1}&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Undo a pushed commit&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git revert &amp;lt;commit&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Commit a fix into an earlier commit&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git commit --fixup=&amp;lt;commit&amp;gt;&lt;/code&gt;,&amp;nbsp;then &lt;code&gt;git rebase -i main&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Rebase a stack of branches&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git rebase -i --update-refs main&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Compare two versions of a branch&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git range-diff main &amp;lt;branch&amp;gt;@{1} &amp;lt;branch&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Which commits are not in a branch yet&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git cherry -v &amp;lt;upstream&amp;gt; &amp;lt;branch&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backport a commit&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git cherry-pick -x &amp;lt;commit&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Second working directory&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git worktree add ../repo-hotfix -b hotfix main&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Review a Forgejo &lt;span class="caps"&gt;PR&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git fetch origin pull/&amp;lt;n&amp;gt;/head:pr-&amp;lt;n&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Remove a worktree&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git worktree remove ../repo-hotfix&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Find when a string appeared&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git log -S '&amp;lt;string&amp;gt;'&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Find the commit that broke something&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git bisect start &amp;lt;bad&amp;gt; &amp;lt;good&amp;gt;&lt;/code&gt;, &lt;code&gt;git bisect run &amp;lt;script&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sign a commit&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git commit -S&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sign during a rebase&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git rebase -S main&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show signature status&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git log --format='%h %G? %s'&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Signed release tag&lt;/td&gt;
&lt;td&gt;&lt;code&gt;git tag -s v1.0 -m "Release 1.0"&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;None of the features in this article are new. Worktrees have been around for over a decade, cherry-pick and the reflog for even longer. They just never make it into the tutorials, which stop as soon as the first push&amp;nbsp;succeeds.&lt;/p&gt;
&lt;p&gt;What changed my daily work most was not any single command. It was understanding that commits are snapshots, branches are labels and almost nothing is lost. After that, worktrees replaced my stash juggling, fixup commits replaced my &amp;#8220;fix typo&amp;#8221; commits, and I stopped being afraid of rebasing because I knew how to get back. Signing is the last piece: once every commit I push carries a signature from a key that lives on a smartcard, a leaked token is no longer enough to put code in my&amp;nbsp;name.&lt;/p&gt;
&lt;p&gt;If you take only one thing from this article, make&amp;nbsp;it &lt;code&gt;git worktree add&lt;/code&gt;. The next time somebody asks for a quick review while you are in the middle of something, you will see&amp;nbsp;why.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://git-scm.com/book/en/v2"&gt;Pro Git&lt;/a&gt;, the free book by Scott Chacon and Ben Straub, especially the chapters on rewriting history and Git&amp;nbsp;internals&lt;/li&gt;
&lt;li&gt;&lt;a href="https://git-scm.com/docs/git-worktree"&gt;git-worktree(1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://git-scm.com/docs/git-cherry-pick"&gt;git-cherry-pick(1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://git-scm.com/docs/git-rebase"&gt;git-rebase(1)&lt;/a&gt;,&amp;nbsp;including &lt;code&gt;--autosquash&lt;/code&gt; and &lt;code&gt;--update-refs&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://git-scm.com/docs/git-range-diff"&gt;git-range-diff(1)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://git-scm.com/docs/git-bisect"&gt;git-bisect(1)&lt;/a&gt;, including the exit codes&amp;nbsp;for &lt;code&gt;bisect run&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://git-scm.com/docs/git-config#Documentation/git-config.txt-gpgformat"&gt;Signing in Git&lt;/a&gt;,&amp;nbsp;the &lt;code&gt;gpg.*&lt;/code&gt; configuration&amp;nbsp;reference&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/jesseduffield/lazygit"&gt;lazygit&lt;/a&gt;, including its &lt;a href="https://github.com/jesseduffield/lazygit/blob/master/docs/keybindings/Keybindings_en.md"&gt;keybinding&amp;nbsp;reference&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://codeberg.org/openpgp-card/oct-git"&gt;oct-git&lt;/a&gt; from the OpenPGP Card&amp;nbsp;project&lt;/li&gt;
&lt;li&gt;&lt;a href="https://copr.fedorainfracloud.org/coprs/larvitz/openpgp-card/"&gt;larvitz/openpgp-card on Fedora &lt;span class="caps"&gt;COPR&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freshports.org/security/openpgp-card-tools-git/"&gt;security/openpgp-card-tools-git on&amp;nbsp;FreshPorts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/hardware-keys-without-gnupg-the-rust-openpgp-card-stack-now-in-freebsd-ports/"&gt;Hardware Keys Without GnuPG: The Rust OpenPGP Card&amp;nbsp;Stack&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/chezmoi-and-git-dotfiles-without-pretending-every-machine-is-the-same/"&gt;Chezmoi and Git: Dotfiles Without Pretending Every Machine Is the&amp;nbsp;Same&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="git"/><category term="version-control"/><category term="lazygit"/><category term="worktree"/><category term="cherry-pick"/><category term="rebase"/><category term="signing"/><category term="openpgp"/><category term="smartcard"/><category term="oct-git"/><category term="ssh"/><category term="forgejo"/><category term="fedora"/><category term="freebsd"/></entry><entry><title>An IdM Replica at Home: Kerberos That Survives a Dead Tunnel</title><link href="https://hofstede.it/an-idm-replica-at-home-kerberos-that-survives-a-dead-tunnel/" rel="alternate"/><published>2026-11-14T00:00:00+01:00</published><updated>2026-11-14T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-11-14:/an-idm-replica-at-home-kerberos-that-survives-a-dead-tunnel/</id><summary type="html">&lt;p&gt;My two Red Hat Identity Management servers live in a data center, and every &lt;span class="caps"&gt;SSH&lt;/span&gt; login at home depends on a WireGuard tunnel to reach them. This is how a third replica ended up in a bhyve &lt;span class="caps"&gt;VM&lt;/span&gt; on my home hypervisor: a &lt;span class="caps"&gt;DMZ&lt;/span&gt; that only exists inside the host, a tunnel that only worked in one direction, location-aware &lt;span class="caps"&gt;DNS&lt;/span&gt; that was less location-aware than I thought, and a test with the tunnel switched&amp;nbsp;off.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A small rust-colored robot next to a caged hypervisor running a bhyve VM, an IdM replica box with green checkmarks, a jump host showing GSSAPI and Kerberos OK, and a locked gate in front of the data center with a red offline sign" src="https://hofstede.it/images/2026-11-14-idm-replica-at-home.webp" title="The tunnel is down. The tickets keep coming."&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#should-you-even-do-this"&gt;Should You Even Do&amp;nbsp;This?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-starting-point"&gt;The Starting&amp;nbsp;Point&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-real-blocker-a-tunnel-that-only-worked-one-way"&gt;The Real Blocker: A Tunnel That Only Worked One&amp;nbsp;Way&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-dmz-that-only-exists-inside-the-hypervisor"&gt;A &lt;span class="caps"&gt;DMZ&lt;/span&gt; That Only Exists Inside the&amp;nbsp;Hypervisor&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#addressing"&gt;Addressing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#firewall-rules"&gt;Firewall&amp;nbsp;Rules&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#test-both-directions-before-installing-anything"&gt;Test Both Directions Before Installing&amp;nbsp;Anything&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#detour-satellite-says-422"&gt;Detour: Satellite Says&amp;nbsp;422&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#installing-the-replica"&gt;Installing the&amp;nbsp;Replica&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#topology-and-locations"&gt;Topology and Locations&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#why-there-is-no-ca-at-home"&gt;Why There Is No &lt;span class="caps"&gt;CA&lt;/span&gt; at&amp;nbsp;Home&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#dns-at-home-and-where-locations-stop-helping"&gt;&lt;span class="caps"&gt;DNS&lt;/span&gt; at Home, and Where Locations Stop&amp;nbsp;Helping&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-clients"&gt;The Clients&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-laptop"&gt;The&amp;nbsp;Laptop&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#jump01-the-freebsd-jump-host"&gt;jump01, the FreeBSD Jump&amp;nbsp;Host&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#app01-a-regular-idm-client"&gt;app01, a Regular IdM&amp;nbsp;Client&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#autostart"&gt;Autostart&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-test"&gt;The&amp;nbsp;Test&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#caveats"&gt;Caveats&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;My identity setup has a single point of failure, and it is not a server. It is a&amp;nbsp;tunnel.&lt;/p&gt;
&lt;p&gt;The&amp;nbsp;realm &lt;code&gt;IPA.EXAMPLE.COM&lt;/code&gt; is served by two Red Hat Identity Management (IdM)&amp;nbsp;servers, &lt;code&gt;idm1&lt;/code&gt; and &lt;code&gt;idm2&lt;/code&gt;. Both run on different nodes of a Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; cluster in a data center. My home network reaches them through a site-to-site WireGuard tunnel between two OPNsense firewalls, one at home and one in the&amp;nbsp;CoLo.&lt;/p&gt;
&lt;p&gt;A quick note on naming: Red Hat Identity Management (IdM) is Red Hat&amp;#8217;s supported, &lt;span class="caps"&gt;RHEL&lt;/span&gt;-integrated distribution of the upstream &lt;a href="https://www.freeipa.org/"&gt;FreeIPA&lt;/a&gt; project.&amp;nbsp;The &lt;code&gt;ipa&lt;/code&gt; commands, the installers, the replication topology, and the &lt;span class="caps"&gt;DNS&lt;/span&gt; locations in this post work the same way in FreeIPA. I say IdM because that is what runs here, on &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;10.&lt;/p&gt;
&lt;p&gt;All hostnames, addresses, and network names in this post are examples. The structure is real, the names are&amp;nbsp;not.&lt;/p&gt;
&lt;p&gt;At home, &lt;span class="caps"&gt;SSH&lt;/span&gt; goes through a single choke point: a jail&amp;nbsp;called &lt;code&gt;jump01&lt;/code&gt; on my FreeBSD&amp;nbsp;hypervisor &lt;code&gt;hv01&lt;/code&gt;. It accepts &lt;span class="caps"&gt;GSSAPI&lt;/span&gt; and nothing else. No keys, no passwords, no keyboard-interactive. If I want a shell&amp;nbsp;on &lt;code&gt;hv01&lt;/code&gt;, I need a Kerberos service ticket&amp;nbsp;for &lt;code&gt;jump01&lt;/code&gt;, and to get one, I need a &lt;span class="caps"&gt;KDC&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;As long as the tunnel is up, that works well. When it is down, my existing tickets keep working until they expire, and after that I am locked out of my own hypervisor from my own sofa. I have a PiKVM and a second way in through the firewall, so this is an inconvenience rather than a disaster. But it is a silly dependency, and it bothered&amp;nbsp;me.&lt;/p&gt;
&lt;p&gt;So the silly idea: put a third IdM replica at&amp;nbsp;home.&lt;/p&gt;
&lt;h2 id="should-you-even-do-this"&gt;Should You Even Do&amp;nbsp;This?&lt;/h2&gt;
&lt;p&gt;I was not sure at first, and the answer is not an automatic&amp;nbsp;yes.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;What a local replica gives&amp;nbsp;you:&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;A &lt;span class="caps"&gt;KDC&lt;/span&gt; and &lt;span class="caps"&gt;LDAP&lt;/span&gt; server that do not care about the tunnel.&lt;/strong&gt; Tickets and user lookups keep working at home. When the tunnel comes back, the replica catches up with whatever&amp;nbsp;changed.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Lower latency.&lt;/strong&gt; A ticket request no longer makes a round trip through the&amp;nbsp;tunnel.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;What it does not give&amp;nbsp;you:&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Anything else behind the tunnel.&lt;/strong&gt; Satellite, other internal services, and the IdM web &lt;span class="caps"&gt;UI&lt;/span&gt; in the data center stay unreachable. A locally issued ticket does not fix&amp;nbsp;routing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Anything for a laptop that already copes offline.&lt;/strong&gt; My laptop logs in with a smartcard and works without a &lt;span class="caps"&gt;KDC&lt;/span&gt; (that is a story for another post). It just has no fresh ticket&amp;nbsp;afterwards.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;What it&amp;nbsp;costs:&lt;/strong&gt;&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;A full copy of the realm.&lt;/strong&gt; Every IdM replica holds every Kerberos key and every password hash. A replica at home means the most valuable system I run now also sits on a home hypervisor, in its backups, and in its&amp;nbsp;snapshots.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Another server to patch and monitor.&lt;/strong&gt; It also needs reliable time and open ports in both directions through the tunnel. That last part turned out to be the interesting&amp;nbsp;one.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you only want Kerberos to work without the tunnel, IdM&amp;#8217;s &lt;span class="caps"&gt;KDC&lt;/span&gt; proxy is the leaner option. It carries Kerberos over &lt;span class="caps"&gt;HTTPS&lt;/span&gt;, needs one reverse proxy entry, and works on the road as well, without copying the realm&amp;nbsp;anywhere.&lt;/p&gt;
&lt;p&gt;What tipped the decision for me was the &lt;span class="caps"&gt;SSH&lt;/span&gt; setup. With &lt;span class="caps"&gt;GSSAPI&lt;/span&gt; as the only way&amp;nbsp;into &lt;code&gt;jump01&lt;/code&gt;, a &lt;span class="caps"&gt;KDC&lt;/span&gt; at home is not a nice-to-have. It is the difference between &amp;#8220;the tunnel is down, annoying&amp;#8221; and &amp;#8220;the tunnel is down, where is the PiKVM&amp;nbsp;password&amp;#8221;.&lt;/p&gt;
&lt;p&gt;Two decisions came out of that&amp;nbsp;weighing:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;The replica gets its own &lt;span class="caps"&gt;VM&lt;/span&gt;.&lt;/strong&gt; I already had a &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 &lt;span class="caps"&gt;VM&lt;/span&gt; at&amp;nbsp;home, &lt;code&gt;app01&lt;/code&gt;, but it runs a rootful container with a network-facing Java application. If that container is ever compromised, the attacker is root on the host. I do not want that host to also hold the&amp;nbsp;realm.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No &lt;span class="caps"&gt;CA&lt;/span&gt; role.&lt;/strong&gt; The &lt;span class="caps"&gt;CA&lt;/span&gt; private key stays where it already is, on the two servers in the data center. More on what that means&amp;nbsp;later.&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="the-starting-point"&gt;The Starting&amp;nbsp;Point&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;  Home (hv01, FreeBSD 15, vm-bhyve)               Data center (Proxmox VE)
 ┌────────────────────────────────────┐        ┌─────────────────────────┐
 │  LAN     192.168.10.0/24           │        │  idm1                   │
 │  IOT     192.168.20.0/24           │        │  idm2                   │
 │  WORK    192.168.30.0/24           │        │  satellite              │
 │                                    │        │  ...                    │
 │  fw-home (OPNsense VM) ── wg0 ─────┼────────┼── fw-dc (OPNsense)      │
 │                                    │  WG    │  172.16.0.0/16          │
 │  jump01 (jail, SSH choke point)    │        │                         │
 └────────────────────────────────────┘        └─────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Home &lt;span class="caps"&gt;DNS&lt;/span&gt; is Unbound on the home OPNsense&amp;nbsp;(&lt;code&gt;fw-home&lt;/code&gt;). It answers&amp;nbsp;for &lt;code&gt;home.example.com&lt;/code&gt;, &lt;code&gt;iot.example.com&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;work.example.com&lt;/code&gt; itself, and&amp;nbsp;forwards &lt;code&gt;ipa.example.com&lt;/code&gt; through the tunnel to the IdM servers, which run the zone with integrated &lt;span class="caps"&gt;DNS&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;Clients at home have to resolve that zone when the tunnel is down, so the replica needs the &lt;span class="caps"&gt;DNS&lt;/span&gt; role as well. Otherwise the &lt;span class="caps"&gt;KDC&lt;/span&gt; at home is perfectly healthy and nobody can find&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="the-real-blocker-a-tunnel-that-only-worked-one-way"&gt;The Real Blocker: A Tunnel That Only Worked One&amp;nbsp;Way&lt;/h2&gt;
&lt;p&gt;The first serious obstacle had nothing to do with&amp;nbsp;IdM.&lt;/p&gt;
&lt;p&gt;Home to data center worked. Data center to home did not. Years ago, I had set up the tunnel so that the home &lt;span class="caps"&gt;LAN&lt;/span&gt; was masqueraded behind a single tunnel address. The roughly ten networks in the data center had no idea my private home ranges existed, and I preferred to keep it that&amp;nbsp;way.&lt;/p&gt;
&lt;p&gt;That is fatal for a replica. IdM replication is not a client pulling from a server. Every server opens its own connections to its replication partners to push changes, and the installer checks connectivity in both directions before it changes anything. Behind masquerading, nothing from the data center would ever reach the&amp;nbsp;replica.&lt;/p&gt;
&lt;p&gt;The fix does not require announcing all home networks to the data center. It only requires one address, the replica&amp;#8217;s own, to be routable in both directions without &lt;span class="caps"&gt;NAT&lt;/span&gt;. A port forward on the masqueraded address would technically work too, but then the replica would have one address at home and another one in the data center, and IdM finds its peers by hostname. One name, one set of addresses, everywhere. Anything else is a debugging session waiting to&amp;nbsp;happen.&lt;/p&gt;
&lt;p&gt;When I looked at the &lt;span class="caps"&gt;NAT&lt;/span&gt; rules on the home OPNsense, the situation turned out better than I&amp;nbsp;remembered:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;IPv4:&lt;/strong&gt; the masquerading rule only&amp;nbsp;matched &lt;code&gt;LAN net&lt;/code&gt; towards &lt;code&gt;172.16.0.0/16&lt;/code&gt;. A new network would simply not be covered by it. On the CoLo side, the home peer&amp;#8217;s allowed IPs only contained the tunnel&amp;nbsp;address &lt;code&gt;10.99.0.2/32&lt;/code&gt;, so return traffic for anything else had nowhere to&amp;nbsp;go.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;IPv6:&lt;/strong&gt; the CoLo peer already allowed my&amp;nbsp;entire &lt;code&gt;2001:db8:aa00::/56&lt;/code&gt;. IPv6 had been crossing the tunnel unmasqueraded and routable in both directions all along. I had just never needed it from that&amp;nbsp;side.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Which sets the priorities nicely. The replica speaks IPv6 first, and IPv4 gets a&amp;nbsp;single &lt;code&gt;/32&lt;/code&gt; added to the CoLo peer, so that clients and tools that pick the A record do not run into a timeout. Legacy &lt;span class="caps"&gt;IP&lt;/span&gt; is a fallback, not the&amp;nbsp;plan.&lt;/p&gt;
&lt;h2 id="a-dmz-that-only-exists-inside-the-hypervisor"&gt;A &lt;span class="caps"&gt;DMZ&lt;/span&gt; That Only Exists Inside the&amp;nbsp;Hypervisor&lt;/h2&gt;
&lt;p&gt;The replica gets its own network, the &lt;span class="caps"&gt;DMZ&lt;/span&gt;: &lt;code&gt;192.168.40.0/24&lt;/code&gt; and &lt;code&gt;2001:db8:aa00:40::/64&lt;/code&gt;. Clients from the home network reach it through the firewall. The replica itself is not allowed to open anything towards my other home networks. If it is ever compromised, it does not get a free path into the rest of the&amp;nbsp;house.&lt;/p&gt;
&lt;p&gt;It also needs no &lt;span class="caps"&gt;SSID&lt;/span&gt; and no switch port. Nothing ever plugs into this network&amp;nbsp;directly. &lt;code&gt;fw-home&lt;/code&gt; is a &lt;span class="caps"&gt;VM&lt;/span&gt; on the same host, so the network only has to exist&amp;nbsp;inside &lt;code&gt;hv01&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;My existing layout already follows that pattern. Each network is a host bridge, vm-bhyve attaches to it as&amp;nbsp;a &lt;code&gt;manual&lt;/code&gt; switch, and the OPNsense &lt;span class="caps"&gt;VM&lt;/span&gt; has one &lt;span class="caps"&gt;NIC&lt;/span&gt; per&amp;nbsp;network:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;root&lt;/span&gt;&lt;span class="nv"&gt;@hv01&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;vm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;switch&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;list&lt;/span&gt;
&lt;span class="n"&gt;NAME&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;TYPE&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;IFACE&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;ADDRESS&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;PRIVATE&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;MTU&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;VLAN&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;PORTS&lt;/span&gt;
&lt;span class="n"&gt;vm&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;lan&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="n"&gt;manual&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;bridge0&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="k"&gt;no&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
&lt;span class="n"&gt;vm&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;iot&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="n"&gt;manual&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;bridge1&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="k"&gt;no&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
&lt;span class="n"&gt;vm&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;mgmt&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;manual&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;bridge2&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="k"&gt;no&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
&lt;span class="n"&gt;vm&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="k"&gt;work&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;manual&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;bridge3&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="k"&gt;no&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Only the trunk towards the access point carries tags. So the &lt;span class="caps"&gt;DMZ&lt;/span&gt; becomes one more bridge, with no physical member at&amp;nbsp;all:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;cloned_interfaces&lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot; bridge4&amp;quot;&lt;/span&gt;
sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;ifconfig_bridge4&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;up&amp;quot;&lt;/span&gt;
ifconfig&lt;span class="w"&gt; &lt;/span&gt;bridge4&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;up
vm&lt;span class="w"&gt; &lt;/span&gt;switch&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;manual&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;bridge4&lt;span class="w"&gt; &lt;/span&gt;vm-dmz
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two details matter&amp;nbsp;here:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;No &lt;code&gt;addm&lt;/code&gt;.&lt;/strong&gt; &lt;code&gt;bridge4&lt;/code&gt; has no physical port and no &lt;span class="caps"&gt;VLAN&lt;/span&gt; interface. The network never appears on the trunk, and nobody can plug a cable into&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No address on the&amp;nbsp;bridge.&lt;/strong&gt; &lt;code&gt;hv01&lt;/code&gt; itself has no &lt;span class="caps"&gt;IP&lt;/span&gt; in the &lt;span class="caps"&gt;DMZ&lt;/span&gt;, so the hypervisor is not reachable from&amp;nbsp;it.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;code&gt;fw-home&lt;/code&gt; gets a fifth &lt;span class="caps"&gt;NIC&lt;/span&gt;, appended at the end so that the existing interfaces in OPNsense keep their&amp;nbsp;order:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;network0_type&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;virtio-net&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;network0_switch&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vm-lan&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Home&lt;/span&gt;
&lt;span class="nx"&gt;network1_type&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;virtio-net&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;network1_switch&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vm-iot&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IoT&lt;/span&gt;
&lt;span class="nx"&gt;network2_type&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;virtio-net&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;network2_switch&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vm-mgmt&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;MGMT&lt;/span&gt;
&lt;span class="nx"&gt;network3_type&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;virtio-net&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;network3_switch&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vm-work&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;WORK&lt;/span&gt;
&lt;span class="nx"&gt;network4_type&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;virtio-net&amp;quot;&lt;/span&gt;
&lt;span class="nx"&gt;network4_switch&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;vm-dmz&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;DMZ&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;vm-bhyve generates the &lt;span class="caps"&gt;MAC&lt;/span&gt; address on first start and writes it into the file itself. My first attempt&amp;nbsp;had &lt;code&gt;vm-work&lt;/code&gt; in the last line, copy and paste being what it is. That would have put the &lt;span class="caps"&gt;DMZ&lt;/span&gt; interface straight into the network for my work devices. Read your&amp;nbsp;diffs.&lt;/p&gt;
&lt;p&gt;bhyve cannot hot-add NICs, so the new device only appears when bhyve starts the &lt;span class="caps"&gt;VM&lt;/span&gt; again. I did not rely&amp;nbsp;on &lt;code&gt;vm restart&lt;/code&gt; or a guest reboot and did a full stop and start&amp;nbsp;of &lt;code&gt;fw-home&lt;/code&gt;. Restarting the firewall &lt;span class="caps"&gt;VM&lt;/span&gt; takes down internet, Wi-Fi routing, and the tunnel, and in my case also the &lt;span class="caps"&gt;KDC&lt;/span&gt; path for any new &lt;span class="caps"&gt;SSH&lt;/span&gt; login. I made sure I had an open session&amp;nbsp;on &lt;code&gt;hv01&lt;/code&gt; and the PiKVM as a fallback before I touched it. It came back fine, as it usually does when you prepare for it not&amp;nbsp;to.&lt;/p&gt;
&lt;p&gt;In OPNsense, the new &lt;span class="caps"&gt;NIC&lt;/span&gt; shows up as one&amp;nbsp;more &lt;code&gt;vtnet&lt;/code&gt; interface. I assigned it&amp;nbsp;as &lt;code&gt;DMZ&lt;/code&gt; with static addresses in both&amp;nbsp;families.&lt;/p&gt;
&lt;h2 id="addressing"&gt;Addressing&lt;/h2&gt;
&lt;p&gt;The replica&amp;nbsp;got:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;idm-home.ipa.example.com&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;2001:db8:aa00:40::10/64&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;192.168.40.10/24&lt;/code&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Using a global IPv6 address for an IdM server is only sane if the prefix never changes. Mine is static, part of a business line, and has not changed in years. With a dynamic prefix, I would have put a &lt;span class="caps"&gt;ULA&lt;/span&gt; prefix on the &lt;span class="caps"&gt;DMZ&lt;/span&gt; and registered only that in IdM, because the replica is supposed to work precisely when the uplink is broken, and a delegated prefix can disappear along with the&amp;nbsp;uplink.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;VM&lt;/span&gt; is configured statically, with no &lt;span class="caps"&gt;SLAAC&lt;/span&gt; and no DHCPv6. IdM writes the server&amp;#8217;s addresses into &lt;span class="caps"&gt;DNS&lt;/span&gt; during installation, and an extra privacy address that appears on the interface has no business ending up&amp;nbsp;there.&lt;/p&gt;
&lt;p&gt;The name is final. An IdM server cannot be renamed after installation. The &lt;span class="caps"&gt;VM&lt;/span&gt; name in bhyve does not matter to IdM. What matters is the hostname inside the&amp;nbsp;guest.&lt;/p&gt;
&lt;h2 id="firewall-rules"&gt;Firewall&amp;nbsp;Rules&lt;/h2&gt;
&lt;p&gt;With aliases, the rule set stays readable. They are identical on both&amp;nbsp;firewalls:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Alias&lt;/th&gt;
&lt;th&gt;Type&lt;/th&gt;
&lt;th&gt;Contents&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;IDM_HOME&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Hosts&lt;/td&gt;
&lt;td&gt;&lt;code&gt;192.168.40.10&lt;/code&gt;, &lt;code&gt;2001:db8:aa00:40::10&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;IDM_DC&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Hosts&lt;/td&gt;
&lt;td&gt;v4 and v6 addresses&amp;nbsp;of &lt;code&gt;idm1&lt;/code&gt; and &lt;code&gt;idm2&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;IDM_TCP&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Ports&lt;/td&gt;
&lt;td&gt;80, 443, 389, 636, 88, 464, 53&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;IDM_UDP&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Ports&lt;/td&gt;
&lt;td&gt;88, 464, 53&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The port list matches what the &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 documentation lists for IdM, and what&amp;nbsp;the &lt;code&gt;freeipa-4&lt;/code&gt; and &lt;code&gt;dns&lt;/code&gt; firewalld services open on the &lt;span class="caps"&gt;VM&lt;/span&gt; itself. Port 53 comes&amp;nbsp;from &lt;code&gt;dns&lt;/code&gt;, the rest&amp;nbsp;from &lt;code&gt;freeipa-4&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Home&amp;nbsp;OPNsense:&lt;/strong&gt;&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Interface&lt;/th&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;Destination&lt;/th&gt;
&lt;th&gt;Ports&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;DMZ&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_HOME&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Satellite&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;TCP&lt;/span&gt; 443&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;DMZ&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_HOME&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_DC&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_TCP&lt;/code&gt;, &lt;code&gt;IDM_UDP&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;wg0&lt;/code&gt; (inbound)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_DC&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_HOME&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_TCP&lt;/code&gt;, &lt;code&gt;IDM_UDP&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;LAN&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;LAN&lt;/span&gt; net&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;DMZ&lt;/span&gt; net&lt;/td&gt;
&lt;td&gt;any&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;strong&gt;CoLo&amp;nbsp;OPNsense:&lt;/strong&gt;&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Interface&lt;/th&gt;
&lt;th&gt;Source&lt;/th&gt;
&lt;th&gt;Destination&lt;/th&gt;
&lt;th&gt;Ports&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;WireGuard (inbound)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_HOME&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Satellite&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;TCP&lt;/span&gt; 443&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;WireGuard (inbound)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_HOME&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_DC&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_TCP&lt;/code&gt;, &lt;code&gt;IDM_UDP&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Server &lt;span class="caps"&gt;VLAN&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_DC&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_HOME&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;IDM_TCP&lt;/code&gt;, &lt;code&gt;IDM_UDP&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;My &lt;span class="caps"&gt;LAN&lt;/span&gt; rule allows everything towards the &lt;span class="caps"&gt;DMZ&lt;/span&gt;, because that is how I treat my trusted &lt;span class="caps"&gt;LAN&lt;/span&gt;. The minimum the clients need is Kerberos, &lt;span class="caps"&gt;LDAP&lt;/span&gt;, &lt;span class="caps"&gt;HTTPS&lt;/span&gt;, and &lt;span class="caps"&gt;DNS&lt;/span&gt;,&amp;nbsp;so &lt;code&gt;IDM_TCP&lt;/code&gt; and &lt;code&gt;IDM_UDP&lt;/code&gt; would work there as&amp;nbsp;well.&lt;/p&gt;
&lt;p&gt;Plus &lt;code&gt;192.168.40.10/32&lt;/code&gt; in the allowed IPs of the home peer on the CoLo side. OPNsense creates the route from the allowed IPs on its&amp;nbsp;own.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;DMZ&lt;/span&gt; rules sit above my standard &amp;#8220;everything&amp;nbsp;except &lt;code&gt;INTERNAL_NETS&lt;/code&gt;&amp;#8221; rule that every &lt;span class="caps"&gt;VLAN&lt;/span&gt; has. That rule lets the network reach the internet but none of my own networks, which includes the data center. Without the explicit rules above it, the replica could reach the internet but not its own replication&amp;nbsp;partners.&lt;/p&gt;
&lt;p&gt;The work network does not appear anywhere in these tables, and that is deliberate. The work network is for work devices, including customer laptops. Nothing from there gets near my private infrastructure, and certainly not a server holding every key of the&amp;nbsp;realm.&lt;/p&gt;
&lt;p&gt;Three things I tripped over on the&amp;nbsp;way:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;No internet&amp;#8221; in the installer was not the firewall.&lt;/strong&gt; Unbound on OPNsense simply was not listening on the new interface. &lt;span class="caps"&gt;DNS&lt;/span&gt; timeouts look exactly like a routing problem until you ping an &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;address.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A missing &lt;span class="caps"&gt;UDP&lt;/span&gt; rule.&lt;/strong&gt; On the CoLo WireGuard interface, I had cloned the &lt;span class="caps"&gt;TCP&lt;/span&gt; rule and never created its &lt;span class="caps"&gt;UDP&lt;/span&gt; twin. &lt;span class="caps"&gt;TCP&lt;/span&gt; checks would have passed, and Kerberos and &lt;span class="caps"&gt;DNS&lt;/span&gt; over &lt;span class="caps"&gt;UDP&lt;/span&gt; would have hung. Reviewing all four rule sets side by side caught&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;ICMP&lt;/span&gt;.&lt;/strong&gt; I allow &lt;span class="caps"&gt;ICMP&lt;/span&gt; everywhere, via floating rules that apply to every internal interface. Twenty years of debugging path &lt;span class="caps"&gt;MTU&lt;/span&gt; problems through tunnels will do that to you. On a&amp;nbsp;new &lt;code&gt;wg0&lt;/code&gt; inbound rule set, though, &lt;span class="caps"&gt;ICMP&lt;/span&gt; was not covered, and my first ping test from the data center failed for reasons that said nothing about &lt;span class="caps"&gt;TCP&lt;/span&gt; or &lt;span class="caps"&gt;UDP&lt;/span&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="test-both-directions-before-installing-anything"&gt;Test Both Directions Before Installing&amp;nbsp;Anything&lt;/h2&gt;
&lt;p&gt;The direction that never existed before is data center to home. Before running a single IdM command, I wanted proof that it worked on a real IdM port, and that the source addresses arrived&amp;nbsp;unchanged.&lt;/p&gt;
&lt;p&gt;Nothing listens on 389 on a fresh &lt;span class="caps"&gt;VM&lt;/span&gt;, so I started a throwaway listener on it. The &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;#8217;s own firewall has to allow the port first, otherwise the test fails at the last hop. Opening the IdM services now saves a step later&amp;nbsp;anyway:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--permanent&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;={&lt;/span&gt;freeipa-4,dns&lt;span class="o"&gt;}&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--reload
&lt;span class="nb"&gt;cd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;mktemp&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;python3&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;http.server&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;389&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--bind&lt;span class="w"&gt; &lt;/span&gt;::
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;--bind ::&lt;/code&gt; is not optional. My first attempt without it reported &amp;#8220;Serving &lt;span class="caps"&gt;HTTP&lt;/span&gt; on 0.0.0.0&amp;#8221;, listened on IPv4 only, and the IPv6 test from the data center failed. I spent a minute suspecting the firewall before reading the listener&amp;#8217;s own&amp;nbsp;output.&lt;/p&gt;
&lt;p&gt;From &lt;code&gt;idm1&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;admin@idm1 ~&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;curl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;null&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%{http_code} %{remote_ip}\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="k"&gt;connect&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;http://[2001:db8:aa00:40::10]:389/&amp;#39;&lt;/span&gt;
&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="nl"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nl"&gt;aa00&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;40&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;admin@idm1 ~&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;curl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;o&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;dev&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="k"&gt;null&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%{http_code} %{remote_ip}\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="k"&gt;connect&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;http&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;//&lt;/span&gt;&lt;span class="mf"&gt;192.168.40.10&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;389&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;span class="mi"&gt;200&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;192.168.40.10&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The listener&amp;#8217;s log on the &lt;span class="caps"&gt;VM&lt;/span&gt; shows the source of each request. It has to be the real address&amp;nbsp;of &lt;code&gt;idm1&lt;/code&gt;, not a tunnel address. For the other direction, the Satellite&amp;#8217;s request log did the job for free once the &lt;span class="caps"&gt;VM&lt;/span&gt; registered&amp;nbsp;there:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;Started&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;POST&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;/register&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mh"&gt;2001&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="nl"&gt;db8:aa00:&lt;/span&gt;&lt;span class="mh"&gt;40&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="mh"&gt;10&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Both families, both directions, real addresses. Only now does IdM get&amp;nbsp;involved.&lt;/p&gt;
&lt;h2 id="detour-satellite-says-422"&gt;Detour: Satellite Says&amp;nbsp;422&lt;/h2&gt;
&lt;p&gt;The &lt;span class="caps"&gt;VM&lt;/span&gt; gets its content from my Satellite, like every other &lt;span class="caps"&gt;RHEL&lt;/span&gt; system I run. Registration failed with an &lt;span class="caps"&gt;HTTP&lt;/span&gt; 422 from the host initialization step, after the subscription-manager part had already succeeded. The Satellite log was clear about&amp;nbsp;why:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Not queueing Host::Managed: [&amp;quot;Ip does not match selected subnet&amp;quot;]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The host group I registered into has a fixed subnet from the data center. A host&amp;nbsp;in &lt;code&gt;192.168.40.0/24&lt;/code&gt; cannot be in it. The fix was a nested host group, &amp;#8220;Home-&lt;span class="caps"&gt;DMZ&lt;/span&gt;&amp;#8221;, that inherits everything from the parent and only overrides the IPv4 and IPv6 subnets with two new ones for the &lt;span class="caps"&gt;DMZ&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The next attempt failed again, with a different&amp;nbsp;message:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="s2"&gt;&amp;quot;Subnet is not defined for host&amp;#39;s location&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Subnet is not defined for host&amp;#39;s organization&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I had created the new subnets in the &amp;#8220;Any Organization / Any Location&amp;#8221; context, and subnets created there are not assigned to any location or organization. With a specific context selected, Satellite assigns them automatically. Once they were assigned, registration went through. Insights then failed with a 401, which is expected in my setup and harmless. Everything that mattered had already&amp;nbsp;happened.&lt;/p&gt;
&lt;h2 id="installing-the-replica"&gt;Installing the&amp;nbsp;Replica&lt;/h2&gt;
&lt;p&gt;On &lt;code&gt;idm1&lt;/code&gt;, with an admin ticket, the &lt;span class="caps"&gt;DNS&lt;/span&gt; records go in first. The installer would try to add them itself, but the host is not allowed to, and having them in place early lets you check resolution before the installer&amp;nbsp;does:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;dnsrecord-add&lt;span class="w"&gt; &lt;/span&gt;ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;idm-home&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--aaaa-rec&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8:aa00:40::10&lt;span class="w"&gt; &lt;/span&gt;--a-rec&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.168.40.10
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then the host entry with a one-time password, and membership&amp;nbsp;in &lt;code&gt;ipaservers&lt;/code&gt;, which allows the host to promote itself to a&amp;nbsp;replica:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;host-add&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;--random
ipa&lt;span class="w"&gt; &lt;/span&gt;hostgroup-add-member&lt;span class="w"&gt; &lt;/span&gt;ipaservers&lt;span class="w"&gt; &lt;/span&gt;--hosts&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The one-time password means I never type an admin password on the new machine. It fits a setup where admins authenticate with smartcards, and it is the cleaner path&amp;nbsp;anyway.&lt;/p&gt;
&lt;p&gt;On the &lt;span class="caps"&gt;VM&lt;/span&gt;, with the firewall already open from the listener&amp;nbsp;test:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;ipa-server&lt;span class="w"&gt; &lt;/span&gt;ipa-server-dns
ipa-client-install&lt;span class="w"&gt; &lt;/span&gt;--domain&lt;span class="w"&gt; &lt;/span&gt;ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;--realm&lt;span class="w"&gt; &lt;/span&gt;IPA.EXAMPLE.COM&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--mkhomedir&lt;span class="w"&gt; &lt;/span&gt;--password&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ONE-TIME-PASSWORD&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The client installer complained that the host had no A or &lt;span class="caps"&gt;AAAA&lt;/span&gt; record, although I had just created both. Most likely, Unbound at home had cached a negative answer from an earlier lookup. A few minutes later, four queries against three resolvers all returned the right addresses. The replica installer checks its own name early and aborts if it does not resolve, so it is worth checking before you start&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dig&lt;span class="w"&gt; &lt;/span&gt;+short&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;AAAA
dig&lt;span class="w"&gt; &lt;/span&gt;+short&lt;span class="w"&gt; &lt;/span&gt;@idm1.ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;AAAA
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then the promotion,&amp;nbsp;in &lt;code&gt;tmux&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa-replica-install&lt;span class="w"&gt; &lt;/span&gt;--setup-dns&lt;span class="w"&gt; &lt;/span&gt;--forwarder&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8:aa00:40::1&lt;span class="w"&gt; &lt;/span&gt;--no-reverse&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ip-address&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8:aa00:40::10&lt;span class="w"&gt; &lt;/span&gt;--ip-address&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.168.40.10
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;--setup-dns&lt;/code&gt; makes the replica a &lt;span class="caps"&gt;DNS&lt;/span&gt; server&amp;nbsp;for &lt;code&gt;ipa.example.com&lt;/code&gt;. Without it, the &lt;span class="caps"&gt;KDC&lt;/span&gt; at home works and nobody finds it when the tunnel is&amp;nbsp;down.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;--no-reverse&lt;/code&gt; keeps IdM away from reverse zones. The home ranges belong to the&amp;nbsp;OPNsense.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;--ip-address&lt;/code&gt; twice, explicitly, so that only these two addresses end up in &lt;span class="caps"&gt;DNS&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;No &lt;code&gt;--setup-ca&lt;/code&gt;. The replica gets its certificates from the existing &lt;span class="caps"&gt;CA&lt;/span&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The installer opened with a&amp;nbsp;question:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;Unable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;resolve&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;IP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;172.16.1.12&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;check&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;hosts&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;DNS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;resolution&lt;/span&gt;
&lt;span class="n"&gt;Lookup&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;failed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Preferred&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idm&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;home&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ipa&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;does&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;provide&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;DNS&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="nf"&gt;Reverse&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;DNS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;resolution&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;...&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;idm2&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ipa&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;failed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Clients&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;may&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;not&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;function&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;properly&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Please&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;check&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;your&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;DNS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;setup&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="k"&gt;Continue&lt;/span&gt;&lt;span class="vm"&gt;?&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;no&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;My data center servers have never had &lt;span class="caps"&gt;PTR&lt;/span&gt; records. IdM&amp;nbsp;sets &lt;code&gt;rdns = false&lt;/code&gt; in its Kerberos configuration, and the two existing servers have replicated happily for years without them. The second line is informational: the &lt;span class="caps"&gt;VM&lt;/span&gt; is not a &lt;span class="caps"&gt;DNS&lt;/span&gt; server yet, that is what this run is for.&amp;nbsp;So: &lt;code&gt;yes&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;After&amp;nbsp;that, &lt;code&gt;Connection check OK&lt;/code&gt;, 40 steps of Directory Server setup, an initial replication that took 26 seconds, &lt;span class="caps"&gt;KDC&lt;/span&gt;, httpd, and &lt;span class="caps"&gt;DNS&lt;/span&gt;. Three lines in the output looked alarming and were&amp;nbsp;not:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Replica &lt;span class="caps"&gt;DNS&lt;/span&gt; records could not be added on master: Insufficient access&amp;#8221;.&lt;/strong&gt; The host may not create its own records. They were already&amp;nbsp;there.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Could not add certificate to service &amp;#8230; no modifications to be performed&amp;#8221;.&lt;/strong&gt; The certificate was already in the service&amp;nbsp;entry.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Global forwarders: 198.51.100.53, Forward policy: only&amp;#8221;.&lt;/strong&gt; This one made me look twice. My IdM &lt;span class="caps"&gt;DNS&lt;/span&gt; has global forwarders configured, and the installer points out that they override the forwarders in the&amp;nbsp;local &lt;code&gt;named.conf&lt;/code&gt;. That does not mean they win&amp;nbsp;over &lt;code&gt;--forwarder&lt;/code&gt;. The installer stores the option as a per-server setting in &lt;span class="caps"&gt;LDAP&lt;/span&gt;, and the per-server setting takes precedence over the global&amp;nbsp;one:&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;dnsserver-show&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com
&lt;span class="w"&gt;  &lt;/span&gt;Server&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com
&lt;span class="w"&gt;  &lt;/span&gt;SOA&lt;span class="w"&gt; &lt;/span&gt;mname&lt;span class="w"&gt; &lt;/span&gt;override:&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com.
&lt;span class="w"&gt;  &lt;/span&gt;Forwarders:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2001&lt;/span&gt;:db8:aa00:40::1
&lt;span class="w"&gt;  &lt;/span&gt;Forward&lt;span class="w"&gt; &lt;/span&gt;policy:&lt;span class="w"&gt; &lt;/span&gt;only
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So the replica sends foreign names to Unbound on the home OPNsense, as intended, while the data center servers keep using their own forwarders. If the internet connection is down, it still answers&amp;nbsp;for &lt;code&gt;ipa.example.com&lt;/code&gt;, which is the only thing I need from it in that&amp;nbsp;situation.&lt;/p&gt;
&lt;p&gt;And&amp;nbsp;then: &lt;code&gt;The ipa-replica-install command was successful&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="topology-and-locations"&gt;Topology and&amp;nbsp;Locations&lt;/h2&gt;
&lt;p&gt;The installer creates one replication segment, to the server it installed against. Discovery had&amp;nbsp;picked &lt;code&gt;idm2&lt;/code&gt;. A replica connected to a single partner loses replication when that partner goes away, so the second segment is manual. Segments are bidirectional, so one call per pair is&amp;nbsp;enough:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;topologysegment-add&lt;span class="w"&gt; &lt;/span&gt;domain&lt;span class="w"&gt; &lt;/span&gt;idm-home-to-idm1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--leftnode&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;--rightnode&lt;span class="w"&gt; &lt;/span&gt;idm1.ipa.example.com
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then a &lt;span class="caps"&gt;DNS&lt;/span&gt; location for home, and the replica in&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;location-add&lt;span class="w"&gt; &lt;/span&gt;home&lt;span class="w"&gt; &lt;/span&gt;--description&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Home network DMZ&amp;quot;&lt;/span&gt;
ipa&lt;span class="w"&gt; &lt;/span&gt;server-mod&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;--location&lt;span class="w"&gt; &lt;/span&gt;home
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;named&lt;/code&gt; on the replica needs a restart to pick up the change. After that, the replica answers &lt;span class="caps"&gt;SRV&lt;/span&gt; queries with a location-specific view. It points&amp;nbsp;to &lt;code&gt;home._locations&lt;/code&gt; and lists itself with priority 0, ahead of the data center servers with&amp;nbsp;50:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;dig&lt;span class="w"&gt; &lt;/span&gt;+short&lt;span class="w"&gt; &lt;/span&gt;@idm-home.ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;_kerberos._udp.ipa.example.com&lt;span class="w"&gt; &lt;/span&gt;SRV
_kerberos._udp.home._locations.ipa.example.com.
&lt;span class="m"&gt;50&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;88&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;idm2.ipa.example.com.
&lt;span class="m"&gt;50&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;88&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;idm1.ipa.example.com.
&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;88&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The resulting domain topology is a triangle, while &lt;span class="caps"&gt;CA&lt;/span&gt; replication stays a single line between the two data center&amp;nbsp;servers:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;topologysegment-find&lt;span class="w"&gt; &lt;/span&gt;domain
&lt;span class="w"&gt;  &lt;/span&gt;Segment&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;idm1.ipa.example.com-to-idm2.ipa.example.com
&lt;span class="w"&gt;  &lt;/span&gt;Segment&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;idm-home-to-idm1
&lt;span class="w"&gt;  &lt;/span&gt;Segment&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;idm-home.ipa.example.com-to-idm2.ipa.example.com

$&lt;span class="w"&gt; &lt;/span&gt;ipa&lt;span class="w"&gt; &lt;/span&gt;topologysegment-find&lt;span class="w"&gt; &lt;/span&gt;ca
&lt;span class="w"&gt;  &lt;/span&gt;Segment&lt;span class="w"&gt; &lt;/span&gt;name:&lt;span class="w"&gt; &lt;/span&gt;idm1.ipa.example.com-to-idm2.ipa.example.com
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="why-there-is-no-ca-at-home"&gt;Why There Is No &lt;span class="caps"&gt;CA&lt;/span&gt; at&amp;nbsp;Home&lt;/h3&gt;
&lt;p&gt;The topology graph in the web &lt;span class="caps"&gt;UI&lt;/span&gt; shows the missing &lt;span class="caps"&gt;CA&lt;/span&gt; very clearly: dashed &lt;span class="caps"&gt;CA&lt;/span&gt; lines between the two data center servers, nothing&amp;nbsp;to &lt;code&gt;idm-home&lt;/code&gt;. That is the&amp;nbsp;point.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;It is not needed for what I want.&lt;/strong&gt; Kerberos, &lt;span class="caps"&gt;LDAP&lt;/span&gt;, &lt;span class="caps"&gt;DNS&lt;/span&gt;, and smartcard logins all work without a local &lt;span class="caps"&gt;CA&lt;/span&gt;. The &lt;span class="caps"&gt;KDC&lt;/span&gt; validates my card certificate against the &lt;span class="caps"&gt;CA&lt;/span&gt; certificate it already has, and it received its own &lt;span class="caps"&gt;PKINIT&lt;/span&gt; certificate during&amp;nbsp;installation.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A &lt;span class="caps"&gt;CA&lt;/span&gt; at home means a copy of the &lt;span class="caps"&gt;CA&lt;/span&gt; private key at home.&lt;/strong&gt; That is the single most valuable secret in the environment, and it already exists redundantly on two&amp;nbsp;servers.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;What does not work with the tunnel down:&lt;/strong&gt; issuing new certificates, and renewing the replica&amp;#8217;s own. The latter is due every two years, and certmonger keeps retrying until the tunnel is&amp;nbsp;back.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="dns-at-home-and-where-locations-stop-helping"&gt;&lt;span class="caps"&gt;DNS&lt;/span&gt; at Home, and Where Locations Stop&amp;nbsp;Helping&lt;/h2&gt;
&lt;p&gt;On the home OPNsense, Unbound now&amp;nbsp;forwards &lt;code&gt;ipa.example.com&lt;/code&gt; to three servers instead of one: the local replica over IPv6, and both data center servers. Previously, there was&amp;nbsp;only &lt;code&gt;idm1&lt;/code&gt;, which also means home &lt;span class="caps"&gt;DNS&lt;/span&gt; for the zone had been one server away from failing all&amp;nbsp;along.&lt;/p&gt;
&lt;p&gt;I assumed Unbound would settle on the fastest server, the local one, and every client at home would get the location-aware answer.&amp;nbsp;A &lt;code&gt;dig&lt;/code&gt; from my laptop seemed to confirm&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;It does not work that way. The location only shapes answers that come&amp;nbsp;from &lt;code&gt;idm-home&lt;/code&gt;. The data center servers have no location assigned and answer the same question&amp;nbsp;differently:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;== idm-home
_kerberos._udp.home._locations.ipa.example.com.
50 100 88 idm2.ipa.example.com.
50 100 88 idm1.ipa.example.com.
0 100 88 idm-home.ipa.example.com.
== idm1
0 100 88 idm2.ipa.example.com.
0 100 88 idm-home.ipa.example.com.
0 100 88 idm1.ipa.example.com.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;All three servers, all priority 0. And Unbound does not stick to the fastest upstream. It picks randomly among all servers whose round-trip time is within a band of the best one. At 25 ms through the tunnel versus 1 ms locally, all three are well inside that band. So roughly two out of three answers for my home clients came without any preference for the local replica. My&amp;nbsp;laptop&amp;#8217;s &lt;code&gt;dig&lt;/code&gt; had simply been&amp;nbsp;lucky.&lt;/p&gt;
&lt;p&gt;The obvious alternative, forwarding the zone only&amp;nbsp;to &lt;code&gt;idm-home&lt;/code&gt;, would make the location work every time and make all name resolution for the zone at home depend on a single &lt;span class="caps"&gt;VM&lt;/span&gt;. I would rather keep three forwarders for resilience and pin the preference on the clients that care. There are exactly three of&amp;nbsp;them.&lt;/p&gt;
&lt;h2 id="the-clients"&gt;The&amp;nbsp;Clients&lt;/h2&gt;
&lt;h3 id="the-laptop"&gt;The&amp;nbsp;Laptop&lt;/h3&gt;
&lt;p&gt;My Fedora laptop is not an IdM client. It uses a local &lt;span class="caps"&gt;SSSD&lt;/span&gt; setup for smartcard logins with &lt;span class="caps"&gt;PKINIT&lt;/span&gt;, and has the &lt;span class="caps"&gt;KDC&lt;/span&gt; configured explicitly. The replica goes in&amp;nbsp;front:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[realms]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;IPA.EXAMPLE.COM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;kdc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;idm-home.ipa.example.com:88&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;kdc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;172.16.1.12:88&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;...&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And&amp;nbsp;in &lt;code&gt;/etc/sssd/sssd.conf&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;krb5_server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;idm-home.ipa.example.com, 172.16.1.12&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On the road, the first name either does not resolve or does not answer, and the client moves on to the second entry after a short timeout. Password changes still go to the data center, which is where they&amp;nbsp;belong.&lt;/p&gt;
&lt;h3 id="jump01-the-freebsd-jump-host"&gt;jump01, the FreeBSD Jump&amp;nbsp;Host&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;jump01&lt;/code&gt; is integrated the way I described in &lt;a href="https://hofstede.it/integrating-freebsd-15-with-freeipa-native-kerberos-and-ldap-authentication/"&gt;Integrating FreeBSD 15 with FreeIPA&lt;/a&gt;: native&amp;nbsp;Kerberos, &lt;code&gt;nslcd&lt;/code&gt; for &lt;span class="caps"&gt;LDAP&lt;/span&gt;, no &lt;span class="caps"&gt;SSSD&lt;/span&gt;. Both&amp;nbsp;had &lt;code&gt;idm1&lt;/code&gt; hardcoded. Both get the replica first, and the second data center server as a third&amp;nbsp;option:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;/etc/krb5.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[libdefaults]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;default_realm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;IPA.EXAMPLE.COM&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;dns_lookup_kdc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;dns_lookup_realm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;false&lt;/span&gt;

&lt;span class="k"&gt;[realms]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;IPA.EXAMPLE.COM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;kdc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;idm-home.ipa.example.com&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;kdc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;idm1.ipa.example.com&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;kdc&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;idm2.ipa.example.com&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="na"&gt;admin_server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;idm1.ipa.example.com&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;/usr/local/etc/nslcd.conf&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;uri ldap://idm-home.ipa.example.com
uri ldap://idm1.ipa.example.com
uri ldap://idm2.ipa.example.com

base dc=ipa,dc=example,dc=com
sasl_mech GSSAPI
sasl_realm IPA.EXAMPLE.COM
map passwd loginShell &amp;quot;/bin/sh&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;nslcd&lt;/code&gt; works through the list from the top and fails over to the next one.&amp;nbsp;The &lt;code&gt;ldap/idm-home.ipa.example.com&lt;/code&gt; service principal was created during installation, so &lt;span class="caps"&gt;GSSAPI&lt;/span&gt; binds against the replica work without further&amp;nbsp;steps.&lt;/p&gt;
&lt;p&gt;My first check&amp;nbsp;was &lt;code&gt;getent passwd alice&lt;/code&gt;, which promptly returned an entry, and I promptly misread it. &lt;span class="caps"&gt;UID&lt;/span&gt; 1001 and a password hash: that is the local account&amp;nbsp;from &lt;code&gt;master.passwd&lt;/code&gt;, not &lt;span class="caps"&gt;LDAP&lt;/span&gt;. My own login&amp;nbsp;on &lt;code&gt;jump01&lt;/code&gt; is a local account with&amp;nbsp;a &lt;code&gt;.k5login&lt;/code&gt;, which only needs the host keytab and my ticket. &lt;span class="caps"&gt;LDAP&lt;/span&gt; is not involved at all. The check that actually&amp;nbsp;proves &lt;code&gt;nslcd&lt;/code&gt; works is an account that exists only in&amp;nbsp;IdM:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;alice&lt;/span&gt;&lt;span class="nv"&gt;@jump01&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;getent&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;passwd&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;alice&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;adm&lt;/span&gt;
&lt;span class="n"&gt;alice&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nl"&gt;adm&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1734400003&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;1734400003&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="n"&gt;Alice&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;Admin&lt;/span&gt;&lt;span class="err"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;home&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;alice&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nl"&gt;adm&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sh&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A &lt;span class="caps"&gt;UID&lt;/span&gt; from the IdM range. That one came from the&amp;nbsp;directory.&lt;/p&gt;
&lt;h3 id="app01-a-regular-idm-client"&gt;app01, a Regular IdM&amp;nbsp;Client&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;app01&lt;/code&gt; was enrolled&amp;nbsp;with &lt;code&gt;ipa-client-install&lt;/code&gt; and &lt;span class="caps"&gt;DNS&lt;/span&gt; discovery. Its &lt;span class="caps"&gt;SSSD&lt;/span&gt; config&amp;nbsp;read &lt;code&gt;ipa_server = _srv_, idm1.ipa.example.com&lt;/code&gt;, which looks like it should pick up the location automatically. After a&amp;nbsp;restart, &lt;code&gt;sssctl&lt;/code&gt; showed it had discovered all three servers and was still talking&amp;nbsp;to &lt;code&gt;idm1&lt;/code&gt;. That is the Unbound effect from above. The fix is the same as for the other two clients, with &lt;span class="caps"&gt;DNS&lt;/span&gt; discovery and the old server kept as&amp;nbsp;fallbacks:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;s|^ipa_server = _srv_, idm1.ipa.example.com|ipa_server = idm-home.ipa.example.com, _srv_, idm1.ipa.example.com|&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/etc/sssd/sssd.conf
systemctl&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;sssd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;root@app01 ~&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;sssctl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;domain&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipa&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;
&lt;span class="n"&gt;Online&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;status&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Online&lt;/span&gt;

&lt;span class="n"&gt;Active&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nl"&gt;servers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="nl"&gt;IPA&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idm&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;home&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ipa&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;com&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On an IdM client, &lt;span class="caps"&gt;SSSD&lt;/span&gt; writes the server it is connected to into&amp;nbsp;a &lt;code&gt;kdcinfo&lt;/code&gt; file, and the Kerberos library asks &lt;span class="caps"&gt;SSSD&lt;/span&gt;&amp;#8217;s locator plugin before it looks&amp;nbsp;at &lt;code&gt;krb5.conf&lt;/code&gt;. The&amp;nbsp;hardcoded &lt;code&gt;kdc&lt;/code&gt; line&amp;nbsp;in &lt;code&gt;app01&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s &lt;code&gt;krb5.conf&lt;/code&gt; only matters when &lt;span class="caps"&gt;SSSD&lt;/span&gt; is not&amp;nbsp;running.&lt;/p&gt;
&lt;h3 id="autostart"&gt;Autostart&lt;/h3&gt;
&lt;p&gt;The last detail&amp;nbsp;on &lt;code&gt;hv01&lt;/code&gt;: the replica has to come up after the&amp;nbsp;firewall.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sysrc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;vm_list&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;fw-home app01 idm-home&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A replica that does not start after a power cut is a replica that is missing precisely when the tunnel is likely to be down as&amp;nbsp;well.&lt;/p&gt;
&lt;h2 id="the-test"&gt;The&amp;nbsp;Test&lt;/h2&gt;
&lt;p&gt;All of this is theory until the tunnel is actually gone. So I switched it&amp;nbsp;off:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;❯&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tunnel&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;off&lt;/span&gt;

&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;❯&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ping&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;172.16.1.12&lt;/span&gt;
&lt;span class="n"&gt;PING&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;172.16.1.12&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mf"&gt;172.16.1.12&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;56&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;84&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;bytes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;data&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="o"&gt;^&lt;/span&gt;&lt;span class="n"&gt;C&lt;/span&gt;
&lt;span class="o"&gt;---&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;172.16.1.12&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ping&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;statistics&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;---&lt;/span&gt;
&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;packets&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;transmitted&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;received&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;packet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loss&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;time&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9239&lt;/span&gt;&lt;span class="n"&gt;ms&lt;/span&gt;

&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;9&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;❯&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kdestroy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;A&lt;/span&gt;

&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;❯&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;kinit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;X&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;X509_user_identity&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;PKCS11:token=MyEID (Basic PIN)&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;alice&lt;/span&gt;&lt;span class="nv"&gt;@IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;COM&lt;/span&gt;
&lt;span class="n"&gt;MyEID&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;Basic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PIN&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nl"&gt;PIN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;

&lt;span class="o"&gt;~&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;took&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;❯&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rm&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;~/&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ssh&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;controlmasters&lt;/span&gt;&lt;span class="cm"&gt;/*&lt;/span&gt;

&lt;span class="cm"&gt;~ took 2s ❯ ssh -J jump01 hv01&lt;/span&gt;
&lt;span class="cm"&gt;alice@hv01:~ $ id&lt;/span&gt;
&lt;span class="cm"&gt;uid=1001(alice) gid=1001(alice) groups=0(wheel),1001(alice)&lt;/span&gt;
&lt;span class="cm"&gt;alice@hv01:~ $ ^D&lt;/span&gt;
&lt;span class="cm"&gt;Shared connection to 192.168.99.1 closed.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The data center is unreachable. The smartcard login produces a fresh &lt;span class="caps"&gt;TGT&lt;/span&gt; in four seconds. Removing the &lt;span class="caps"&gt;SSH&lt;/span&gt; control sockets makes sure no existing connection carries the login. Then a jump&amp;nbsp;through &lt;code&gt;jump01&lt;/code&gt; to &lt;code&gt;hv01&lt;/code&gt;, authenticated purely with&amp;nbsp;Kerberos.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;KDC&lt;/span&gt; log on the replica shows where the ticket came from. A &lt;span class="caps"&gt;PKINIT&lt;/span&gt; exchange, first the expected request for pre-authentication, then the&amp;nbsp;ticket:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;krb5kdc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;18070&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="nx"&gt;info&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;AS_REQ&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;aa00&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;NEEDED_PREAUTH&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;alice&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;COM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;krbtgt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;COM&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;COM&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Additional&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;pre&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;authentication&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;required&lt;/span&gt;
&lt;span class="nx"&gt;krb5kdc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;18070&lt;/span&gt;&lt;span class="p"&gt;](&lt;/span&gt;&lt;span class="nx"&gt;info&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;AS_REQ&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;aa00&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="o"&gt;::&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ISSUE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;authtime&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1794680169&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;etypes&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="nx"&gt;rep&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;aes256&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;cts&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;hmac&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;sha1&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;96&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tkt&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;aes256&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;cts&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;hmac&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;sha384&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;192&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ses&lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="nx"&gt;aes256&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;cts&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;hmac&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;sha1&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;96&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;18&lt;/span&gt;&lt;span class="p"&gt;)},&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;alice&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;COM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;krbtgt&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;COM&lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;IPA&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;EXAMPLE&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;COM&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After switching the tunnel back on, the topology check on the data center&amp;nbsp;side:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="n"&gt;admin@idm1 ~&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="err"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ipa&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;topologysuffix&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;verify&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;domain&lt;/span&gt;
&lt;span class="o"&gt;====================================================&lt;/span&gt;
&lt;span class="k"&gt;Replication&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;topology&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;suffix&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;domain&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="ow"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;order&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;
&lt;span class="o"&gt;====================================================&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The silly idea from lunchtime worked by the&amp;nbsp;evening.&lt;/p&gt;
&lt;h2 id="caveats"&gt;Caveats&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The realm now lives at home.&lt;/strong&gt; That is the trade, and I made it knowingly. The &lt;span class="caps"&gt;VM&lt;/span&gt; sits in a &lt;span class="caps"&gt;DMZ&lt;/span&gt; that cannot open connections into my other networks, holds no &lt;span class="caps"&gt;CA&lt;/span&gt; key, and its host is reachable only through the same choke point as everything else. It is still a full copy of every key in the realm, on a hypervisor at home, in its &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots, and in its backups. Mine are encrypted, and that is not optional&amp;nbsp;anymore.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;bhyve is not a supported hypervisor for &lt;span class="caps"&gt;RHEL&lt;/span&gt;.&lt;/strong&gt; In a support case, Red Hat could ask you to reproduce the problem on a certified platform. For a home replica, I can live with that. Satellite shows the hardware model as &amp;#8220;&lt;span class="caps"&gt;BHYVE&lt;/span&gt;&amp;#8221; and gets a few facts wrong in its overview card, which is&amp;nbsp;cosmetic.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Locations are not magic.&lt;/strong&gt; They only shape answers from servers in that location. If your clients reach the zone through a forwarder that spreads queries across several IdM servers, the preference applies to a fraction of answers. Pin it on the clients that matter, or forward only to the local server and accept that&amp;nbsp;dependency.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No &lt;span class="caps"&gt;CA&lt;/span&gt; at home&lt;/strong&gt; means no certificate issuance while the tunnel is down. For me, that is a&amp;nbsp;feature.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Keep an eye on&amp;nbsp;replication.&lt;/strong&gt; &lt;code&gt;ipa topologysuffix-verify domain&lt;/code&gt; checks the topology. For actual replication health over the tunnel, I will be watching the replica for a while before I stop thinking about&amp;nbsp;it.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The IdM part of this project was the easy&amp;nbsp;part. &lt;code&gt;ipa-replica-install&lt;/code&gt; did exactly what the documentation says, the installer&amp;#8217;s own connection check caught nothing because there was nothing left to catch, and the initial replication took 26&amp;nbsp;seconds.&lt;/p&gt;
&lt;p&gt;Almost all of the work was networking. A tunnel designed years ago for one direction. A &lt;span class="caps"&gt;NAT&lt;/span&gt; rule I had to look up to find out what it actually matched. A &lt;span class="caps"&gt;DMZ&lt;/span&gt; that exists only as a bridge without members. A missing &lt;span class="caps"&gt;UDP&lt;/span&gt; rule. An Unbound that was not listening. And one assumption about &lt;span class="caps"&gt;DNS&lt;/span&gt; locations that turned out to be true for the replica and not for the forwarder in front of&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;What I would keep from this is the order of operations. Prove both directions on a real port, with real source addresses, before any identity software touches the machine. Every network problem I found was found there, where it was cheap. The replica installation itself was an anticlimax, and an anticlimax is exactly what you want from the system that holds every key you&amp;nbsp;have.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html-single/installing_identity_management/index"&gt;Installing Identity Management (&lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;10)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/managing_replication_in_identity_management/managing-replication-topology"&gt;Managing replication topology (&lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;10)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/working_with_dns_in_identity_management/index"&gt;Working with &lt;span class="caps"&gt;DNS&lt;/span&gt; in Identity Management (&lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;10)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freeipa.org/page/V4/DNS_Location_Mechanism"&gt;FreeIPA: &lt;span class="caps"&gt;DNS&lt;/span&gt;&amp;nbsp;Locations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/churchers/vm-bhyve"&gt;vm-bhyve&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/integrating-freebsd-15-with-freeipa-native-kerberos-and-ldap-authentication/"&gt;Integrating FreeBSD 15 with FreeIPA: Native Kerberos and &lt;span class="caps"&gt;LDAP&lt;/span&gt;&amp;nbsp;Authentication&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="freeipa"/><category term="idm"/><category term="kerberos"/><category term="rhel"/><category term="bhyve"/><category term="freebsd"/><category term="opnsense"/><category term="wireguard"/><category term="ipv6"/><category term="dns"/><category term="homelab"/></entry><entry><title>A Read-Only AI Reviewer for Forgejo Pull Requests</title><link href="https://hofstede.it/a-read-only-ai-reviewer-for-forgejo-pull-requests/" rel="alternate"/><published>2026-11-08T00:00:00+01:00</published><updated>2026-11-08T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-11-08:/a-read-only-ai-reviewer-for-forgejo-pull-requests/</id><summary type="html">&lt;p&gt;A small Python service that reviews every pull request on my self-hosted Forgejo with an &lt;span class="caps"&gt;LLM&lt;/span&gt;, posts a summary and inline findings, and is not allowed to do anything else. The interesting part was not getting a model to comment on a diff. It was getting it to stop being confidently wrong about code it had never seen, and making sure a misbehaving or manipulated bot could not change a single&amp;nbsp;line.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A small rust-colored robot holding a pen sits on a box with a screen full of green checkmarks, wired to a server rack, next to a glass box labeled confirmed and refuted, a locked gate, and a small screen with a Git branch and an anvil" src="https://hofstede.it/images/2026-11-08-forgejo-pr-review-bot.webp" title="Reads everything, comments on some of it, merges nothing."&gt;&lt;/p&gt;
&lt;p&gt;Most of my repositories have exactly one contributor. That contributor also reviews the pull requests, approves them, and merges them, usually within the same minute. It is a very efficient process. It is not a very good&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;GitHub users have had &lt;span class="caps"&gt;AI&lt;/span&gt; review bots for a while now. My code lives on a self-hosted Forgejo instance, and I was not interested in giving a SaaS product a token for it. So I built my own: a Forgejo &lt;span class="caps"&gt;PR&lt;/span&gt; review bot that reads every new pull request, asks an &lt;span class="caps"&gt;LLM&lt;/span&gt; for a review, and posts the result as ordinary&amp;nbsp;comments.&lt;/p&gt;
&lt;p&gt;The first version took an afternoon. Making it useful took three more rounds, and nearly all of that work went into a single problem: the model was very confident about code it had never&amp;nbsp;seen.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-it-does"&gt;What It&amp;nbsp;Does&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#architecture"&gt;Architecture&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#line-numbers-the-model-can-actually-use"&gt;Line Numbers the Model Can Actually&amp;nbsp;Use&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#chunking-and-cost-guards"&gt;Chunking and Cost&amp;nbsp;Guards&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#reasoning-models-and-the-empty-answer"&gt;Reasoning Models and the Empty&amp;nbsp;Answer&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-real-problem-confident-guesses"&gt;The Real Problem: Confident Guesses&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-verification-pass"&gt;The Verification&amp;nbsp;Pass&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#precision-is-not-the-whole-story"&gt;Precision Is Not the Whole&amp;nbsp;Story&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#least-privilege-enforced-by-forgejo"&gt;Least Privilege, Enforced by Forgejo&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#prompt-injection"&gt;Prompt&amp;nbsp;Injection&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#not-reachable-from-the-internet"&gt;Not Reachable From the&amp;nbsp;Internet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-quadlet"&gt;The&amp;nbsp;Quadlet&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-real-review"&gt;A Real Review&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-catch-that-justified-the-whole-thing"&gt;The Catch That Justified the Whole&amp;nbsp;Thing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-correction-that-worked"&gt;A Correction That&amp;nbsp;Worked&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-correction-that-did-not"&gt;A Correction That Did&amp;nbsp;Not&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#knowing-what-is-already-known"&gt;Knowing What Is Already&amp;nbsp;Known&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-i-did-with-it"&gt;What I Did With&amp;nbsp;It&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#dry-runs"&gt;Dry&amp;nbsp;Runs&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#limitations"&gt;Limitations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="what-it-does"&gt;What It&amp;nbsp;Does&lt;/h2&gt;
&lt;p&gt;When a pull request is opened, Forgejo sends a webhook. The bot fetches the diff, sends it to a model through &lt;a href="https://openrouter.ai"&gt;OpenRouter&lt;/a&gt;, and posts two&amp;nbsp;things:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;A summary comment&lt;/strong&gt; that describes what the pull request changes, lists the reviewed files, and shows the model, token count, and cost of the review in a&amp;nbsp;footer.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A review&lt;/strong&gt; with &lt;strong&gt;inline comments&lt;/strong&gt; on the affected lines: bugs, risks, oversights, and optimizations, each with a severity and, where it makes sense, a suggested change. Findings that cannot be tied to a visible line go into the review&amp;nbsp;body.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;That is the entire feature set. The bot never pushes, never merges, never approves, and never requests changes. It is an advisor, not a gatekeeper. Every review it posts&amp;nbsp;uses &lt;code&gt;event: COMMENT&lt;/code&gt;, so it cannot block a merge, and it cannot unblock one&amp;nbsp;either.&lt;/p&gt;
&lt;p&gt;The triggers are deliberately&amp;nbsp;boring:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Event&lt;/th&gt;
&lt;th&gt;Reviewed?&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;PR&lt;/span&gt; opened or reopened&lt;/td&gt;
&lt;td&gt;yes, once per head commit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;New commits pushed&lt;/td&gt;
&lt;td&gt;only&amp;nbsp;if &lt;code&gt;REVIEW_ON_SYNC=true&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Review requested from the bot user&lt;/td&gt;
&lt;td&gt;yes, always&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Draft, &lt;code&gt;WIP:&lt;/code&gt; title,&amp;nbsp;or &lt;code&gt;no-ai-review&lt;/code&gt; label&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;PR&lt;/span&gt; authored by the bot&lt;/td&gt;
&lt;td&gt;no&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The &amp;#8220;review requested&amp;#8221; trigger turned out to be the most useful one. Forgejo already has a button for asking a specific user to review a pull request. Making the bot a collaborator puts it in that list, so &amp;#8220;review this again&amp;#8221; needs no extra &lt;span class="caps"&gt;UI&lt;/span&gt;. I just click the button I would click for a&amp;nbsp;human.&lt;/p&gt;
&lt;h2 id="architecture"&gt;Architecture&lt;/h2&gt;
&lt;p&gt;The service is roughly 1,700 lines of Python with three runtime dependencies: FastAPI, uvicorn, and httpx. No database, no queue, no &lt;span class="caps"&gt;ORM&lt;/span&gt;. The only state is a small in-memory cache that remembers which head commit was already&amp;nbsp;reviewed.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;PR opened ──webhook──▶ review-bot:8080/webhook   (internal network only)
                          │  verify HMAC signature, answer 202 right away
                          ├─ GET  /repos/{o}/{r}/pulls/{n}             (metadata)
                          ├─ GET  /repos/{o}/{r}/pulls/{n}.diff        (diff)
                          ├─ POST openrouter.ai/api/v1/chat/completions
                          ├─ GET  /repos/{o}/{r}/archive/{sha}.tar.gz  (verification)
                          ├─ POST /repos/{o}/{r}/issues/{n}/comments   (summary)
                          └─ POST /repos/{o}/{r}/pulls/{n}/reviews     (findings)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The webhook handler&amp;nbsp;returns &lt;code&gt;202&lt;/code&gt; immediately and runs the review as a background task. Forgejo has a short delivery timeout, and a review takes anywhere from one to several minutes. A semaphore limits parallel reviews, and a set of in-flight PRs stops two reviews of the same &lt;span class="caps"&gt;PR&lt;/span&gt; from racing each&amp;nbsp;other.&lt;/p&gt;
&lt;h3 id="line-numbers-the-model-can-actually-use"&gt;Line Numbers the Model Can Actually&amp;nbsp;Use&lt;/h3&gt;
&lt;p&gt;The first thing that went wrong was predictable: the model referred to lines that did not exist. A unified diff only carries line numbers in its hunk headers, and LLMs are not great at counting. So the bot parses the diff and renders every line with its new-file line number before the model sees&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;### File: app/main.py (modified)&lt;/span&gt;
&lt;span class="o"&gt;@@&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;@@&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;os&lt;/span&gt;
&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;sys&lt;/span&gt;
&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kn"&gt;import&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;sys&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nn"&gt;json&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The model now answers with&amp;nbsp;a &lt;code&gt;line&lt;/code&gt; value it can read rather than calculate, and the bot can check it. The parser knows which lines Forgejo actually displays in the &lt;span class="caps"&gt;PR&lt;/span&gt; diff. If a finding points at a line within three lines of a visible one, it snaps there. Otherwise it goes into the review body. And if Forgejo rejects the inline placement anyway, the bot posts the same review again with everything in the body. Nothing the model found gets lost because of a coordinate&amp;nbsp;mismatch.&lt;/p&gt;
&lt;h3 id="chunking-and-cost-guards"&gt;Chunking and Cost&amp;nbsp;Guards&lt;/h3&gt;
&lt;p&gt;Diffs are split into chunks of at most 50,000 characters, roughly 12,000 to 15,000 tokens. My first version used 120,000. Smaller chunks cost a few more requests, but each part gets a noticeably more careful read. Every chunk also receives the full file list, so the model knows what else changed. When there is more than one chunk, a final call merges the partial summaries into&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;Pull requests larger than 600,000 characters are not reviewed at all. The bot posts a short comment explaining why and suggesting that the &lt;span class="caps"&gt;PR&lt;/span&gt; be split. A model reviewing a 600k diff is not a review. It is an expensive&amp;nbsp;skim.&lt;/p&gt;
&lt;p&gt;The hard limit, however, sits outside the code. The OpenRouter key has a weekly spending limit configured on the OpenRouter side. When it is exhausted, OpenRouter answers with &lt;span class="caps"&gt;HTTP&lt;/span&gt; 402, the bot aborts the review, logs the reason, and posts nothing. I prefer a budget enforced by the party that sends the&amp;nbsp;invoice.&lt;/p&gt;
&lt;p&gt;What a review actually costs shows up in every summary footer and in the&amp;nbsp;log:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;LLM reply: provider=DigitalOcean finish=stop prompt=37718 completion=22737 reasoning=20195 chars=9794
Reviewed chris/&amp;lt;repo&amp;gt;#1: 8 findings (8 inline, 0 general, 8 filtered), usage={... &amp;#39;cost&amp;#39;: 0.0404}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Four cents for eight findings, on one of the very first reviews. The current setup runs on Claude Haiku 5.5, and the largest real review so far, a 44-file pull request with the full verification pass, came to &lt;strong&gt;$0.09&lt;/strong&gt;. More on that one&amp;nbsp;below.&lt;/p&gt;
&lt;h3 id="reasoning-models-and-the-empty-answer"&gt;Reasoning Models and the Empty&amp;nbsp;Answer&lt;/h3&gt;
&lt;p&gt;One detail cost me an evening. Reasoning models like DeepSeek V4.1 Flash will happily think until they&amp;nbsp;hit &lt;code&gt;MAX_OUTPUT_TOKENS&lt;/code&gt;, and then return an empty answer&amp;nbsp;with &lt;code&gt;finish_reason=length&lt;/code&gt;. With a 32k token limit, this happened regularly.&amp;nbsp;Setting &lt;code&gt;REASONING_EFFORT=low&lt;/code&gt;, which the client sends&amp;nbsp;as &lt;code&gt;reasoning.effort&lt;/code&gt;, fixed it. The verdicts did not get worse. The model simply stopped writing essays to&amp;nbsp;itself.&lt;/p&gt;
&lt;p&gt;The client is defensive about replies in general. It requests &lt;span class="caps"&gt;JSON&lt;/span&gt; mode, extracts &lt;span class="caps"&gt;JSON&lt;/span&gt; from plain text or fenced code blocks if the model ignores that, and retries once on an empty or invalid answer. If a chunk still fails, it is split in half and each half retried, recursively down to single files. Files that fail even on their own are listed in an &lt;strong&gt;&amp;#8220;Incomplete review&amp;#8221;&lt;/strong&gt; warning at the top of both the summary and the review. A partial review that says it is partial is fine. A partial review that pretends to be complete is worse than&amp;nbsp;none.&lt;/p&gt;
&lt;h2 id="the-real-problem-confident-guesses"&gt;The Real Problem: Confident&amp;nbsp;Guesses&lt;/h2&gt;
&lt;p&gt;The first real review produced a handful of findings that looked excellent. Clear title, a plausible explanation, a severity, a suggested fix. Several of them were simply&amp;nbsp;wrong.&lt;/p&gt;
&lt;p&gt;When I went through them, every wrong finding had the same root cause. &lt;strong&gt;The model was guessing how code outside the diff behaves.&lt;/strong&gt; A helper it never read, which already handled the edge case. A lock taken in the caller. A validation check that had moved to another file in the same &lt;span class="caps"&gt;PR&lt;/span&gt;. A test that already covered exactly the scenario it was worried&amp;nbsp;about.&lt;/p&gt;
&lt;p&gt;That is not a model defect, really. A human reviewer who only sees the diff makes the same assumptions. The difference is that a human reviewer opens the file and checks. A model given only a diff&amp;nbsp;cannot.&lt;/p&gt;
&lt;p&gt;So I gave it the means to&amp;nbsp;check.&lt;/p&gt;
&lt;h3 id="the-verification-pass"&gt;The Verification&amp;nbsp;Pass&lt;/h3&gt;
&lt;p&gt;Before anything is posted, every finding at or above a configurable severity goes through a second model call that can look at the actual&amp;nbsp;repository:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Snapshot.&lt;/strong&gt; The bot downloads the &lt;span class="caps"&gt;PR&lt;/span&gt; head as&amp;nbsp;a &lt;code&gt;tar.gz&lt;/code&gt; archive through the Forgejo &lt;span class="caps"&gt;API&lt;/span&gt;, streamed, and unpacks the text files &lt;strong&gt;into memory&lt;/strong&gt;. Nothing touches the disk. Nothing is checked out. Nothing is&amp;nbsp;executed.&lt;/p&gt;
&lt;p&gt;The limits deserve precision, since&amp;nbsp;a &lt;code&gt;tar.gz&lt;/code&gt; is a classic way to turn a small download into a large amount of memory. The 100 &lt;span class="caps"&gt;MB&lt;/span&gt; cap applies to the &lt;strong&gt;compressed&lt;/strong&gt; download, which is aborted once it gets bigger. Each file in the archive is checked against a 512 &lt;span class="caps"&gt;KB&lt;/span&gt; limit using the size in its tar header, before its content is read. Anything with a null byte in the first 8 &lt;span class="caps"&gt;KB&lt;/span&gt; is treated as binary and skipped. What is missing is a cap on the &lt;strong&gt;total&lt;/strong&gt; unpacked size. An archive full of highly compressible text files, each just under 512 &lt;span class="caps"&gt;KB&lt;/span&gt;, could still unpack into several gigabytes. On my single-user instance, the only person who can push such a branch is me, so I accepted this for now. A running total that aborts the snapshot is the obvious fix, and it is on the list.
2. &lt;strong&gt;Tools.&lt;/strong&gt; The verifier gets the finding, the diff of the affected file, and three read-only&amp;nbsp;tools:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;Does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;read_file(path, start_line, end_line)&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;numbered lines, at most 300 per call&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;grep(pattern, path_glob)&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;regex search across the snapshot, at most 60 hits&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;list_dir(path)&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;directory listing&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;/li&gt;
&lt;li&gt;
&lt;p&gt;&lt;strong&gt;Budget.&lt;/strong&gt; After eight tool calls, the tools are withdrawn and a verdict is demanded. Four findings are verified in&amp;nbsp;parallel.&lt;/p&gt;
&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Verdict.&lt;/strong&gt; The verifier returns &lt;span class="caps"&gt;JSON&lt;/span&gt; with one of four verdicts and&amp;nbsp;concrete &lt;code&gt;file:line&lt;/code&gt; evidence.&lt;/li&gt;
&lt;/ol&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Verdict&lt;/th&gt;
&lt;th&gt;What happens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;confirmed&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;posted inline, marked &amp;#8220;checked against the code&amp;#8221;, evidence attached, severity possibly re-rated&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;corrected&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;the claim was wrong, but a closely related real defect exists in the same code, so the finding is rewritten and posted&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;refuted&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;moved to a collapsed &amp;#8220;discarded after checking the code&amp;#8221; list, with the reason&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;uncertain&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;moved to a collapsed &amp;#8220;needs verification&amp;#8221; section, never posted inline&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Refuted findings are not deleted. They stay visible, collapsed, with the verifier&amp;#8217;s reason. The verifier can be wrong too, and a human should be able to overrule it. If a refuted claim had made it into the summary, one more call rewrites the summary without&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;With verification enabled, only checked findings are ever posted inline.&lt;/strong&gt; If a &lt;span class="caps"&gt;PR&lt;/span&gt; produces more findings than the verifier is allowed to check, the rest go to &amp;#8220;needs verification&amp;#8221; rather than slipping through&amp;nbsp;unchecked.&lt;/p&gt;
&lt;h3 id="precision-is-not-the-whole-story"&gt;Precision Is Not the Whole&amp;nbsp;Story&lt;/h3&gt;
&lt;p&gt;The third round posted no wrong findings at all. I was pleased for about ten minutes. Then I noticed it had also missed real bugs, which a zero false-positive rate is very good at&amp;nbsp;hiding.&lt;/p&gt;
&lt;p&gt;One case was particularly instructive. The first pass had pointed at a real bug, but described the wrong scenario. The verifier checked the scenario as stated, correctly found that it could not happen, and refuted it. The bug was right there in the same function, and the strict yes-or-no check threw it&amp;nbsp;away.&lt;/p&gt;
&lt;p&gt;Hence the fourth&amp;nbsp;verdict, &lt;code&gt;corrected&lt;/code&gt;. If the claim is wrong but the verifier finds a closely related real defect in the same code while checking it, it may describe that defect instead. The prompt is explicit that this is only for defects it can show in the code, not for new speculation. The original title is shown alongside, so it stays transparent what was claimed and what was&amp;nbsp;found.&lt;/p&gt;
&lt;p&gt;The second change was to tune the first pass for &lt;strong&gt;recall&lt;/strong&gt; instead of precision. Since every finding is verified anyway, the first pass is now asked to report plausible problems rather than self-censor, to set honest confidence levels, and to describe the exact triggering scenario. A finding like &amp;#8220;this might have a race condition&amp;#8221; cannot be checked. A finding like &amp;#8220;two requests renew the same token at once, both read the old row at line 40 before either writes, and the second renewal overwrites the first&amp;#8221;&amp;nbsp;can.&lt;/p&gt;
&lt;p&gt;In short: one model call casts a wide net, a second one with access to the code throws back what does not hold up. That split worked much better than trying to make a single pass both thorough and&amp;nbsp;careful.&lt;/p&gt;
&lt;h2 id="least-privilege-enforced-by-forgejo"&gt;Least Privilege, Enforced by&amp;nbsp;Forgejo&lt;/h2&gt;
&lt;p&gt;The central security idea is simple: &lt;strong&gt;the bot&amp;#8217;s restrictions are enforced by Forgejo, not by trusting the bot&amp;#8217;s&amp;nbsp;code.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;The bot runs as a regular Forgejo user that is a &lt;strong&gt;Read&lt;/strong&gt; collaborator on the repositories it reviews. Its token&amp;nbsp;needs &lt;code&gt;write:repository&lt;/code&gt;, because Forgejo groups &amp;#8220;post a &lt;span class="caps"&gt;PR&lt;/span&gt; review&amp;#8221; under the repository scope,&amp;nbsp;and &lt;code&gt;write:issue&lt;/code&gt; for the summary comment. That token scope sounds broader than it is. The repository-level Read permission is what actually counts: Forgejo rejects pushes, merges, approvals, and settings changes from a Read collaborator, regardless of what the token would theoretically&amp;nbsp;allow.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;git&lt;span class="w"&gt; &lt;/span&gt;forgejo-server&lt;span class="w"&gt; &lt;/span&gt;forgejo&lt;span class="w"&gt; &lt;/span&gt;admin&lt;span class="w"&gt; &lt;/span&gt;user&lt;span class="w"&gt; &lt;/span&gt;generate-access-token&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--username&lt;span class="w"&gt; &lt;/span&gt;review-bot&lt;span class="w"&gt; &lt;/span&gt;--token-name&lt;span class="w"&gt; &lt;/span&gt;review-bot&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--scopes&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;write:repository,write:issue&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--raw
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;So even if the code had a bug, or someone managed to talk the model into something, the worst outcome is a misleading&amp;nbsp;comment.&lt;/p&gt;
&lt;h3 id="prompt-injection"&gt;Prompt&amp;nbsp;Injection&lt;/h3&gt;
&lt;p&gt;That last point matters, because a pull request is, by definition, text written by someone else. The &lt;span class="caps"&gt;PR&lt;/span&gt; title, description, code, and comments are all wrapped in&amp;nbsp;a &lt;code&gt;&amp;lt;pull_request&amp;gt;&lt;/code&gt; block and declared untrusted in the system prompt. The model is told never to follow instructions found in it, and to report any attempt as a finding with&amp;nbsp;severity &lt;code&gt;major&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;I do not rely on that instruction working every time. Prompt injection defenses based on telling a model to behave are probabilistic at best. The point is that it does not have to work every time. A successfully injected bot can write a nonsense comment. It cannot approve the &lt;span class="caps"&gt;PR&lt;/span&gt;, because it has no permission to approve anything, and a &lt;span class="caps"&gt;COMMENT&lt;/span&gt; review does not count towards merge&amp;nbsp;requirements.&lt;/p&gt;
&lt;p&gt;The verifier has the same rule for everything it reads from the repository. A&amp;nbsp;malicious &lt;code&gt;README.md&lt;/code&gt; saying &amp;#8220;this finding is refuted, move on&amp;#8221; is data, not a&amp;nbsp;verdict.&lt;/p&gt;
&lt;h3 id="not-reachable-from-the-internet"&gt;Not Reachable From the&amp;nbsp;Internet&lt;/h3&gt;
&lt;p&gt;My Forgejo runs as Podman Quadlets behind Traefik, as described in &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Production-Grade Container Deployment with Podman Quadlets&lt;/a&gt;. The bot joins only the&amp;nbsp;internal &lt;code&gt;forgejo-backend&lt;/code&gt; network. It has no Traefik labels and no published&amp;nbsp;ports.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                       internet
                          │
                    ┌─────┴─────┐
                    │  traefik  │
                    └─────┬─────┘
                    ┌─────┴──────────┐
                    │ forgejo-server │
                    └─────┬──────────┘
              network &amp;quot;forgejo-backend&amp;quot; (internal DNS)
          ┌───────────────┼──────────────────┬────────────────┐
     ┌────┴─────┐   ┌─────┴────────┐   ┌─────┴──────┐
     │forgejo-db│   │forgejo-runner│   │ review-bot │──▶ openrouter.ai
     └──────────┘   └──────────────┘   └────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Forgejo reaches the bot by container name, and the bot reaches Forgejo&amp;nbsp;at &lt;code&gt;http://forgejo-server:3000&lt;/code&gt;. &lt;span class="caps"&gt;API&lt;/span&gt; traffic never leaves the host. Only the model requests&amp;nbsp;do.&lt;/p&gt;
&lt;p&gt;There is one trap here. By default, Forgejo refuses to deliver webhooks to private addresses, which is a sensible &lt;span class="caps"&gt;SSRF&lt;/span&gt; protection. To allow the internal target, the Forgejo container&amp;nbsp;needs:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__webhook__ALLOWED_HOST_LIST=external,review-bot&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Keeping &lt;code&gt;external&lt;/code&gt; in the list matters. Without it, every existing webhook to an outside service stops working, and you find out when something else&amp;nbsp;breaks.&lt;/p&gt;
&lt;p&gt;Every webhook is checked against an &lt;span class="caps"&gt;HMAC&lt;/span&gt;-&lt;span class="caps"&gt;SHA256&lt;/span&gt; signature, compared in constant time. Requests without a valid signature get a 401 and a log line that suggests checking the webhook secret, because in practice that is always the&amp;nbsp;reason.&lt;/p&gt;
&lt;h3 id="the-quadlet"&gt;The&amp;nbsp;Quadlet&lt;/h3&gt;
&lt;p&gt;The container itself gets the usual&amp;nbsp;hardening:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;review-bot&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry.example.com/review-bot:latest&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-backend.network&lt;/span&gt;

&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO_URL=http://forgejo-server:3000&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;OPENROUTER_MODEL=anthropic/claude-haiku-5.5&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;REASONING_EFFORT=low&lt;/span&gt;

&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;review-bot-forgejo-token,type=mount,target=/run/secrets/forgejo_token,uid=10001,mode=0400&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;review-bot-webhook-secret,type=mount,target=/run/secrets/webhook_secret,uid=10001,mode=0400&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;review-bot-openrouter-key,type=mount,target=/run/secrets/openrouter_key,uid=10001,mode=0400&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO_TOKEN_FILE=/run/secrets/forgejo_token&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;WEBHOOK_SECRET_FILE=/run/secrets/webhook_secret&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;OPENROUTER_API_KEY_FILE=/run/secrets/openrouter_key&lt;/span&gt;

&lt;span class="na"&gt;NoNewPrivileges&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;DropCapability&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;ALL&lt;/span&gt;
&lt;span class="na"&gt;ReadOnly&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Non-root user, read-only root filesystem, no capabilities, and secrets mounted as files readable only by the service&amp;nbsp;user. &lt;code&gt;ReadOnly=true&lt;/code&gt; does not cause trouble for Python here.&amp;nbsp;Podman&amp;#8217;s &lt;code&gt;--read-only&lt;/code&gt; mounts a tmpfs&amp;nbsp;on &lt;code&gt;/tmp&lt;/code&gt;, &lt;code&gt;/var/tmp&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;/run&lt;/code&gt; by default, and when Python cannot write a bytecode cache file, it simply skips it instead of failing. So the quadlet runs as shown, without an&amp;nbsp;extra &lt;code&gt;Tmpfs=&lt;/code&gt; line&amp;nbsp;or &lt;code&gt;PYTHONDONTWRITEBYTECODE&lt;/code&gt;. Every secret can be passed&amp;nbsp;as &lt;code&gt;NAME_FILE=/path&lt;/code&gt;, so none of them appear&amp;nbsp;in &lt;code&gt;podman inspect&lt;/code&gt; output. The usual caveat about Podman&amp;#8217;s default secret driver applies: the secrets are protected by filesystem permissions, not&amp;nbsp;encryption.&lt;/p&gt;
&lt;p&gt;The server is aarch64, so the image is built for both architectures from my x86_64&amp;nbsp;workstation:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;--platform&lt;span class="w"&gt; &lt;/span&gt;linux/arm64,linux/amd64&lt;span class="w"&gt; &lt;/span&gt;--manifest&lt;span class="w"&gt; &lt;/span&gt;registry.example.com/review-bot:latest&lt;span class="w"&gt; &lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With &lt;code&gt;AutoUpdate=registry&lt;/code&gt;, &lt;code&gt;podman auto-update&lt;/code&gt; pulls a new image and restarts the unit. No &lt;span class="caps"&gt;CI&lt;/span&gt; pipeline yet, which is on the&amp;nbsp;list.&lt;/p&gt;
&lt;h2 id="a-real-review"&gt;A Real&amp;nbsp;Review&lt;/h2&gt;
&lt;p&gt;Design documents are patient. So here is what the bot actually did on two pull requests in one of my Go side projects, a business application platform with its own &lt;span class="caps"&gt;DSL&lt;/span&gt; and a time-tracking module. Both PRs were written with Claude Code, which makes this an &lt;span class="caps"&gt;AI&lt;/span&gt; reviewing &lt;span class="caps"&gt;AI&lt;/span&gt;-written code, with me in the middle deciding what to do about&amp;nbsp;it.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;&lt;span class="caps"&gt;PR&lt;/span&gt; A&lt;/th&gt;
&lt;th&gt;&lt;span class="caps"&gt;PR&lt;/span&gt; B&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;What&lt;/td&gt;
&lt;td&gt;audit fixes plus a time-management feature&lt;/td&gt;
&lt;td&gt;fixes for the confirmed findings from &lt;span class="caps"&gt;PR&lt;/span&gt; A&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Size&lt;/td&gt;
&lt;td&gt;44 files, +3,046 / -236&lt;/td&gt;
&lt;td&gt;8 files, +137 / -21&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Findings checked against the code&lt;/td&gt;
&lt;td&gt;29&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Confirmed (including corrected)&lt;/td&gt;
&lt;td&gt;11 (4 corrected)&lt;/td&gt;
&lt;td&gt;0&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Discarded as wrong&lt;/td&gt;
&lt;td&gt;18&lt;/td&gt;
&lt;td&gt;3&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Posted inline&lt;/td&gt;
&lt;td&gt;2 major, 9 minor&lt;/td&gt;
&lt;td&gt;none&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Tokens&lt;/td&gt;
&lt;td&gt;697,246&lt;/td&gt;
&lt;td&gt;53,221&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Cost&lt;/td&gt;
&lt;td&gt;$0.0923&lt;/td&gt;
&lt;td&gt;$0.0080&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Look at the &lt;span class="caps"&gt;PR&lt;/span&gt; A column once more: &lt;strong&gt;18 of 29 findings were thrown away&lt;/strong&gt; after checking the code. Without the verification pass, all of them would have landed in the pull request as confident inline comments. That is the difference between a reviewer I read and one I learn to&amp;nbsp;ignore.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;PR&lt;/span&gt; A was large and mixed: kernel hardening, documentation, and a new feature in one branch, including a commit that had been sitting unpushed&amp;nbsp;on &lt;code&gt;master&lt;/code&gt;. The summary said exactly that in its first paragraph, which is the kind of thing I stop noticing in my own&amp;nbsp;PRs.&lt;/p&gt;
&lt;h3 id="the-catch-that-justified-the-whole-thing"&gt;The Catch That Justified the Whole&amp;nbsp;Thing&lt;/h3&gt;
&lt;p&gt;The most valuable finding was a &lt;strong&gt;major&lt;/strong&gt; one in the rewritten timesheet &lt;span class="caps"&gt;UI&lt;/span&gt;. Its title: &lt;em&gt;&amp;#8220;Edits made during a save are discarded when the save&amp;nbsp;succeeds.&amp;#8221;&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;The save function took a snapshot of each buffered edit and sent the requests one after another, awaiting each. If I changed the same cell while its request was in flight, the newer value went into the edits map. Then the request succeeded, and the success handler deleted the map entry for that cell. The newer edit with it. The &lt;span class="caps"&gt;UI&lt;/span&gt; reloaded the server value, and what I had typed was silently&amp;nbsp;gone.&lt;/p&gt;
&lt;p&gt;The verifier&amp;#8217;s evidence was precise: the&amp;nbsp;unconditional &lt;code&gt;edits.delete()&lt;/code&gt; in the save loop, the fact that only the Save button was disabled during a save, and the input handler that kept accepting typing and replacing the map entry. Three line references, one sequence of events, one real bug. No test would have caught it, because no test types while a save is running. A user would have, eventually, on a Friday, with eight hours of timesheet to&amp;nbsp;redo.&lt;/p&gt;
&lt;p&gt;This is exactly the category the first-pass prompt calls out explicitly: &amp;#8220;user input that arrives while a request is in flight&amp;#8221;. Telling a model where to look hardest pays&amp;nbsp;off.&lt;/p&gt;
&lt;h3 id="a-correction-that-worked"&gt;A Correction That&amp;nbsp;Worked&lt;/h3&gt;
&lt;p&gt;The first pass reported that a lint&amp;nbsp;check, &lt;code&gt;actionGuarded&lt;/code&gt;, &amp;#8220;accepts any condition mentioning the literal &lt;span class="caps"&gt;ACTION&lt;/span&gt;&amp;#8221;. It backed that up with an example&amp;nbsp;condition.&lt;/p&gt;
&lt;p&gt;The verifier read the code and came back&amp;nbsp;with &lt;code&gt;corrected&lt;/code&gt;. The example was wrong: that particular condition really is a valid guard, and flagging it would have been a false positive. But the underlying suspicion was right. The function only checked that the&amp;nbsp;string &lt;code&gt;"ACTION"&lt;/code&gt; appears somewhere in the condition, never that the condition is actually true for an action.&amp;nbsp;So &lt;code&gt;if $operation != "ACTION" { return; }&lt;/code&gt;, which does the exact opposite of guarding, silenced the warning just as&amp;nbsp;well.&lt;/p&gt;
&lt;p&gt;A strict yes-or-no verifier would have refuted the finding and dropped the bug.&amp;nbsp;With &lt;code&gt;corrected&lt;/code&gt;, it was fixed in &lt;span class="caps"&gt;PR&lt;/span&gt; B, along with a new test covering eight different guard&amp;nbsp;shapes.&lt;/p&gt;
&lt;h3 id="a-correction-that-did-not"&gt;A Correction That Did&amp;nbsp;Not&lt;/h3&gt;
&lt;p&gt;Here is the one that went wrong, and it is the most&amp;nbsp;instructive.&lt;/p&gt;
&lt;p&gt;The first pass worried that a new&amp;nbsp;two-minute &lt;code&gt;ReadTimeout&lt;/code&gt; on the &lt;span class="caps"&gt;HTTP&lt;/span&gt; server could cut off large uploads. The verifier looked into it, found the upload concern minor, and then found what it described as a much more serious problem: the editor&amp;#8217;s language server runs over a WebSocket, and the verifier claimed&amp;nbsp;Go&amp;#8217;s &lt;code&gt;net/http&lt;/code&gt; does not reset the connection&amp;#8217;s read deadline when a handler hijacks it. Every editor session would be disconnected after two minutes, idle or&amp;nbsp;not.&lt;/p&gt;
&lt;p&gt;The evidence looked impeccable. Exact line numbers for the WebSocket upgrade, the read loop, the timeout setting, and even a note that the existing end-to-end test only reads for ten seconds and would therefore never notice. Upgraded to &lt;strong&gt;major&lt;/strong&gt;, marked &amp;#8220;checked against the code&amp;nbsp;(corrected)&amp;#8221;.&lt;/p&gt;
&lt;p&gt;It is&amp;nbsp;wrong. &lt;code&gt;net/http&lt;/code&gt; clears the connection deadlines when a connection is hijacked. I wrote that into &lt;span class="caps"&gt;PR&lt;/span&gt; B as &amp;#8220;not a bug&amp;#8221; and left the timeout as it&amp;nbsp;was.&lt;/p&gt;
&lt;p&gt;The reason is structural. The verifier reads the &lt;strong&gt;repository snapshot&lt;/strong&gt;. Go&amp;#8217;s standard library is not in it. Every line reference about my code was correct. The one claim about code it could not see was a guess, delivered in exactly the same confident tone as the facts around it. That is the original problem all over again, just one layer further&amp;nbsp;out.&lt;/p&gt;
&lt;h3 id="knowing-what-is-already-known"&gt;Knowing What Is Already&amp;nbsp;Known&lt;/h3&gt;
&lt;p&gt;One finding that was not wrong but also not new: the &lt;span class="caps"&gt;REST&lt;/span&gt; read budget is charged after the rows are loaded, so the limit caps what is returned rather than what the database reads. The verifier confirmed it, and then pointed out that the repository&amp;#8217;s own audit review document already tracks this as a known residual issue, with a statement timeout as the actual bound. It also rejected the second half of the finding, about nested expansions, as speculative, because the &lt;span class="caps"&gt;API&lt;/span&gt; has no nested expansion syntax&amp;nbsp;today.&lt;/p&gt;
&lt;p&gt;I like this one. It is what a colleague would say: &amp;#8220;yes, that is real, and yes, you already&amp;nbsp;know&amp;#8221;.&lt;/p&gt;
&lt;h3 id="what-i-did-with-it"&gt;What I Did With&amp;nbsp;It&lt;/h3&gt;
&lt;p&gt;Of the 11 inline findings on &lt;span class="caps"&gt;PR&lt;/span&gt; A, roughly half led to a fix: the timesheet race, the lint guard, a draft delete that returned raw database errors instead of the runtime&amp;#8217;s typed ones, and two related gaps in the database reconnect logic. One was wrong. The rest I judged as hardening or documentation items and wrote down as deliberately not changed, each with a one-line&amp;nbsp;reason.&lt;/p&gt;
&lt;p&gt;That list in the &lt;span class="caps"&gt;PR&lt;/span&gt; description matters to me. The bot does not decide anything. It produces a set of claims with evidence, and I decide which of them are worth code. A finding I reject with a reason is not a failure of the process. It is the&amp;nbsp;process.&lt;/p&gt;
&lt;p&gt;Then the bot reviewed the fix itself. &lt;span class="caps"&gt;PR&lt;/span&gt; B: no issues found, three findings checked and all three discarded, eight tenths of a cent. The summary closed with a sentence I appreciated more than any&amp;nbsp;finding:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;The diff matches the description. I have not verified the claimed test&amp;nbsp;runs.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;Which is precisely the right amount of trust to place in a &lt;span class="caps"&gt;PR&lt;/span&gt;&amp;nbsp;description.&lt;/p&gt;
&lt;h3 id="dry-runs"&gt;Dry&amp;nbsp;Runs&lt;/h3&gt;
&lt;p&gt;For testing prompt changes, a &lt;span class="caps"&gt;CLI&lt;/span&gt; runs the full review against a real &lt;span class="caps"&gt;PR&lt;/span&gt; and prints the result instead of posting&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;review-bot&lt;span class="w"&gt; &lt;/span&gt;python&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;review_bot.cli&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;chris/&amp;lt;repo&amp;gt;#2&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That made iterating on prompts far less embarrassing than doing it live in a pull&amp;nbsp;request.&lt;/p&gt;
&lt;h2 id="limitations"&gt;Limitations&lt;/h2&gt;
&lt;p&gt;A few things I know about and have not&amp;nbsp;fixed:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The first pass only sees the diff.&lt;/strong&gt; The verifier can look things up, but only to check findings, not to find new ones. Problems that only become visible with more context can still be&amp;nbsp;missed.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Re-reviews are full reviews.&lt;/strong&gt;&amp;nbsp;With &lt;code&gt;REVIEW_ON_SYNC&lt;/code&gt;, the whole &lt;span class="caps"&gt;PR&lt;/span&gt; is reviewed again, and earlier findings may be repeated. The better approach is to review only the new commits and pass the previous findings along. All bot output already carries a hidden &lt;span class="caps"&gt;HTML&lt;/span&gt; marker to make that possible&amp;nbsp;later.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No cap on the total unpacked snapshot size.&lt;/strong&gt; Individual files are capped, the compressed download is capped, the sum is not. See the verification pass&amp;nbsp;above.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Deduplication lives in memory.&lt;/strong&gt; After a restart, a reopened &lt;span class="caps"&gt;PR&lt;/span&gt; may get reviewed again. For a single-user Forgejo, that is a few cents I can live&amp;nbsp;with.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The verifier can be wrong.&lt;/strong&gt; It only sees the repository, not dependencies or the language&amp;#8217;s standard library, and it is just as confident about those as about the code it actually read. That is why refuted findings stay visible, and why a confirmed finding is still a claim to check, not a&amp;nbsp;verdict.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;It is still a language model.&lt;/strong&gt; A clean review does not mean the code is correct. It means two model calls did not find anything that survived a&amp;nbsp;check.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;On the roadmap: a per-repository config file for focus areas and ignore paths, incremental reviews, a mention command for asking the bot to explain something, and a proper Forgejo Actions&amp;nbsp;build.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;I expected the hard part to be the integration: webhooks, Forgejo&amp;#8217;s &lt;span class="caps"&gt;API&lt;/span&gt;, line placement, getting reasoning models to produce output at all. Those took a&amp;nbsp;day.&lt;/p&gt;
&lt;p&gt;The hard part was trust. A review bot that is wrong one time in three teaches you to ignore it, and then it is worse than no bot at all. On my largest real &lt;span class="caps"&gt;PR&lt;/span&gt;, the first pass alone would have been wrong 18 times out of 29. The fix was not a better prompt. It was the same thing a good human reviewer does: before claiming that something is broken, go and look at the code. And the WebSocket finding shows where that stops working: at the edge of the code it can look&amp;nbsp;at.&lt;/p&gt;
&lt;p&gt;The other lesson is about where to put the guardrails. I spent a lot of time on the prompt injection section of the system prompt, and I am glad it is there. But the reason I sleep fine is not the prompt. It is that the bot&amp;#8217;s Forgejo user simply cannot merge anything, no matter what it has been convinced&amp;nbsp;of.&lt;/p&gt;
&lt;p&gt;My one-person review process now has a second reviewer. It is cheap, it is fast, it does not get bored on the eleventh file, and it has never once said &amp;#8220;&lt;span class="caps"&gt;LGTM&lt;/span&gt;&amp;#8221; without reading the diff. That alone puts it ahead of me on a Friday&amp;nbsp;afternoon.&lt;/p&gt;</content><category term="Linux"/><category term="forgejo"/><category term="ai"/><category term="llm"/><category term="code-review"/><category term="podman"/><category term="quadlets"/><category term="python"/><category term="self-hosting"/><category term="security"/><category term="openrouter"/></entry><entry><title>Linux Foundationals: From Silicon to Userspace - The Boot and Init Chain</title><link href="https://hofstede.it/linux-foundationals-from-silicon-to-userspace-the-boot-and-init-chain/" rel="alternate"/><published>2026-10-31T00:00:00+01:00</published><updated>2026-10-31T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-10-31:/linux-foundationals-from-silicon-to-userspace-the-boot-and-init-chain/</id><summary type="html">&lt;p&gt;The first article in the Linux Foundationals series. It traces a Fedora system from the &lt;span class="caps"&gt;UEFI&lt;/span&gt; handoff through shim and Secure Boot, &lt;span class="caps"&gt;MOK&lt;/span&gt; enrollment with mokutil, &lt;span class="caps"&gt;GRUB2&lt;/span&gt; and Boot Loader Specification entries, the dracut initramfs and switch_root, all the way to systemd as &lt;span class="caps"&gt;PID&lt;/span&gt; 1. Then it covers units, dependencies versus ordering, targets, and how to measure and fix a slow boot with&amp;nbsp;systemd-analyze.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A dimly lit desk with a monitor showing boot log messages next to a UEFI, shim, GRUB, kernel, initramfs and systemd chain diagram, a laptop with a terminal, and a notebook with the same chain sketched by hand" src="https://hofstede.it/images/2026-10-31-linux-foundationals-boot.webp" title="Linux Foundationals: from silicon to userspace."&gt;&lt;/p&gt;
&lt;p&gt;Every Linux system you have ever logged into went through the same sequence a few seconds before you got a prompt. Firmware found a bootloader, the bootloader found a kernel, the kernel found a root filesystem, and something called &lt;span class="caps"&gt;PID&lt;/span&gt; 1 turned all of that into a running system. Most of the time nobody looks at this chain. It only gets attention when it breaks, and then it gets a lot of attention at&amp;nbsp;once.&lt;/p&gt;
&lt;p&gt;This is the first article in the &lt;strong&gt;Linux Foundationals&lt;/strong&gt; series, the Linux counterpart to &lt;a href="https://hofstede.it/freebsd-foundationals-the-boot-process-from-the-loader-to-boot-environments/"&gt;FreeBSD Foundationals&lt;/a&gt;. The reference system throughout is Fedora (45 at the time of writing), but nearly everything here applies unchanged to &lt;span class="caps"&gt;RHEL&lt;/span&gt; 9 and 10, CentOS Stream and the other members of the family. The series follows the order in which a system actually comes alive: this part gets us from power-on to a running systemd, and the following parts build on top of&amp;nbsp;that.&lt;/p&gt;
&lt;p&gt;By the end of this article you&amp;#8217;ll know what every file on the &lt;span class="caps"&gt;EFI&lt;/span&gt; System Partition is for, what Secure Boot actually verifies (and what it doesn&amp;#8217;t), how to make your own kernel modules trusted&amp;nbsp;with &lt;code&gt;mokutil&lt;/code&gt;, where &lt;span class="caps"&gt;GRUB&lt;/span&gt; on Fedora gets its menu entries from, what happens inside the initramfs, how systemd decides what to start and in which order, and how to find out why your machine takes a minute to&amp;nbsp;boot.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-boot-chain-at-a-glance"&gt;The Boot Chain at a&amp;nbsp;Glance&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#uefi-and-the-efi-system-partition"&gt;&lt;span class="caps"&gt;UEFI&lt;/span&gt; and the &lt;span class="caps"&gt;EFI&lt;/span&gt; System&amp;nbsp;Partition&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#secure-boot-shim-and-the-chain-of-trust"&gt;Secure Boot, shim and the Chain of Trust&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#revocation-and-sbat"&gt;Revocation and &lt;span class="caps"&gt;SBAT&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#mok-becoming-a-machine-owner-with-mokutil"&gt;&lt;span class="caps"&gt;MOK&lt;/span&gt;: Becoming a Machine Owner with mokutil&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-enrollment-workflow"&gt;The enrollment&amp;nbsp;workflow&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#signing-your-own-modules"&gt;Signing your own&amp;nbsp;modules&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#other-useful-mokutil-operations"&gt;Other useful mokutil&amp;nbsp;operations&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#grub2-and-the-boot-loader-specification"&gt;&lt;span class="caps"&gt;GRUB2&lt;/span&gt; and the Boot Loader Specification&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#where-the-configuration-actually-lives"&gt;Where the configuration actually&amp;nbsp;lives&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#bls-entries"&gt;&lt;span class="caps"&gt;BLS&lt;/span&gt;&amp;nbsp;entries&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#managing-entries-with-grubby"&gt;Managing entries with&amp;nbsp;grubby&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#persistent-state-grubenv"&gt;Persistent state:&amp;nbsp;grubenv&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-kernel-and-the-initramfs"&gt;The Kernel and the Initramfs&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#dracut-builds-the-initramfs"&gt;dracut builds the&amp;nbsp;initramfs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#systemd-inside-the-initramfs"&gt;systemd inside the&amp;nbsp;initramfs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#debugging-the-initramfs"&gt;Debugging the&amp;nbsp;initramfs&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#systemd-pid-1-takes-over"&gt;systemd: &lt;span class="caps"&gt;PID&lt;/span&gt; 1 Takes Over&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#generators"&gt;Generators&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#units-and-where-they-live"&gt;Units and where they&amp;nbsp;live&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#dependencies-are-not-ordering"&gt;Dependencies are not&amp;nbsp;ordering&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#targets-synchronization-points-not-runlevels"&gt;Targets: synchronization points, not&amp;nbsp;runlevels&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#measuring-the-boot-systemd-analyze"&gt;Measuring the Boot: systemd-analyze&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#blame-who-took-long"&gt;blame: who took&amp;nbsp;long&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#critical-chain-who-actually-held-up-the-boot"&gt;critical-chain: who actually held up the&amp;nbsp;boot&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#plot-and-dot-the-whole-picture"&gt;plot and dot: the whole&amp;nbsp;picture&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#typical-fixes"&gt;Typical&amp;nbsp;fixes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-journal-what-happened-during-boot"&gt;The journal: what happened during&amp;nbsp;boot&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#when-the-boot-goes-wrong-a-recovery-cheat-sheet"&gt;When the Boot Goes Wrong: A Recovery Cheat&amp;nbsp;Sheet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#common-pitfalls"&gt;Common&amp;nbsp;Pitfalls&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-boot-chain-at-a-glance"&gt;The Boot Chain at a&amp;nbsp;Glance&lt;/h2&gt;
&lt;p&gt;Linux on modern hardware boots in a chain where each stage verifies and starts the next one. On a Fedora machine with Secure Boot enabled, it looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;┌────────────────────────┐
│     UEFI firmware      │  reads BootOrder from NVRAM,
│  (db / dbx / KEK / PK) │  verifies shim against db
└───────────┬────────────┘
            │ loads \EFI\fedora\shimx64.efi from the ESP
┌───────────┴────────────┐
│          shim          │  carries the Fedora/Red Hat cert,
│   (+ MokManager, MOK)  │  verifies GRUB and the kernel
└───────────┬────────────┘
            │ loads \EFI\fedora\grubx64.efi
┌───────────┴────────────┐
│         GRUB2          │  reads grub.cfg + BLS entries
│                        │  from /boot/loader/entries/
└───────────┬────────────┘
            │ loads vmlinuz + initramfs, passes cmdline
┌───────────┴────────────┐
│     Linux kernel       │  unpacks initramfs into a tmpfs,
│                        │  runs its /init (systemd)
└───────────┬────────────┘
            │
┌───────────┴────────────┐
│  initramfs (dracut)    │  finds and unlocks the root device,
│  systemd in the initrd │  mounts it at /sysroot
└───────────┬────────────┘
            │ switch_root /sysroot
┌───────────┴────────────┐
│   systemd (PID 1)      │  generators, units, targets,
│   on the real root     │  default.target
└───────────┬────────────┘
            │
      login / desktop
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Compared to FreeBSD, where a single loader reads a text file and starts the kernel, the Linux chain has more moving parts. There are two bootloader-ish stages (shim and &lt;span class="caps"&gt;GRUB&lt;/span&gt;), there is a whole mini operating system in the initramfs, and &lt;span class="caps"&gt;PID&lt;/span&gt; 1 is a dependency-resolving service manager rather than a shell script runner. Each of these exists for a reason, so let&amp;#8217;s walk through them in&amp;nbsp;order.&lt;/p&gt;
&lt;h2 id="uefi-and-the-efi-system-partition"&gt;&lt;span class="caps"&gt;UEFI&lt;/span&gt; and the &lt;span class="caps"&gt;EFI&lt;/span&gt; System&amp;nbsp;Partition&lt;/h2&gt;
&lt;p&gt;When the machine powers on, the &lt;span class="caps"&gt;UEFI&lt;/span&gt; firmware initializes the hardware and then consults its boot entries, which live in &lt;span class="caps"&gt;NVRAM&lt;/span&gt;, not on any disk. You can read and modify them from a running system&amp;nbsp;with &lt;code&gt;efibootmgr&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;efibootmgr&lt;span class="w"&gt; &lt;/span&gt;-v
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Here&amp;#8217;s the output from my ThinkPad, shortened to the interesting entries. Recent versions&amp;nbsp;of &lt;code&gt;efibootmgr&lt;/code&gt; also print every device path as a hex dump below each entry, which I&amp;#8217;ve&amp;nbsp;removed:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;BootCurrent: 0001
Timeout: 0 seconds
BootOrder: 0001,0000,001C,001D,001E,001F,0020,0021,0022,0023,0024,0002
Boot0000* Windows Boot Manager    HD(1,GPT,f640ee19-...,0x800,0x12c000)/\EFI\Microsoft\Boot\bootmgfw.efi...
Boot0001* Fedora                  HD(1,GPT,f640ee19-...,0x800,0x12c000)/\EFI\fedora\shimx64.efi
Boot0002* Linux-Firmware-Updater  HD(1,GPT,f640ee19-...,0x800,0x12c000)/\EFI\fedora\shimx64.efi5c00660077...
Boot0010  Setup                   FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0013  Lenovo Diagnostics      FvFile(3f7e615b-0d45-4f80-88dc-26b234958560)
Boot0016  ThinkShield secure wipe FvFile(3593a0d5-bd52-43a0-808e-cbff5ece2477)
Boot001E* NVMe0                   VenMsg(bc7838d2-...,001c199932d94c4eae9aa0b6e98eb8a400)
Boot0020* PXE BOOT                VenMsg(bc7838d2-...,78a84aaf2b2afc4ea79cf5cc8f3d3803)
Boot0021* LENOVO CLOUD            VenMsg(bc7838d2-...)/Uri(https://download.lenovo.com/pccbbs/cdeploy/efi/boot.efi)
...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;BootCurrent: 0001&lt;/code&gt; tells you which entry this boot came from,&amp;nbsp;and &lt;code&gt;BootOrder&lt;/code&gt; is the order the firmware tries them in. The asterisk marks an entry as active. There are three kinds of entries in this&amp;nbsp;list:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;HD(...)&lt;/code&gt; entries&lt;/strong&gt; point at a file on a partition, identified by &lt;span class="caps"&gt;GPT&lt;/span&gt; partition number and partition &lt;span class="caps"&gt;GUID&lt;/span&gt;. These are the ones operating systems create. All three point at the same partition, partition 1 of the NVMe&amp;nbsp;drive.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;FvFile(...)&lt;/code&gt; entries&lt;/strong&gt; are applications built into the firmware volume itself: the setup screen, Lenovo&amp;#8217;s diagnostics, the secure wipe tool. They don&amp;#8217;t exist on any&amp;nbsp;disk.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;VenMsg(...)&lt;/code&gt; entries&lt;/strong&gt; are vendor-defined generic entries like &amp;#8220;boot whatever is on NVMe0&amp;#8221; or &amp;#8220;boot from the&amp;nbsp;network&amp;#8221;. &lt;code&gt;LENOVO CLOUD&lt;/code&gt; is a nice reminder that modern firmware can boot over &lt;span class="caps"&gt;HTTPS&lt;/span&gt; straight from the vendor&amp;#8217;s servers to reinstall the&amp;nbsp;machine.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Note that the Fedora entry points&amp;nbsp;at &lt;code&gt;shimx64.efi&lt;/code&gt;, not at &lt;span class="caps"&gt;GRUB&lt;/span&gt;. More on why in a moment.&amp;nbsp;The &lt;code&gt;Linux-Firmware-Updater&lt;/code&gt; entry, created&amp;nbsp;by &lt;code&gt;fwupd&lt;/code&gt;, points at shim &lt;em&gt;too&lt;/em&gt;. The hex blob after the path is optional data that tells shim to&amp;nbsp;load &lt;code&gt;\fwupdx64.efi&lt;/code&gt; instead of &lt;span class="caps"&gt;GRUB&lt;/span&gt;. That&amp;#8217;s how firmware capsule updates get applied on a Secure Boot machine: shim verifies and starts fwupd&amp;#8217;s signed &lt;span class="caps"&gt;EFI&lt;/span&gt; helper, which hands the update to the&amp;nbsp;firmware.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;HD(...)&lt;/code&gt; entries point at a &lt;span class="caps"&gt;FAT&lt;/span&gt;-formatted partition called the &lt;strong&gt;&lt;span class="caps"&gt;EFI&lt;/span&gt; System Partition (&lt;span class="caps"&gt;ESP&lt;/span&gt;)&lt;/strong&gt;. Fedora mounts it&amp;nbsp;at &lt;code&gt;/boot/efi&lt;/code&gt;. Here&amp;#8217;s what mine&amp;nbsp;contains:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;find&lt;span class="w"&gt; &lt;/span&gt;/boot/efi&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/boot/efi/EFI/BOOT/BOOTIA32.EFI
/boot/efi/EFI/BOOT/BOOTX64.EFI        # fallback copy of shim
/boot/efi/EFI/BOOT/fbia32.efi
/boot/efi/EFI/BOOT/fbx64.efi          # fallback helper that recreates NVRAM entries
/boot/efi/EFI/fedora/grub.cfg         # a tiny stub, not the real config
/boot/efi/EFI/fedora/fwupdx64.efi     # fwupd&amp;#39;s capsule update helper
/boot/efi/EFI/fedora/BOOTIA32.CSV
/boot/efi/EFI/fedora/BOOTX64.CSV      # tells fbx64.efi which entry to create
/boot/efi/EFI/fedora/gcdia32.efi
/boot/efi/EFI/fedora/gcdx64.efi       # GRUB build used for booting from optical media
/boot/efi/EFI/fedora/grubia32.efi
/boot/efi/EFI/fedora/grubx64.efi      # GRUB2, signed by Fedora
/boot/efi/EFI/fedora/mmia32.efi
/boot/efi/EFI/fedora/mmx64.efi        # MokManager, the MOK enrollment UI
/boot/efi/EFI/fedora/shim.efi         # copy of shimx64.efi under a generic name
/boot/efi/EFI/fedora/shimia32.efi
/boot/efi/EFI/fedora/shimx64.efi      # shim, signed by Microsoft
/boot/efi/System/Library/CoreServices/SystemVersion.plist
/boot/efi/mach_kernel
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every component exists twice: once&amp;nbsp;as &lt;code&gt;x64&lt;/code&gt; and once&amp;nbsp;as &lt;code&gt;ia32&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;ia32&lt;/code&gt; variants are for machines with 64-bit CPUs but &lt;strong&gt;32-bit &lt;span class="caps"&gt;UEFI&lt;/span&gt; firmware&lt;/strong&gt;, a cost-saving combination found on some cheap tablets and early Intel Macs. Fedora installs them everywhere so the same &lt;span class="caps"&gt;ESP&lt;/span&gt; works on such hardware. On a ThinkPad they&amp;#8217;re never&amp;nbsp;used.&lt;/p&gt;
&lt;p&gt;The two oddballs at the&amp;nbsp;bottom, &lt;code&gt;mach_kernel&lt;/code&gt; and &lt;code&gt;System/Library/CoreServices/SystemVersion.plist&lt;/code&gt;, are for Apple hardware: Apple&amp;#8217;s boot picker only offers a partition if it looks like a macOS volume, so Fedora leaves these placeholder files there. On anything that isn&amp;#8217;t a Mac, they&amp;#8217;re harmless&amp;nbsp;leftovers.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;EFI/BOOT/&lt;/code&gt; directory deserves a&amp;nbsp;mention. &lt;code&gt;BOOTX64.EFI&lt;/code&gt; is the path every &lt;span class="caps"&gt;UEFI&lt;/span&gt; firmware tries when it has no valid boot entries, for example after a firmware reset or when you move the disk to another machine. On Fedora that file is a copy of shim. When shim is started from this fallback path, it&amp;nbsp;runs &lt;code&gt;fbx64.efi&lt;/code&gt;, which&amp;nbsp;reads &lt;code&gt;BOOTX64.CSV&lt;/code&gt; in each vendor directory, recreates the matching &lt;span class="caps"&gt;NVRAM&lt;/span&gt; boot entry, and reboots into it. That&amp;#8217;s why a Fedora disk usually &amp;#8220;just boots&amp;#8221; in a new&amp;nbsp;machine.&lt;/p&gt;
&lt;p&gt;On my laptop, which uses Fedora&amp;#8217;s default layout with full-disk encryption, the disk looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/dev/nvme0n1p1   vfat         /boot/efi   ESP (shim, GRUB, MokManager)
/dev/nvme0n1p2   ext4         /boot       kernels, initramfs images, BLS entries, grub.cfg
/dev/nvme0n1p3   crypto_LUKS              LUKS2 container
  └─ luks-03ef...  btrfs      /, /home    subvolumes &amp;quot;root&amp;quot; and &amp;quot;home&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The separate, unencrypted&amp;nbsp;ext4 &lt;code&gt;/boot&lt;/code&gt; is there so &lt;span class="caps"&gt;GRUB&lt;/span&gt; never has to understand &lt;span class="caps"&gt;LUKS&lt;/span&gt; or the more exotic Btrfs features. Unlocking the encrypted root is the initramfs&amp;#8217;s job, as we&amp;#8217;ll see. The storage stack itself is the topic of part&amp;nbsp;2.&lt;/p&gt;
&lt;h2 id="secure-boot-shim-and-the-chain-of-trust"&gt;Secure Boot, shim and the Chain of&amp;nbsp;Trust&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;UEFI&lt;/span&gt; Secure Boot is a simple idea: the firmware only executes &lt;span class="caps"&gt;EFI&lt;/span&gt; binaries signed by a key it trusts. The firmware stores its trust configuration in a few &lt;span class="caps"&gt;NVRAM&lt;/span&gt;&amp;nbsp;variables:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Variable&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;span class="caps"&gt;PK&lt;/span&gt;&lt;/strong&gt; (Platform Key)&lt;/td&gt;
&lt;td&gt;The owner of the platform, usually the hardware vendor. Signs updates to &lt;span class="caps"&gt;KEK&lt;/span&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;span class="caps"&gt;KEK&lt;/span&gt;&lt;/strong&gt; (Key Exchange Keys)&lt;/td&gt;
&lt;td&gt;Keys allowed to update db and dbx. Usually Microsoft and the vendor.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;db&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Certificates and hashes of binaries that are allowed to run.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;dbx&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;The revocation list: binaries and certificates that must never run.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Here&amp;#8217;s the catch for Linux distributions: almost every &lt;span class="caps"&gt;PC&lt;/span&gt; ships with Microsoft&amp;#8217;s certificates in db and nothing else. Getting Fedora&amp;#8217;s key into the firmware of every laptop on the planet isn&amp;#8217;t realistic. So the industry settled on &lt;strong&gt;shim&lt;/strong&gt;: a small, deliberately boring first-stage loader that Microsoft signs with its third-party &lt;span class="caps"&gt;UEFI&lt;/span&gt; &lt;span class="caps"&gt;CA&lt;/span&gt;. Shim itself contains the distribution&amp;#8217;s certificate (Fedora&amp;#8217;s, or Red Hat&amp;#8217;s on &lt;span class="caps"&gt;RHEL&lt;/span&gt;), and from that point on the distribution verifies its own&amp;nbsp;binaries.&lt;/p&gt;
&lt;p&gt;The chain of trust works like&amp;nbsp;this:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;The firmware&amp;nbsp;verifies &lt;code&gt;shimx64.efi&lt;/code&gt; against the Microsoft certificate in&amp;nbsp;db.&lt;/li&gt;
&lt;li&gt;Shim&amp;nbsp;verifies &lt;code&gt;grubx64.efi&lt;/code&gt; against the Fedora certificate built into&amp;nbsp;shim.&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;GRUB&lt;/span&gt; doesn&amp;#8217;t verify the kernel itself. It asks shim to do it, through the verification interface shim provides to the binaries it starts. The kernel&amp;nbsp;(&lt;code&gt;vmlinuz&lt;/code&gt;) is a signed &lt;span class="caps"&gt;PE&lt;/span&gt; binary, so the same check&amp;nbsp;applies.&lt;/li&gt;
&lt;li&gt;The kernel, once running with Secure Boot active, enforces signatures on kernel modules and switches itself into &lt;strong&gt;lockdown&lt;/strong&gt;&amp;nbsp;mode.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;You can check whether Secure Boot is active, and what the kernel did about it, without&amp;nbsp;rebooting:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mokutil&lt;span class="w"&gt; &lt;/span&gt;--sb-state
&lt;span class="c1"&gt;# SecureBoot enabled&lt;/span&gt;

cat&lt;span class="w"&gt; &lt;/span&gt;/sys/kernel/security/lockdown
&lt;span class="c1"&gt;# none [integrity] confidentiality&lt;/span&gt;

bootctl&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-A3&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Secure Boot&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The lockdown value in brackets is the active&amp;nbsp;mode. &lt;code&gt;integrity&lt;/code&gt; means the kernel refuses anything that could let userspace modify the running kernel: unsigned&amp;nbsp;modules, &lt;code&gt;/dev/mem&lt;/code&gt; writes, kexec of unsigned kernels, hibernation to an unencrypted swap and a handful of similar operations. That&amp;#8217;s the actual security payoff of Secure Boot. Without lockdown, a signed boot chain would be pointless, because root could load arbitrary code into the kernel a second after&amp;nbsp;boot.&lt;/p&gt;
&lt;p&gt;It&amp;#8217;s equally important to understand what Secure Boot does &lt;strong&gt;not&lt;/strong&gt; do. It doesn&amp;#8217;t verify the initramfs (it&amp;#8217;s not a &lt;span class="caps"&gt;PE&lt;/span&gt; binary, and on Fedora it&amp;#8217;s generated locally on each machine). It doesn&amp;#8217;t verify &lt;span class="caps"&gt;GRUB&lt;/span&gt;&amp;#8217;s configuration or the kernel command line. And it doesn&amp;#8217;t protect anything once userspace is running. If you want to detect changes to those parts, you need measured boot with the &lt;span class="caps"&gt;TPM&lt;/span&gt;, which is&amp;nbsp;what &lt;code&gt;systemd-cryptenroll&lt;/code&gt; builds on in part 2. The other answer is &lt;strong&gt;Unified Kernel Images&lt;/strong&gt; (UKIs), which bundle kernel, initramfs and command line into a single signed &lt;span class="caps"&gt;PE&lt;/span&gt; binary, so Secure Boot covers all three. Fedora already ships UKIs for virtual machines&amp;nbsp;(&lt;code&gt;kernel-uki-virt&lt;/code&gt;), and we&amp;#8217;ll come back to them in part&amp;nbsp;8.&lt;/p&gt;
&lt;h3 id="revocation-and-sbat"&gt;Revocation and &lt;span class="caps"&gt;SBAT&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Secure Boot has a revocation problem. Every time a vulnerability in &lt;span class="caps"&gt;GRUB&lt;/span&gt; or shim is found, every vulnerable binary has to be blocked. Adding each one&amp;#8217;s hash to dbx doesn&amp;#8217;t scale, because dbx is a small &lt;span class="caps"&gt;NVRAM&lt;/span&gt; variable and there are thousands of signed &lt;span class="caps"&gt;GRUB&lt;/span&gt; builds out&amp;nbsp;there.&lt;/p&gt;
&lt;p&gt;The fix is &lt;strong&gt;&lt;span class="caps"&gt;SBAT&lt;/span&gt;&lt;/strong&gt; (Secure Boot Advanced Targeting). Every shim and &lt;span class="caps"&gt;GRUB&lt;/span&gt; binary carries&amp;nbsp;a &lt;code&gt;.sbat&lt;/code&gt; section with a generation number per component. When a vulnerability is fixed, the generation number is bumped, and shim updates a variable&amp;nbsp;called &lt;code&gt;SbatLevel&lt;/code&gt; that sets the minimum generation it accepts. Old, vulnerable binaries are then rejected by generation number, without anyone touching&amp;nbsp;dbx.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Show the SBAT revocations currently applied on this machine&lt;/span&gt;
mokutil&lt;span class="w"&gt; &lt;/span&gt;--list-sbat-revocations

&lt;span class="c1"&gt;# Show the SBAT metadata of the installed GRUB&lt;/span&gt;
objdump&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;.sbat&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;/boot/efi/EFI/fedora/grubx64.efi
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The practical consequence: an old Fedora live &lt;span class="caps"&gt;USB&lt;/span&gt; stick may refuse to boot on a machine that already ran a newer shim. That isn&amp;#8217;t a bug, it&amp;#8217;s the revocation doing its job. Use a current&amp;nbsp;image.&lt;/p&gt;
&lt;p&gt;One more thing worth knowing about in 2026: the Microsoft certificates from 2011 that signed shim for over a decade are being replaced by a new generation of 2023 CAs. &lt;span class="caps"&gt;UEFI&lt;/span&gt; firmware doesn&amp;#8217;t enforce certificate expiry dates, so shims that are already signed keep booting. Newer shims, however, are signed with the 2023 &lt;span class="caps"&gt;CA&lt;/span&gt;, and older machines only accept those once the new certificate has been added to db, either through a firmware update from the vendor or&amp;nbsp;through &lt;code&gt;fwupd&lt;/code&gt;. If an older box suddenly refuses to boot a fresh install image, check for firmware and db updates&amp;nbsp;first.&lt;/p&gt;
&lt;h2 id="mok-becoming-a-machine-owner-with-mokutil"&gt;&lt;span class="caps"&gt;MOK&lt;/span&gt;: Becoming a Machine Owner with&amp;nbsp;mokutil&lt;/h2&gt;
&lt;p&gt;Shim adds a second trust store on top of the firmware&amp;#8217;s: the &lt;strong&gt;Machine Owner Key&lt;/strong&gt; (&lt;span class="caps"&gt;MOK&lt;/span&gt;) list. &lt;span class="caps"&gt;MOK&lt;/span&gt; lets you, the owner of the machine, trust additional keys without touching the firmware&amp;#8217;s db. The typical reason is a kernel module that Fedora doesn&amp;#8217;t sign: the &lt;span class="caps"&gt;NVIDIA&lt;/span&gt; driver built&amp;nbsp;by &lt;code&gt;akmods&lt;/code&gt;, a &lt;span class="caps"&gt;DKMS&lt;/span&gt; module like the one from the &lt;a href="https://hofstede.it/bcachefs-on-rhel-102-the-kernel-that-said-no/"&gt;bcachefs on &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 article&lt;/a&gt;, or &lt;a href="https://hofstede.it/rhel-on-zfs-root-an-unholy-experiment/"&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; on &lt;span class="caps"&gt;RHEL&lt;/span&gt;&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Without an enrolled key, loading such a module fails like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;modprobe: ERROR: could not insert &amp;#39;nvidia&amp;#39;: Key was rejected by service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;#8217;s lockdown doing its job. Turning off Secure Boot &amp;#8220;fixes&amp;#8221; it, and also throws away the protection described above. The proper fix is enrolling the key that signed the&amp;nbsp;module.&lt;/p&gt;
&lt;h3 id="the-enrollment-workflow"&gt;The enrollment&amp;nbsp;workflow&lt;/h3&gt;
&lt;p&gt;Enrolling a key is deliberately a two-step process that requires physical presence at the console. From the running system, you only &lt;em&gt;request&lt;/em&gt; the&amp;nbsp;enrollment:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Example: the key akmods uses to sign locally built modules on Fedora&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;kmodgenca&lt;span class="w"&gt; &lt;/span&gt;-a
sudo&lt;span class="w"&gt; &lt;/span&gt;mokutil&lt;span class="w"&gt; &lt;/span&gt;--import&lt;span class="w"&gt; &lt;/span&gt;/etc/pki/akmods/certs/public_key.der
&lt;span class="c1"&gt;# input password:&lt;/span&gt;
&lt;span class="c1"&gt;# input password again:&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The password is a one-time password you&amp;#8217;ll need exactly once, on the next boot. It doesn&amp;#8217;t have to be strong, but you need to remember it for about a&amp;nbsp;minute.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;mokutil --import&lt;/code&gt; doesn&amp;#8217;t enroll anything. It writes a pending request into an &lt;span class="caps"&gt;NVRAM&lt;/span&gt; variable&amp;nbsp;(&lt;code&gt;MokNew&lt;/code&gt;). On the next reboot, shim sees the pending request and starts &lt;strong&gt;MokManager&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;mmx64.efi&lt;/code&gt;), the blue screen many people have seen once and never&amp;nbsp;understood:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Press any key to perform &lt;span class="caps"&gt;MOK&lt;/span&gt; management&amp;#8221; appears for about ten seconds. Press a&amp;nbsp;key.&lt;/li&gt;
&lt;li&gt;Choose &lt;strong&gt;Enroll &lt;span class="caps"&gt;MOK&lt;/span&gt;&lt;/strong&gt;, optionally &lt;strong&gt;View key&lt;/strong&gt; to check the fingerprint, then &lt;strong&gt;Continue&lt;/strong&gt;.&lt;/li&gt;
&lt;li&gt;Confirm with &lt;strong&gt;Yes&lt;/strong&gt; and enter the password you set&amp;nbsp;with &lt;code&gt;mokutil --import&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Choose &lt;strong&gt;Reboot&lt;/strong&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;If you miss the ten-second window, the system boots normally without the new key. Just run the import again and&amp;nbsp;reboot.&lt;/p&gt;
&lt;p&gt;This ceremony is the whole point. Malware running as root can create a pending request, but it can&amp;#8217;t press keys at the physical console. The key only becomes trusted when a person at the machine confirms&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;After the reboot, verify that the key made&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# List all enrolled MOKs&lt;/span&gt;
mokutil&lt;span class="w"&gt; &lt;/span&gt;--list-enrolled&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-E&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Subject:|Issuer:&amp;#39;&lt;/span&gt;

&lt;span class="c1"&gt;# Check a specific key&lt;/span&gt;
mokutil&lt;span class="w"&gt; &lt;/span&gt;--test-key&lt;span class="w"&gt; &lt;/span&gt;/etc/pki/akmods/certs/public_key.der
&lt;span class="c1"&gt;# /etc/pki/akmods/certs/public_key.der is already enrolled&lt;/span&gt;

&lt;span class="c1"&gt;# The kernel sees MOK keys in the .platform / .machine keyrings&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;keyctl&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt; &lt;/span&gt;%:.machine
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="signing-your-own-modules"&gt;Signing your own&amp;nbsp;modules&lt;/h3&gt;
&lt;p&gt;If you build a module yourself instead of going through akmods or &lt;span class="caps"&gt;DKMS&lt;/span&gt;, you sign it with the&amp;nbsp;kernel&amp;#8217;s &lt;code&gt;sign-file&lt;/code&gt; helper, using the private key that belongs to the certificate you&amp;nbsp;enrolled:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/usr/src/kernels/&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="k"&gt;)&lt;/span&gt;/scripts/sign-file&lt;span class="w"&gt; &lt;/span&gt;sha256&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;signing_key.priv&lt;span class="w"&gt; &lt;/span&gt;signing_key.der&lt;span class="w"&gt; &lt;/span&gt;mymodule.ko

modinfo&lt;span class="w"&gt; &lt;/span&gt;mymodule.ko&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-E&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;signer|sig_key&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Keep the private key off the machine if you can, or at least readable only by root. Anyone who has it can sign code your kernel will&amp;nbsp;trust.&lt;/p&gt;
&lt;h3 id="other-useful-mokutil-operations"&gt;Other useful mokutil&amp;nbsp;operations&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Request removal of an enrolled key (again confirmed in MokManager)&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;mokutil&lt;span class="w"&gt; &lt;/span&gt;--delete&lt;span class="w"&gt; &lt;/span&gt;key.der

&lt;span class="c1"&gt;# Show requests that are still pending for the next boot&lt;/span&gt;
mokutil&lt;span class="w"&gt; &lt;/span&gt;--list-new

&lt;span class="c1"&gt;# Disable shim&amp;#39;s own verification. Needs MokManager confirmation.&lt;/span&gt;
&lt;span class="c1"&gt;# Useful for debugging, almost never a good idea to leave on.&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;mokutil&lt;span class="w"&gt; &lt;/span&gt;--disable-validation
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="grub2-and-the-boot-loader-specification"&gt;&lt;span class="caps"&gt;GRUB2&lt;/span&gt; and the Boot Loader&amp;nbsp;Specification&lt;/h2&gt;
&lt;p&gt;Shim hands over&amp;nbsp;to &lt;code&gt;grubx64.efi&lt;/code&gt;. If you&amp;#8217;ve used &lt;span class="caps"&gt;GRUB&lt;/span&gt; on older distributions, you probably remember a giant&amp;nbsp;generated &lt;code&gt;grub.cfg&lt;/code&gt; with&amp;nbsp;one &lt;code&gt;menuentry&lt;/code&gt; block per kernel. Fedora (since 30) and &lt;span class="caps"&gt;RHEL&lt;/span&gt; (since 8) work differently, and this is where many people get&amp;nbsp;confused.&lt;/p&gt;
&lt;h3 id="where-the-configuration-actually-lives"&gt;Where the configuration actually&amp;nbsp;lives&lt;/h3&gt;
&lt;p&gt;First,&amp;nbsp;the &lt;code&gt;grub.cfg&lt;/code&gt; on the &lt;span class="caps"&gt;ESP&lt;/span&gt; isn&amp;#8217;t the real configuration. It&amp;#8217;s a three-line&amp;nbsp;stub:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/boot/efi/EFI/fedora/grub.cfg
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;search --no-floppy --fs-uuid --set=dev 4a2b...e91c
set prefix=($dev)/grub2

export $prefix
configfile $prefix/grub.cfg
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It finds&amp;nbsp;the &lt;code&gt;/boot&lt;/code&gt; partition by &lt;span class="caps"&gt;UUID&lt;/span&gt; and&amp;nbsp;loads &lt;code&gt;/boot/grub2/grub.cfg&lt;/code&gt; from there. Fedora introduced this stub in Fedora 34. Before that, &lt;span class="caps"&gt;BIOS&lt;/span&gt; systems&amp;nbsp;used &lt;code&gt;/boot/grub2/grub.cfg&lt;/code&gt; and &lt;span class="caps"&gt;UEFI&lt;/span&gt; systems&amp;nbsp;used &lt;code&gt;/boot/efi/EFI/fedora/grub.cfg&lt;/code&gt;, and&amp;nbsp;running &lt;code&gt;grub2-mkconfig -o&lt;/code&gt; with the wrong path was one of the most common ways to end up with a stale boot menu. Now there&amp;#8217;s one path on both, and the real configuration no longer lives on an unjournaled &lt;span class="caps"&gt;FAT&lt;/span&gt; partition. That file is the generated main configuration, built&amp;nbsp;by &lt;code&gt;grub2-mkconfig&lt;/code&gt; from &lt;code&gt;/etc/default/grub&lt;/code&gt; and the scripts&amp;nbsp;in &lt;code&gt;/etc/grub.d/&lt;/code&gt;. But even that file doesn&amp;#8217;t contain your kernels. Instead it contains a single command near the&amp;nbsp;end: &lt;code&gt;blscfg&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="bls-entries"&gt;&lt;span class="caps"&gt;BLS&lt;/span&gt;&amp;nbsp;entries&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;blscfg&lt;/code&gt; reads the &lt;strong&gt;Boot Loader Specification&lt;/strong&gt; (&lt;span class="caps"&gt;BLS&lt;/span&gt;) entries&amp;nbsp;in &lt;code&gt;/boot/loader/entries/&lt;/code&gt;, one small file per installed&amp;nbsp;kernel:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ls&lt;span class="w"&gt; &lt;/span&gt;/boot/loader/entries/
&lt;span class="c1"&gt;# 05be...0f06-0-rescue.conf&lt;/span&gt;
&lt;span class="c1"&gt;# 05be...0f06-7.2.11-300.fc45.x86_64.conf&lt;/span&gt;

sudo&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/boot/loader/entries/*-7.2.11-300.fc45.x86_64.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;title Fedora Linux (7.2.11-300.fc45.x86_64) 45 (KDE Plasma Desktop Edition)
version 7.2.11-300.fc45.x86_64
linux /vmlinuz-7.2.11-300.fc45.x86_64
initrd /initramfs-7.2.11-300.fc45.x86_64.img $tuned_initrd
options root=UUID=5c88... ro rootflags=subvol=root rd.luks.uuid=luks-03ef... rhgb quiet $tuned_params
grub_users $grub_users
grub_arg --unrestricted
grub_class fedora
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The prefix of each filename is the machine &lt;span class="caps"&gt;ID&lt;/span&gt;&amp;nbsp;from &lt;code&gt;/etc/machine-id&lt;/code&gt;. The design is simple and robust: installing a kernel drops one file, removing a kernel deletes one file, and there&amp;#8217;s no need to regenerate a large config file every time. When a kernel &lt;span class="caps"&gt;RPM&lt;/span&gt; is&amp;nbsp;installed, &lt;code&gt;kernel-install&lt;/code&gt; runs its plugins&amp;nbsp;in &lt;code&gt;/usr/lib/kernel/install.d/&lt;/code&gt;, which build the initramfs (dracut) and write the &lt;span class="caps"&gt;BLS&lt;/span&gt;&amp;nbsp;entry.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;options&lt;/code&gt; line is the kernel command&amp;nbsp;line. &lt;code&gt;root=&lt;/code&gt; and &lt;code&gt;rootflags=subvol=root&lt;/code&gt; tell the initramfs what to mount,&amp;nbsp;and &lt;code&gt;rd.luks.uuid=&lt;/code&gt; tells it which &lt;span class="caps"&gt;LUKS&lt;/span&gt; container to unlock first. The two&amp;nbsp;variables, &lt;code&gt;$tuned_params&lt;/code&gt; and &lt;code&gt;$tuned_initrd&lt;/code&gt;, belong to &lt;strong&gt;tuned&lt;/strong&gt;: its bootloader plugin stores the kernel arguments of the active tuned profile in &lt;span class="caps"&gt;GRUB&lt;/span&gt;&amp;#8217;s environment block (more on that below), and &lt;span class="caps"&gt;GRUB&lt;/span&gt; expands them at boot. That way switching a tuned profile can change kernel arguments without rewriting any &lt;span class="caps"&gt;BLS&lt;/span&gt; file. On my laptop both are empty, because the active profile doesn&amp;#8217;t set&amp;nbsp;any.&lt;/p&gt;
&lt;h3 id="managing-entries-with-grubby"&gt;Managing entries with&amp;nbsp;grubby&lt;/h3&gt;
&lt;p&gt;Since the kernel entries are individual files, you don&amp;#8217;t&amp;nbsp;run &lt;code&gt;grub2-mkconfig&lt;/code&gt; to change kernel arguments on Fedora or &lt;span class="caps"&gt;RHEL&lt;/span&gt;. You&amp;nbsp;use &lt;code&gt;grubby&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;grubby&lt;span class="w"&gt; &lt;/span&gt;--info&lt;span class="o"&gt;=&lt;/span&gt;ALL
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;index=0
kernel=&amp;quot;/boot/vmlinuz-7.2.11-300.fc45.x86_64&amp;quot;
args=&amp;quot;ro rootflags=subvol=root rd.luks.uuid=luks-03ef... rhgb quiet $tuned_params&amp;quot;
root=&amp;quot;UUID=5c88...&amp;quot;
initrd=&amp;quot;/boot/initramfs-7.2.11-300.fc45.x86_64.img $tuned_initrd&amp;quot;
title=&amp;quot;Fedora Linux (7.2.11-300.fc45.x86_64) 45 (KDE Plasma Desktop Edition)&amp;quot;
id=&amp;quot;05be...0f06-7.2.11-300.fc45.x86_64&amp;quot;
index=1
kernel=&amp;quot;/boot/vmlinuz-0-rescue-05be...0f06&amp;quot;
args=&amp;quot;ro rootflags=subvol=root rd.luks.uuid=luks-03ef... rhgb quiet&amp;quot;
root=&amp;quot;UUID=5c88...&amp;quot;
initrd=&amp;quot;/boot/initramfs-0-rescue-05be...0f06.img&amp;quot;
title=&amp;quot;Fedora Linux (0-rescue-05be...0f06) 41 (KDE Plasma)&amp;quot;
id=&amp;quot;05be...0f06-0-rescue&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Look closely at the title of the second entry: &lt;strong&gt;41&lt;/strong&gt;. That&amp;#8217;s the &lt;strong&gt;rescue entry&lt;/strong&gt;, with a kernel and initramfs built &lt;em&gt;without&lt;/em&gt; host-only optimization. It contains drivers for basically everything, and it&amp;#8217;s what you boot when the regular initramfs is broken. But it&amp;#8217;s only created when none exists yet, so on my machine it&amp;#8217;s still the one from the original Fedora 41 install, two years and four releases ago. A two-year-old rescue kernel usually still boots, but if you want a current one, delete the old one and&amp;nbsp;let &lt;code&gt;kernel-install&lt;/code&gt; create a new&amp;nbsp;one:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;/boot/*rescue*&lt;span class="w"&gt; &lt;/span&gt;/boot/loader/entries/*rescue*
sudo&lt;span class="w"&gt; &lt;/span&gt;kernel-install&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/lib/modules/&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;/vmlinuz
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The rest of the everyday grubby&amp;nbsp;commands:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Which kernel boots by default?&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;grubby&lt;span class="w"&gt; &lt;/span&gt;--default-kernel

&lt;span class="c1"&gt;# Set a different default&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;grubby&lt;span class="w"&gt; &lt;/span&gt;--set-default&lt;span class="w"&gt; &lt;/span&gt;/boot/vmlinuz-&amp;lt;version&amp;gt;

&lt;span class="c1"&gt;# Add or remove kernel arguments, for one kernel or all of them&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;grubby&lt;span class="w"&gt; &lt;/span&gt;--update-kernel&lt;span class="o"&gt;=&lt;/span&gt;ALL&lt;span class="w"&gt; &lt;/span&gt;--args&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mitigations=auto,nosmt&amp;quot;&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;grubby&lt;span class="w"&gt; &lt;/span&gt;--update-kernel&lt;span class="o"&gt;=&lt;/span&gt;ALL&lt;span class="w"&gt; &lt;/span&gt;--remove-args&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;rhgb quiet&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;--update-kernel=ALL&lt;/code&gt; rewrites every existing entry, and Fedora&amp;#8217;s tooling carries the options forward to kernels you install later.&amp;nbsp;When &lt;code&gt;kernel-install&lt;/code&gt; writes the entry for a new kernel, it takes the command line&amp;nbsp;from &lt;code&gt;/etc/kernel/cmdline&lt;/code&gt;, or&amp;nbsp;from &lt;code&gt;/proc/cmdline&lt;/code&gt; if that file doesn&amp;#8217;t exist. You only&amp;nbsp;need &lt;code&gt;grub2-mkconfig -o /boot/grub2/grub.cfg&lt;/code&gt; when you change things that are really &lt;span class="caps"&gt;GRUB&lt;/span&gt; settings, such as the timeout or the terminal,&amp;nbsp;in &lt;code&gt;/etc/default/grub&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="persistent-state-grubenv"&gt;Persistent state:&amp;nbsp;grubenv&lt;/h3&gt;
&lt;p&gt;&lt;span class="caps"&gt;GRUB&lt;/span&gt; can&amp;#8217;t write to most filesystems, but it can write to a single 1 KiB block&amp;nbsp;file: &lt;code&gt;/boot/grub2/grubenv&lt;/code&gt;. Fedora uses it for things like the saved default entry and the &amp;#8220;boot was successful&amp;#8221;&amp;nbsp;flag:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;grub2-editenv&lt;span class="w"&gt; &lt;/span&gt;list
&lt;span class="c1"&gt;# saved_entry=05be...0f06-7.2.11-300.fc45.x86_64&lt;/span&gt;
&lt;span class="c1"&gt;# menu_auto_hide=1&lt;/span&gt;
&lt;span class="c1"&gt;# tuned_params=&lt;/span&gt;
&lt;span class="c1"&gt;# tuned_initrd=&lt;/span&gt;
&lt;span class="c1"&gt;# boot_success=1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That &lt;code&gt;menu_auto_hide&lt;/code&gt; variable is why you never see the &lt;span class="caps"&gt;GRUB&lt;/span&gt; menu on a single-&lt;span class="caps"&gt;OS&lt;/span&gt; Fedora machine. &lt;span class="caps"&gt;GRUB&lt;/span&gt; hides it if the previous boot was successful. Hold &lt;strong&gt;Shift&lt;/strong&gt; (or press &lt;strong&gt;Esc&lt;/strong&gt; or &lt;strong&gt;F8&lt;/strong&gt; repeatedly on some machines) during boot to get the menu, or turn the hiding off&amp;nbsp;permanently:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;grub2-editenv&lt;span class="w"&gt; &lt;/span&gt;-&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;unset&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;menu_auto_hide
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On servers, I always do this. A boot menu that only shows up when you happen to press the right key at the right time on a remote console is not a recovery&amp;nbsp;tool.&lt;/p&gt;
&lt;h2 id="the-kernel-and-the-initramfs"&gt;The Kernel and the&amp;nbsp;Initramfs&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;GRUB&lt;/span&gt; loads two files into memory: the kernel&amp;nbsp;(&lt;code&gt;vmlinuz-*&lt;/code&gt;) and the initramfs&amp;nbsp;(&lt;code&gt;initramfs-*.img&lt;/code&gt;), and jumps into the kernel with the command line from the &lt;span class="caps"&gt;BLS&lt;/span&gt; entry. You can always see what the running kernel was booted&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cat&lt;span class="w"&gt; &lt;/span&gt;/proc/cmdline
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The kernel initializes the hardware it has built-in drivers for, unpacks the initramfs into a tmpfs and runs&amp;nbsp;its &lt;code&gt;/init&lt;/code&gt;. Why the detour? Because the kernel has a chicken-and-egg problem. To mount the root filesystem it might need a storage driver that&amp;#8217;s a module, a &lt;span class="caps"&gt;LUKS&lt;/span&gt; passphrase from the user, a &lt;span class="caps"&gt;TPM&lt;/span&gt; unlock, &lt;span class="caps"&gt;LVM&lt;/span&gt; or &lt;span class="caps"&gt;RAID&lt;/span&gt; assembly, a network connection for an iSCSI root, or a Btrfs subvolume. All of that requires userspace tools, and those tools live on the root filesystem the kernel is trying to reach. The initramfs is a small root filesystem with just enough tooling to find and open the real&amp;nbsp;one.&lt;/p&gt;
&lt;h3 id="dracut-builds-the-initramfs"&gt;dracut builds the&amp;nbsp;initramfs&lt;/h3&gt;
&lt;p&gt;On Fedora and &lt;span class="caps"&gt;RHEL&lt;/span&gt;, the initramfs is built by &lt;strong&gt;dracut&lt;/strong&gt;. It&amp;#8217;s modular (each feature&amp;nbsp;like &lt;code&gt;crypt&lt;/code&gt;, &lt;code&gt;lvm&lt;/code&gt;, &lt;code&gt;btrfs&lt;/code&gt; or &lt;code&gt;tpm2-tss&lt;/code&gt; is a dracut module), and by default on Fedora it runs in &lt;strong&gt;host-only&lt;/strong&gt; mode: it includes only the drivers and modules this specific machine needs. That keeps the image small and the boot fast, and it&amp;#8217;s also why the rescue entry&amp;nbsp;exists.&lt;/p&gt;
&lt;p&gt;Inspecting an initramfs is&amp;nbsp;easy:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Overview: dracut modules included, plus the full file list&lt;/span&gt;
lsinitrd&lt;span class="w"&gt; &lt;/span&gt;/boot/initramfs-&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="k"&gt;)&lt;/span&gt;.img&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;less

&lt;span class="c1"&gt;# Only the dracut modules&lt;/span&gt;
lsinitrd&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;/boot/initramfs-&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="k"&gt;)&lt;/span&gt;.img

&lt;span class="c1"&gt;# Print a single file from inside the image&lt;/span&gt;
lsinitrd&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/crypttab&lt;span class="w"&gt; &lt;/span&gt;/boot/initramfs-&lt;span class="k"&gt;$(&lt;/span&gt;uname&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="k"&gt;)&lt;/span&gt;.img
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Configuration goes into drop-ins&amp;nbsp;in &lt;code&gt;/etc/dracut.conf.d/&lt;/code&gt;. For example, to make sure &lt;span class="caps"&gt;TPM2&lt;/span&gt; unlocking support is in the&amp;nbsp;image:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/dracut.conf.d/tpm2.conf&lt;/span&gt;
&lt;span class="c1"&gt;# Note the spaces inside the quotes, dracut concatenates these strings&lt;/span&gt;
&lt;span class="nv"&gt;add_dracutmodules&lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot; tpm2-tss &amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Rebuild the initramfs for the running kernel&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;dracut&lt;span class="w"&gt; &lt;/span&gt;-f

&lt;span class="c1"&gt;# Or for every installed kernel&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;dracut&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;--regenerate-all
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A rebuild is needed whenever something the early boot depends on&amp;nbsp;changes: &lt;code&gt;/etc/crypttab&lt;/code&gt;, dracut configuration, the keyboard layout for the &lt;span class="caps"&gt;LUKS&lt;/span&gt; prompt, or kernel modules that are needed before root is mounted. Package updates usually trigger it automatically, manual configuration changes&amp;nbsp;don&amp;#8217;t.&lt;/p&gt;
&lt;h3 id="systemd-inside-the-initramfs"&gt;systemd inside the&amp;nbsp;initramfs&lt;/h3&gt;
&lt;p&gt;Here&amp;#8217;s a detail that surprises many people: on Fedora, the initramfs runs &lt;strong&gt;systemd&lt;/strong&gt; as its init. It&amp;#8217;s the same systemd binary, running a reduced set of units. Dracut generates the units for finding and mounting the root device, and the sequence goes roughly like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemd (in initrd)
  ├── initrd-root-device.target   root block device has appeared
  │     (systemd-cryptsetup@... asks for the LUKS passphrase here)
  ├── sysroot.mount               real root mounted at /sysroot
  ├── initrd-root-fs.target
  ├── initrd-parse-etc.service    reads /sysroot/etc/fstab for /usr etc.
  ├── initrd-fs.target
  ├── initrd.target
  └── initrd-switch-root.target
        └── initrd-switch-root.service
              → systemctl switch-root /sysroot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;switch_root&lt;/strong&gt; is the final step. It deletes everything in the initramfs tmpfs to free the memory,&amp;nbsp;moves &lt;code&gt;/sysroot&lt;/code&gt; to &lt;code&gt;/&lt;/code&gt;, and re-executes systemd from the real root filesystem. The new systemd keeps &lt;span class="caps"&gt;PID&lt;/span&gt; 1, picks up the state the initrd systemd serialized, and takes over. From this point on, the system is running from its real&amp;nbsp;root.&lt;/p&gt;
&lt;h3 id="debugging-the-initramfs"&gt;Debugging the&amp;nbsp;initramfs&lt;/h3&gt;
&lt;p&gt;When the boot hangs before the real root is mounted, these kernel command line options help. Press &lt;strong&gt;e&lt;/strong&gt; on a &lt;span class="caps"&gt;GRUB&lt;/span&gt; menu entry, append them to&amp;nbsp;the &lt;code&gt;linux&lt;/code&gt; line, and boot with &lt;strong&gt;Ctrl+x&lt;/strong&gt;:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Option&lt;/th&gt;
&lt;th&gt;Effect&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rd.break&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Drop to a shell inside the initramfs right before switch_root. The real root is mounted&amp;nbsp;at &lt;code&gt;/sysroot&lt;/code&gt;.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rd.break=pre-mount&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Drop to a shell before the root filesystem is mounted.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rd.debug&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Very verbose logging from the dracut scripts.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rd.shell&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Offer a shell if the initramfs fails, instead of hanging.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rd.luks=0&lt;/code&gt;, &lt;code&gt;rd.lvm=0&lt;/code&gt;, &amp;#8230;&lt;/td&gt;
&lt;td&gt;Disable specific dracut features for testing.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Remove &lt;code&gt;rhgb quiet&lt;/code&gt; from the line as&amp;nbsp;well. &lt;code&gt;rhgb&lt;/code&gt; starts the Plymouth graphical splash&amp;nbsp;and &lt;code&gt;quiet&lt;/code&gt; suppresses most kernel messages. Both are nice on a desktop and both hide exactly the messages you need when something goes&amp;nbsp;wrong.&lt;/p&gt;
&lt;p&gt;One catch: Fedora installations created without a root password have a &lt;strong&gt;locked root account&lt;/strong&gt;, and the emergency shell then refuses to start with &amp;#8220;Cannot open access to console, the root account is locked&amp;#8221;. In that case,&amp;nbsp;add &lt;code&gt;systemd.setenv=SYSTEMD_SULOGIN_FORCE=1&lt;/code&gt; to the kernel command line to get a shell&amp;nbsp;anyway.&lt;/p&gt;
&lt;p&gt;That this works at all is worth thinking about. Anyone with access to the console and the &lt;span class="caps"&gt;GRUB&lt;/span&gt; menu can get a root shell. If that&amp;#8217;s not acceptable for a machine, the answer is a &lt;span class="caps"&gt;GRUB&lt;/span&gt; password&amp;nbsp;(&lt;code&gt;grub2-setpassword&lt;/code&gt;) and, more importantly, full-disk encryption, because a &lt;span class="caps"&gt;GRUB&lt;/span&gt; password does nothing against someone who boots a &lt;span class="caps"&gt;USB&lt;/span&gt;&amp;nbsp;stick.&lt;/p&gt;
&lt;h2 id="systemd-pid-1-takes-over"&gt;systemd: &lt;span class="caps"&gt;PID&lt;/span&gt; 1 Takes&amp;nbsp;Over&lt;/h2&gt;
&lt;p&gt;Once systemd is running from the real root, it&amp;#8217;s in charge for the rest of the system&amp;#8217;s lifetime. Its job is to take a large set of declarative &lt;strong&gt;units&lt;/strong&gt; and turn them into a running system, in parallel wherever&amp;nbsp;possible.&lt;/p&gt;
&lt;h3 id="generators"&gt;Generators&lt;/h3&gt;
&lt;p&gt;Before systemd loads any units, it runs &lt;strong&gt;generators&lt;/strong&gt;: small programs&amp;nbsp;in &lt;code&gt;/usr/lib/systemd/system-generators/&lt;/code&gt; that translate other configuration formats into units at boot time. The most important one&amp;nbsp;is &lt;code&gt;systemd-fstab-generator&lt;/code&gt;, which turns every line&amp;nbsp;in &lt;code&gt;/etc/fstab&lt;/code&gt; into&amp;nbsp;a &lt;code&gt;.mount&lt;/code&gt; unit:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ls&lt;span class="w"&gt; &lt;/span&gt;/run/systemd/generator/
&lt;span class="c1"&gt;# -.mount  boot.mount  boot-efi.mount  home.mount  local-fs.target.requires/ ...&lt;/span&gt;

systemctl&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;home.mount
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# /run/systemd/generator/home.mount
# Automatically generated by systemd-fstab-generator

[Unit]
Documentation=man:fstab(5) man:systemd-fstab-generator(8)
SourcePath=/etc/fstab
Before=local-fs.target
...

[Mount]
What=/dev/disk/by-uuid/5c88...
Where=/home
Type=btrfs
Options=subvol=home,compress=zstd:1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That explains a classic surprise. After you&amp;nbsp;edit &lt;code&gt;/etc/fstab&lt;/code&gt;, &lt;code&gt;mount -a&lt;/code&gt; works fine, but systemd still uses the old mount units until you&amp;nbsp;run &lt;code&gt;systemctl daemon-reload&lt;/code&gt;, which reruns the&amp;nbsp;generators.&lt;/p&gt;
&lt;h3 id="units-and-where-they-live"&gt;Units and where they&amp;nbsp;live&lt;/h3&gt;
&lt;p&gt;A unit is a small &lt;span class="caps"&gt;INI&lt;/span&gt;-style file describing one thing systemd manages:&amp;nbsp;a &lt;code&gt;.service&lt;/code&gt;, &lt;code&gt;.mount&lt;/code&gt;, &lt;code&gt;.socket&lt;/code&gt;, &lt;code&gt;.timer&lt;/code&gt;, &lt;code&gt;.path&lt;/code&gt;, &lt;code&gt;.device&lt;/code&gt;, &lt;code&gt;.target&lt;/code&gt; and a few more types. Units are loaded from three main directories, with a clear order of&amp;nbsp;precedence:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Path&lt;/th&gt;
&lt;th&gt;Who owns it&lt;/th&gt;
&lt;th&gt;Precedence&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/etc/systemd/system/&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The administrator (you)&lt;/td&gt;
&lt;td&gt;Highest&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/run/systemd/system/&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Runtime, generated, gone after reboot&lt;/td&gt;
&lt;td&gt;Middle&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;/usr/lib/systemd/system/&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Packages (&lt;span class="caps"&gt;RPM&lt;/span&gt;)&lt;/td&gt;
&lt;td&gt;Lowest&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Never edit files&amp;nbsp;in &lt;code&gt;/usr/lib/systemd/system/&lt;/code&gt; directly, the next package update overwrites them. Use a drop-in&amp;nbsp;instead:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Opens an editor for /etc/systemd/system/sshd.service.d/override.conf&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;edit&lt;span class="w"&gt; &lt;/span&gt;sshd.service

&lt;span class="c1"&gt;# Show the effective unit, including all drop-ins&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;sshd.service

&lt;span class="c1"&gt;# List every unit that is overridden or extended on this system&lt;/span&gt;
systemd-delta&lt;span class="w"&gt; &lt;/span&gt;--type&lt;span class="o"&gt;=&lt;/span&gt;extended,overridden
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="dependencies-are-not-ordering"&gt;Dependencies are not&amp;nbsp;ordering&lt;/h3&gt;
&lt;p&gt;If you take one thing from this section, take this one: in systemd, &lt;strong&gt;requirement dependencies and ordering dependencies are two independent concepts&lt;/strong&gt;.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;Wants=&lt;/code&gt;, &lt;code&gt;Requires=&lt;/code&gt;, &lt;code&gt;BindsTo=&lt;/code&gt; and friends say &lt;em&gt;what&lt;/em&gt; gets started&amp;nbsp;together.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;After=&lt;/code&gt; and &lt;code&gt;Before=&lt;/code&gt; say &lt;em&gt;in which order&lt;/em&gt; things start, if they&amp;#8217;re both being started&amp;nbsp;anyway.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A unit&amp;nbsp;with &lt;code&gt;Requires=postgresql.service&lt;/code&gt; and&amp;nbsp;no &lt;code&gt;After=&lt;/code&gt; starts at the same time as PostgreSQL, not after it. A unit&amp;nbsp;with &lt;code&gt;After=postgresql.service&lt;/code&gt; and&amp;nbsp;no &lt;code&gt;Wants=&lt;/code&gt; or &lt;code&gt;Requires=&lt;/code&gt; doesn&amp;#8217;t pull PostgreSQL in at all. It only waits if something else happens to start PostgreSQL in the same transaction. Most of the &amp;#8220;systemd started my service too early&amp;#8221; complaints come down to having one of these without the&amp;nbsp;other.&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Directive&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Wants=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Start these too. If they fail, I don&amp;#8217;t care. The recommended default.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Requires=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Start these too. If they fail to start or get stopped, stop me as well.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;BindsTo=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Like &lt;code&gt;Requires=&lt;/code&gt;, but also stop me if the unit disappears (e.g. a device is unplugged).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Requisite=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;These must already be active, don&amp;#8217;t start them for me.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;PartOf=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Stop and restart me whenever the other unit is stopped or restarted.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Conflicts=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Never run together with these. Starting one stops the other.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;After=&lt;/code&gt; / &lt;code&gt;Before=&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Ordering only. No unit is pulled in by these.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Here&amp;#8217;s a real-world example: a service that needs the network to be fully up, not just &amp;#8220;NetworkManager&amp;nbsp;started&amp;#8221;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/systemd/system/offsite-sync.service&lt;/span&gt;
&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Sync backups to the offsite target&lt;/span&gt;
&lt;span class="na"&gt;Wants&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;network-online.target&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;network-online.target&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;oneshot&lt;/span&gt;
&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/local/bin/offsite-sync&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;multi-user.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;Wants=&lt;/code&gt; pulls &lt;code&gt;network-online.target&lt;/code&gt; into the boot&amp;nbsp;transaction, &lt;code&gt;After=&lt;/code&gt; makes the service wait for it. Leave out either line and you have a race condition that only shows up on every tenth&amp;nbsp;boot.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;[Install]&lt;/code&gt; section isn&amp;#8217;t used at runtime at all. It only&amp;nbsp;tells &lt;code&gt;systemctl enable&lt;/code&gt; what to do: here, create a symlink&amp;nbsp;in &lt;code&gt;/etc/systemd/system/multi-user.target.wants/&lt;/code&gt;. &amp;#8220;Enabling&amp;#8221; a unit is nothing more than creating that symlink, which you can see&amp;nbsp;with &lt;code&gt;systemctl enable --now offsite-sync.service&lt;/code&gt;, as it prints the link it&amp;nbsp;created.&lt;/p&gt;
&lt;p&gt;You can inspect the dependencies of any&amp;nbsp;unit. &lt;code&gt;list-dependencies&lt;/code&gt; shows the tree of everything a unit pulls in, recursively. Here&amp;#8217;s a heavily shortened version from my&amp;nbsp;laptop:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;list-dependencies&lt;span class="w"&gt; &lt;/span&gt;multi-user.target
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;multi-user.target
● ├─auditd.service
● ├─chronyd.service
● ├─firewalld.service
○ ├─mcelog.service
● ├─NetworkManager.service
● ├─rsyslog.service
● ├─systemd-oomd.service
● ├─tuned.service
  ...
● ├─basic.target
● │ ├─sockets.target
● │ │ ├─dbus.socket
● │ │ ├─sshd-unix-local.socket
● │ │ ├─systemd-journald.socket
● │ │ ├─virtqemud.socket
  ...
● │ ├─sysinit.target
● │ │ ├─systemd-journald.service
● │ │ ├─systemd-resolved.service
● │ │ ├─systemd-udevd.service
  ...
● │ │ ├─cryptsetup.target
● │ │ │ └─systemd-cryptsetup@luks\x2d03ef....service
● │ │ ├─local-fs.target
● │ │ │ ├─-.mount
● │ │ │ ├─boot-efi.mount
● │ │ │ ├─boot.mount
● │ │ │ ├─home.mount
● │ │ │ └─tmp.mount
● │ │ ├─swap.target
● │ │ │ └─dev-zram0.swap
  ...
● │ └─timers.target
● │   ├─fstrim.timer
● │   ├─logrotate.timer
  ...
● ├─getty.target
● └─remote-fs.target
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A filled circle means the unit is active, an empty one means it&amp;#8217;s inactive, typically because it only runs under certain conditions&amp;nbsp;(&lt;code&gt;mcelog.service&lt;/code&gt;, for example, only starts on hardware that supports it). The tree already tells the story of this whole&amp;nbsp;article: &lt;code&gt;local-fs.target&lt;/code&gt; contains the mount units the fstab generator&amp;nbsp;created, &lt;code&gt;cryptsetup.target&lt;/code&gt; contains the unit that unlocked my &lt;span class="caps"&gt;LUKS&lt;/span&gt; container,&amp;nbsp;and &lt;code&gt;swap.target&lt;/code&gt; contains a swap device&amp;nbsp;called &lt;code&gt;zram0&lt;/code&gt;, which is not a disk at all. That&amp;#8217;s a topic for part 2. The full output on a desktop is around 200 lines, and a striking number of them&amp;nbsp;are &lt;code&gt;.socket&lt;/code&gt; units. Most of systemd&amp;#8217;s own services and all of libvirt&amp;#8217;s daemons are socket-activated: systemd holds the socket and starts the service only when the first client&amp;nbsp;connects.&lt;/p&gt;
&lt;p&gt;The reverse direction answers &amp;#8220;who pulls this&amp;nbsp;in?&amp;#8221;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;list-dependencies&lt;span class="w"&gt; &lt;/span&gt;--reverse&lt;span class="w"&gt; &lt;/span&gt;sshd.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sshd.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Nothing. On my&amp;nbsp;laptop &lt;code&gt;sshd.service&lt;/code&gt; isn&amp;#8217;t enabled, so nothing wants it and it never starts at boot. That&amp;#8217;s the quickest way to check whether a service is part of the boot transaction at all. For the raw properties of a unit,&amp;nbsp;use &lt;code&gt;systemctl show&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;show&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;Wants&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;Requires&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;After&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;Before&lt;span class="w"&gt; &lt;/span&gt;sshd.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Requires=system.slice sysinit.target
Wants=sshd-keygen.target ssh-host-keys-migration.service
Before=shutdown.target
After=network.target ssh-host-keys-migration.service sysinit.target sshd-keygen.target basic.target system.slice systemd-journald.socket
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The unit file itself only mentions a few of&amp;nbsp;these. &lt;code&gt;Requires=sysinit.target&lt;/code&gt;, &lt;code&gt;After=basic.target&lt;/code&gt;, &lt;code&gt;Before=shutdown.target&lt;/code&gt; and the slice come from &lt;strong&gt;default dependencies&lt;/strong&gt;: unless a unit&amp;nbsp;sets &lt;code&gt;DefaultDependencies=no&lt;/code&gt;, systemd automatically orders every service after the basic system is up and before shutdown. Only early-boot units&amp;nbsp;like &lt;code&gt;systemd-journald.service&lt;/code&gt; opt out of that. Also note that sshd orders itself&amp;nbsp;after &lt;code&gt;network.target&lt;/code&gt;,&amp;nbsp;not &lt;code&gt;network-online.target&lt;/code&gt;. It doesn&amp;#8217;t need a configured network, it just needs to be started after the network management service. That&amp;#8217;s a deliberate choice, and an important one for boot time, as we&amp;#8217;ll see in a&amp;nbsp;moment.&lt;/p&gt;
&lt;p&gt;Finally, check a unit file for mistakes before you rely on&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemd-analyze&lt;span class="w"&gt; &lt;/span&gt;verify&lt;span class="w"&gt; &lt;/span&gt;/etc/systemd/system/offsite-sync.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="targets-synchronization-points-not-runlevels"&gt;Targets: synchronization points, not&amp;nbsp;runlevels&lt;/h3&gt;
&lt;p&gt;A &lt;strong&gt;target&lt;/strong&gt; is a unit that does nothing by itself. It groups other units and serves as a synchronization point during boot. The important ones, in the order they&amp;#8217;re&amp;nbsp;reached:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;local-fs.target ─┐
swap.target ─────┼─► sysinit.target ─► basic.target ─► multi-user.target ─► graphical.target
cryptsetup.target┘        ▲                                   ▲
                          │                                   │
             early services (udev,              sshd, NetworkManager, crond,
             journald, tmpfiles, ...)           your services ...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Target&lt;/th&gt;
&lt;th&gt;Reached when&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;sysinit.target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Local filesystems are mounted, swap is on, udev and journald run.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;basic.target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sockets, timers, paths and early system services are set up.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;network-online.target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The network is configured (only pulled in if something wants it).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;multi-user.target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;A full non-graphical system. Roughly the old runlevel 3.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;graphical.target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;multi-user.target&lt;/code&gt; plus the display manager. Roughly runlevel 5.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;rescue.target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Single-user: basic system, root shell, no network or services.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;emergency.target&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;The bare minimum: root mounted read-only, a root shell, nothing else.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The target the system boots into&amp;nbsp;is &lt;code&gt;default.target&lt;/code&gt;, which is just a&amp;nbsp;symlink:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;get-default
&lt;span class="c1"&gt;# graphical.target&lt;/span&gt;

&lt;span class="c1"&gt;# Make a server boot to text mode&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;set-default&lt;span class="w"&gt; &lt;/span&gt;multi-user.target

&lt;span class="c1"&gt;# Switch the running system to a different target right now&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;isolate&lt;span class="w"&gt; &lt;/span&gt;multi-user.target
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For a single boot, you can override the target on the kernel command line&amp;nbsp;with &lt;code&gt;systemd.unit=rescue.target&lt;/code&gt; or &lt;code&gt;systemd.unit=emergency.target&lt;/code&gt;. The old runlevel numbers&amp;nbsp;(&lt;code&gt;1&lt;/code&gt;, &lt;code&gt;3&lt;/code&gt;, &lt;code&gt;5&lt;/code&gt;) still work there for compatibility, but the explicit names are&amp;nbsp;clearer.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re coming from the &lt;a href="https://hofstede.it/freebsd-foundationals-rcd-the-service-framework-that-fits-in-your-head/"&gt;FreeBSD rc.d article&lt;/a&gt;: targets play roughly the role&amp;nbsp;that &lt;code&gt;PROVIDE:&lt;/code&gt; and &lt;code&gt;REQUIRE:&lt;/code&gt; placeholders&amp;nbsp;like &lt;code&gt;NETWORKING&lt;/code&gt; and &lt;code&gt;LOGIN&lt;/code&gt; play&amp;nbsp;in &lt;code&gt;rcorder&lt;/code&gt;. The big difference is&amp;nbsp;that &lt;code&gt;rcorder&lt;/code&gt; computes one linear order and runs it, while systemd computes a dependency graph and starts everything in parallel that isn&amp;#8217;t ordered against something&amp;nbsp;else.&lt;/p&gt;
&lt;h2 id="measuring-the-boot-systemd-analyze"&gt;Measuring the Boot:&amp;nbsp;systemd-analyze&lt;/h2&gt;
&lt;p&gt;With the whole chain in your head, measuring it becomes&amp;nbsp;meaningful. &lt;code&gt;systemd-analyze&lt;/code&gt; is the tool, and its first output already splits the time by&amp;nbsp;stage:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemd-analyze
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Startup finished in 21.097s (firmware) + 4.117s (loader) + 1.162s (kernel) + 27.754s (initrd) + 9.984s (userspace) = 1min 4.116s
graphical.target reached after 9.804s in userspace.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;#8217;s my laptop, and at first glance, over a minute is&amp;nbsp;embarrassing.&lt;/p&gt;
&lt;p&gt;Every number maps onto a stage from the diagram at the&amp;nbsp;top:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;firmware&lt;/strong&gt;: &lt;span class="caps"&gt;POST&lt;/span&gt;, hardware init, the firmware&amp;#8217;s own boot logo. Only the firmware setup can change this (fast boot, disabling unused controllers or network boot&amp;nbsp;ROMs).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;loader&lt;/strong&gt;: shim and &lt;span class="caps"&gt;GRUB&lt;/span&gt;, including the menu timeout. On Fedora, &lt;span class="caps"&gt;GRUB&lt;/span&gt; reports these timestamps to systemd through &lt;span class="caps"&gt;EFI&lt;/span&gt; variables, which is why they show up at&amp;nbsp;all.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;kernel&lt;/strong&gt;: from the kernel&amp;#8217;s start until it executes the&amp;nbsp;initramfs.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;initrd&lt;/strong&gt;: everything in the initramfs, including the time you spent typing your &lt;span class="caps"&gt;LUKS&lt;/span&gt; passphrase. Keep that in mind before optimizing the wrong&amp;nbsp;thing.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;userspace&lt;/strong&gt;: from switch_root&amp;nbsp;to &lt;code&gt;default.target&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Now read my numbers with that list in mind. &lt;strong&gt;21 seconds of firmware&lt;/strong&gt; is the ThinkPad doing its own thing before any Fedora code runs: memory training, the &lt;span class="caps"&gt;TPM&lt;/span&gt;, the security chip, the network boot ROMs and Lenovo&amp;#8217;s boot logo. &lt;strong&gt;27.8 seconds of initrd&lt;/strong&gt; is almost entirely me: the initramfs asks for the &lt;span class="caps"&gt;LUKS&lt;/span&gt; passphrase and then waits patiently while I type it. The parts that Linux actually spends working, kernel and userspace, add up to about 11 seconds. A minute-long boot on paper, and roughly ten seconds of it that are worth&amp;nbsp;optimizing.&lt;/p&gt;
&lt;p&gt;That&amp;#8217;s the first lesson of boot measurement: before optimizing anything, find out which stage the time is spent in. Userspace is usually the part you have the most influence over, and it&amp;#8217;s what the rest of this section is&amp;nbsp;about.&lt;/p&gt;
&lt;h3 id="blame-who-took-long"&gt;blame: who took&amp;nbsp;long&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemd-analyze&lt;span class="w"&gt; &lt;/span&gt;blame&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;head
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;28.611s sys-module-fuse.device
28.611s sys-devices-pci0000:00-0000:00:08.3-0000:c6:00.3-usb5-5\x2d1.device
28.611s dev-bus-usb-005-002.device
28.574s sys-devices-pci0000:00-...-usb5-5\x2d1-5\x2d1:1.1-0003:1050:0407.0003-hidraw-hidraw2.device
28.574s dev-hidraw2.device
28.544s sys-devices-platform-STM0176:00-tpm-tpm0.device
28.544s dev-tpm0.device
28.471s sys-devices-pci0000:00-...-usb1-1\x2d1-1\x2d1:1.1-0003:20A0:42B2.0001-hidraw-hidraw0.device
28.471s dev-hidraw0.device
28.462s sys-devices-pci0000:00-0000:00:08.1-0000:c4:00.3-usb1-1\x2d1.device
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;blame&lt;/code&gt; lists every unit by how long it took to start, and this is the real output from my laptop. Taken at face value, it says my boot is held up by almost half a minute of &lt;span class="caps"&gt;USB&lt;/span&gt; devices. In reality the whole userspace part took less than ten seconds, as the next section&amp;nbsp;shows.&lt;/p&gt;
&lt;p&gt;What&amp;#8217;s going on here is&amp;nbsp;that &lt;code&gt;.device&lt;/code&gt; units don&amp;#8217;t start in any meaningful sense. They&amp;#8217;re how systemd represents a device that udev reported, and the&amp;nbsp;time &lt;code&gt;blame&lt;/code&gt; shows for them doesn&amp;#8217;t measure anything that delayed the boot. Compare the numbers with the summary above: about 28.5 seconds is suspiciously close to the 27.8 seconds spent in the initramfs, plus a bit. These devices were already found by udev in the initramfs, and their &amp;#8220;activation&amp;#8221; stretched across the whole time the boot sat waiting for my &lt;span class="caps"&gt;LUKS&lt;/span&gt; passphrase. Here they are my &lt;span class="caps"&gt;TPM&lt;/span&gt;&amp;nbsp;(&lt;code&gt;STM0176&lt;/code&gt;, an STMicroelectronics chip), a YubiKey (&lt;span class="caps"&gt;USB&lt;/span&gt; vendor &lt;span class="caps"&gt;ID&lt;/span&gt; &lt;code&gt;1050&lt;/code&gt; is Yubico) and a Nitrokey&amp;nbsp;(&lt;code&gt;20A0&lt;/code&gt;). Nothing waited for any of them. Filter them out to see the&amp;nbsp;services:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemd-analyze&lt;span class="w"&gt; &lt;/span&gt;blame&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\.device$&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;head
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Even&amp;nbsp;then, &lt;code&gt;blame&lt;/code&gt; is misleading by design. Units start in parallel, so a unit that took six seconds may not have delayed the boot at all if nothing waited for it. A timer-started job&amp;nbsp;like &lt;code&gt;dnf-makecache.service&lt;/code&gt; can top the list while being completely irrelevant for how fast you reach the login&amp;nbsp;screen.&lt;/p&gt;
&lt;h3 id="critical-chain-who-actually-held-up-the-boot"&gt;critical-chain: who actually held up the&amp;nbsp;boot&lt;/h3&gt;
&lt;p&gt;The question you actually want answered is &amp;#8220;what was on the critical path?&amp;#8221;.&amp;nbsp;That&amp;#8217;s &lt;code&gt;critical-chain&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemd-analyze&lt;span class="w"&gt; &lt;/span&gt;critical-chain
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;graphical.target @9.804s
└─multi-user.target @9.804s
  └─rsyslog.service @9.681s +123ms
    └─network-online.target @9.678s
      └─NetworkManager-wait-online.service @3.625s +6.052s
        └─NetworkManager.service @3.359s +251ms
          └─network-pre.target @3.358s
            └─firewalld.service @3.357s
              └─basic.target @3.345s
                └─dbus-broker.service @3.318s +26ms
                  └─dbus.socket @3.308s +236us
                    └─sysinit.target @3.304s
                      └─systemd-resolved.service @1.280s +2.023s
                        └─systemd-sysctl.service @1.205s +41ms
                          └─systemd-modules-load.service @1.191s +11ms
                            └─systemd-journald.socket
                              └─system.slice
                                └─-.slice
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The time&amp;nbsp;after &lt;code&gt;@&lt;/code&gt; is when the unit became active, the time&amp;nbsp;after &lt;code&gt;+&lt;/code&gt; is how long it took to start. Read it from the bottom up and look for the&amp;nbsp;big &lt;code&gt;+&lt;/code&gt; values. On my laptop, two of them stand&amp;nbsp;out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;systemd-resolved.service&lt;/code&gt; takes about &lt;strong&gt;2 seconds&lt;/strong&gt;&amp;nbsp;before &lt;code&gt;sysinit.target&lt;/code&gt; can be&amp;nbsp;reached.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;NetworkManager-wait-online.service&lt;/code&gt; takes about &lt;strong&gt;6 seconds&lt;/strong&gt; to decide that the network is up. That&amp;#8217;s more than half of the entire userspace&amp;nbsp;boot.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;And here&amp;#8217;s the interesting part: what&amp;#8217;s waiting&amp;nbsp;for &lt;code&gt;network-online.target&lt;/code&gt;? Not sshd, as we saw, and not NetworkManager itself. It&amp;#8217;s &lt;strong&gt;rsyslog&lt;/strong&gt;. Its unit file orders it&amp;nbsp;after &lt;code&gt;network-online.target&lt;/code&gt;, presumably so that forwarding logs to a remote syslog server works from the first message&amp;nbsp;on. &lt;code&gt;multi-user.target&lt;/code&gt; waits for&amp;nbsp;rsyslog, &lt;code&gt;graphical.target&lt;/code&gt; waits&amp;nbsp;for &lt;code&gt;multi-user.target&lt;/code&gt;, and so a classic syslog daemon that only writes to local files on a laptop delays the whole boot by six&amp;nbsp;seconds.&lt;/p&gt;
&lt;p&gt;Note&amp;nbsp;that &lt;code&gt;critical-chain&lt;/code&gt; follows &lt;code&gt;After=&lt;/code&gt; ordering,&amp;nbsp;not &lt;code&gt;Wants=&lt;/code&gt; dependencies. It shows what waited for what, which is exactly what matters for boot time. You can also pass a unit to look at its specific&amp;nbsp;chain: &lt;code&gt;systemd-analyze critical-chain sshd.service&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="plot-and-dot-the-whole-picture"&gt;plot and dot: the whole&amp;nbsp;picture&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# An SVG timeline of every unit, with start and active times&lt;/span&gt;
systemd-analyze&lt;span class="w"&gt; &lt;/span&gt;plot&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;boot.svg

&lt;span class="c1"&gt;# A dependency graph, filtered to what leads to graphical.target&lt;/span&gt;
systemd-analyze&lt;span class="w"&gt; &lt;/span&gt;dot&lt;span class="w"&gt; &lt;/span&gt;--to-pattern&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;graphical.target&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--require&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;dot&lt;span class="w"&gt; &lt;/span&gt;-Tsvg&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;deps.svg
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The plot is the best way to see parallelism, or the lack of it. Open the &lt;span class="caps"&gt;SVG&lt;/span&gt; in a browser and look for long bars that everything else lines up&amp;nbsp;behind.&lt;/p&gt;
&lt;h3 id="typical-fixes"&gt;Typical&amp;nbsp;fixes&lt;/h3&gt;
&lt;p&gt;Most slow boots come down to a handful of usual&amp;nbsp;suspects:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Something waits&amp;nbsp;for &lt;code&gt;network-online.target&lt;/code&gt; that doesn&amp;#8217;t need to.&lt;/strong&gt; The fix is usually in the waiting service, not&amp;nbsp;in &lt;code&gt;NetworkManager-wait-online.service&lt;/code&gt;. Disabling the wait-online service makes the boot faster and breaks &lt;span class="caps"&gt;NFS&lt;/span&gt; mounts, remote log shipping and anything else that legitimately needs the network. In my case, the question is whether rsyslog needs to run at all. journald already stores every log message, and on a laptop that doesn&amp;#8217;t forward logs anywhere, rsyslog only writes a second copy&amp;nbsp;to &lt;code&gt;/var/log/messages&lt;/code&gt;. If nothing reads those&amp;nbsp;files, &lt;code&gt;sudo systemctl disable --now rsyslog.service&lt;/code&gt; removes it from the critical chain. On a server that forwards logs to a central syslog host, the six seconds are the price of not losing the first log messages, and that&amp;#8217;s a fair price. For your own units, like&amp;nbsp;the &lt;code&gt;offsite-sync.service&lt;/code&gt; example from earlier, the fix is often to trigger them with&amp;nbsp;a &lt;code&gt;.timer&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;WantedBy=multi-user.target&lt;/code&gt;, so nothing waits for them at&amp;nbsp;all.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A oneshot service in the boot path that doesn&amp;#8217;t need to be there.&lt;/strong&gt; Ask whether a unit really needs to&amp;nbsp;be &lt;code&gt;WantedBy=multi-user.target&lt;/code&gt;, or whether a timer, a socket or a path unit would do the job without blocking&amp;nbsp;anything.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A stale entry&amp;nbsp;in &lt;code&gt;/etc/fstab&lt;/code&gt; or &lt;code&gt;/etc/crypttab&lt;/code&gt;.&lt;/strong&gt; A disk that no longer exists gives you a 90-second wait for&amp;nbsp;the &lt;code&gt;.device&lt;/code&gt; unit.&amp;nbsp;Use &lt;code&gt;nofail&lt;/code&gt; for disks that are optional, and remove entries for disks that are&amp;nbsp;gone.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;systemd-udev-settle.service&lt;/code&gt;.&lt;/strong&gt; It&amp;#8217;s a legacy &amp;#8220;wait until udev has processed everything&amp;#8221; unit. If it shows up, something still pulls it in&amp;nbsp;(&lt;code&gt;systemctl list-dependencies --reverse systemd-udev-settle.service&lt;/code&gt; tells you what), and that&amp;#8217;s usually worth a bug&amp;nbsp;report.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Firmware and loader time.&lt;/strong&gt; Not systemd&amp;#8217;s fault. Shorten the &lt;span class="caps"&gt;GRUB&lt;/span&gt; timeout&amp;nbsp;in &lt;code&gt;/etc/default/grub&lt;/code&gt; and&amp;nbsp;regenerate &lt;code&gt;grub.cfg&lt;/code&gt;, and look at the firmware&amp;#8217;s fast boot&amp;nbsp;options.&lt;/p&gt;
&lt;h3 id="the-journal-what-happened-during-boot"&gt;The journal: what happened during&amp;nbsp;boot&lt;/h3&gt;
&lt;p&gt;Timing tells you &lt;em&gt;what&lt;/em&gt; was slow, the journal tells you &lt;em&gt;why&lt;/em&gt;. A few commands I use&amp;nbsp;constantly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# All boots the journal knows about&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;--list-boots

&lt;span class="c1"&gt;# Errors from the current boot&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;err

&lt;span class="c1"&gt;# Everything from the previous boot, e.g. after a crash&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;-1

&lt;span class="c1"&gt;# The current boot with timestamps relative to boot start&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-b&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;short-monotonic

&lt;span class="c1"&gt;# Units that failed&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;--failed

&lt;span class="c1"&gt;# What is systemd still waiting for? (useful during a hanging boot)&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;list-jobs
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;systemctl list-jobs&lt;/code&gt; needs a shell during the hang, of course. For that,&amp;nbsp;add &lt;code&gt;systemd.debug_shell&lt;/code&gt; to the kernel command line: it starts a root shell on &lt;strong&gt;tty9&lt;/strong&gt; (Ctrl+Alt+F9) early in the boot, without a password. It&amp;#8217;s a fantastic debugging tool, and for the same reason never something to leave&amp;nbsp;enabled.&lt;/p&gt;
&lt;h2 id="when-the-boot-goes-wrong-a-recovery-cheat-sheet"&gt;When the Boot Goes Wrong: A Recovery Cheat&amp;nbsp;Sheet&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Symptom&lt;/th&gt;
&lt;th&gt;First thing to try&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;GRUB&lt;/span&gt; menu doesn&amp;#8217;t show up&lt;/td&gt;
&lt;td&gt;Hold &lt;strong&gt;Shift&lt;/strong&gt; or press &lt;strong&gt;Esc&lt;/strong&gt; during boot,&amp;nbsp;then &lt;code&gt;grub2-editenv - unset menu_auto_hide&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;New kernel doesn&amp;#8217;t boot&lt;/td&gt;
&lt;td&gt;Choose the previous kernel in &lt;span class="caps"&gt;GRUB&lt;/span&gt;,&amp;nbsp;then &lt;code&gt;grubby --set-default&lt;/code&gt; until fixed&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Hang before root is mounted&lt;/td&gt;
&lt;td&gt;Remove &lt;code&gt;rhgb quiet&lt;/code&gt;,&amp;nbsp;add &lt;code&gt;rd.shell&lt;/code&gt;, or boot&amp;nbsp;the &lt;code&gt;0-rescue&lt;/code&gt; entry&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Root is mounted, but boot hangs&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemd.debug_shell&lt;/code&gt;,&amp;nbsp;then &lt;code&gt;systemctl list-jobs&lt;/code&gt; on tty9&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;A service breaks the boot&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemd.unit=rescue.target&lt;/code&gt; or &lt;code&gt;systemd.unit=emergency.target&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&amp;#8220;Key was rejected by service&amp;#8221;&lt;/td&gt;
&lt;td&gt;Module isn&amp;#8217;t signed with an enrolled&amp;nbsp;key: &lt;code&gt;mokutil --import&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Root account locked in emergency mode&lt;/td&gt;
&lt;td&gt;Add &lt;code&gt;systemd.setenv=SYSTEMD_SULOGIN_FORCE=1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fedora disk doesn&amp;#8217;t boot in a new machine&lt;/td&gt;
&lt;td&gt;Check that the firmware&amp;#8217;s fallback boot path is&amp;nbsp;enabled, &lt;code&gt;EFI/BOOT/BOOTX64.EFI&lt;/code&gt; handles the rest&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Secure Boot rejects an old install medium&lt;/td&gt;
&lt;td&gt;Expected after &lt;span class="caps"&gt;SBAT&lt;/span&gt; updates, use a current image&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="common-pitfalls"&gt;Common&amp;nbsp;Pitfalls&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Running &lt;code&gt;grub2-mkconfig&lt;/code&gt; to change kernel arguments.&lt;/strong&gt; On a &lt;span class="caps"&gt;BLS&lt;/span&gt; system, kernel arguments live in the entries&amp;nbsp;under &lt;code&gt;/boot/loader/entries/&lt;/code&gt;.&amp;nbsp;Use &lt;code&gt;grubby --update-kernel=ALL --args=...&lt;/code&gt;.&amp;nbsp;Editing &lt;code&gt;GRUB_CMDLINE_LINUX&lt;/code&gt; alone and&amp;nbsp;regenerating &lt;code&gt;grub.cfg&lt;/code&gt; won&amp;#8217;t change the existing entries the way you&amp;nbsp;expect.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Disabling Secure Boot to load a module.&lt;/strong&gt; It works, and it switches off kernel lockdown at the same time. Enrolling a &lt;span class="caps"&gt;MOK&lt;/span&gt; takes two minutes and one reboot. Do that&amp;nbsp;instead.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Editing &lt;code&gt;/etc/fstab&lt;/code&gt; without &lt;code&gt;daemon-reload&lt;/code&gt;.&lt;/strong&gt; The fstab generator only runs at boot and&amp;nbsp;on &lt;code&gt;systemctl daemon-reload&lt;/code&gt;. Until then, systemd keeps using the old mount units, and you&amp;#8217;ll find out at the next reboot, which is the worst possible&amp;nbsp;time.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Using &lt;code&gt;After=&lt;/code&gt; without &lt;code&gt;Wants=&lt;/code&gt; (or the other way around).&lt;/strong&gt; Ordering and requirement are separate. If you need another unit to be running before yours starts, you need&amp;nbsp;both.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Optimizing&amp;nbsp;for &lt;code&gt;systemd-analyze blame&lt;/code&gt;.&lt;/strong&gt; The slowest unit&amp;nbsp;in &lt;code&gt;blame&lt;/code&gt; is often irrelevant. Look&amp;nbsp;at &lt;code&gt;critical-chain&lt;/code&gt; first, then optimize only what&amp;#8217;s actually on&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Forgetting to rebuild the initramfs.&lt;/strong&gt; Changes&amp;nbsp;to &lt;code&gt;/etc/crypttab&lt;/code&gt;, dracut configuration or early-boot modules only take effect&amp;nbsp;after &lt;code&gt;dracut -f&lt;/code&gt;. The kernel will happily boot with an initramfs that has none of your&amp;nbsp;changes.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Losing the rescue path on remote servers.&lt;/strong&gt; Hidden &lt;span class="caps"&gt;GRUB&lt;/span&gt; menus, a locked root account and no out-of-band console add up to a machine you can&amp;#8217;t recover remotely. Set up the console access and test it before you need it, exactly like on&amp;nbsp;FreeBSD.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The Linux boot chain looks complicated from the outside, but each stage exists for a clear reason. &lt;span class="caps"&gt;UEFI&lt;/span&gt; finds an &lt;span class="caps"&gt;EFI&lt;/span&gt; binary on the &lt;span class="caps"&gt;ESP&lt;/span&gt;. Shim bridges the gap between Microsoft&amp;#8217;s certificates in the firmware and the distribution&amp;#8217;s own key, and &lt;span class="caps"&gt;MOK&lt;/span&gt; lets you add your own. &lt;span class="caps"&gt;GRUB&lt;/span&gt; reads one small &lt;span class="caps"&gt;BLS&lt;/span&gt; file per kernel and asks shim to verify the kernel. The initramfs, built by dracut and running its own systemd, solves the problem of reaching a root filesystem that needs userspace tools to open. And after switch_root, systemd turns a graph of declarative units into a running system, in parallel wherever the ordering allows&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The mental model to keep: &lt;strong&gt;verification&lt;/strong&gt; flows from firmware to shim to &lt;span class="caps"&gt;GRUB&lt;/span&gt; to kernel to modules, but stops at the initramfs and the command line. &lt;strong&gt;Configuration&lt;/strong&gt; for kernels lives in &lt;span class="caps"&gt;BLS&lt;/span&gt; entries and is managed&amp;nbsp;with &lt;code&gt;grubby&lt;/code&gt;, not by&amp;nbsp;regenerating &lt;code&gt;grub.cfg&lt;/code&gt;. And in systemd, &lt;strong&gt;requirement and ordering are separate&lt;/strong&gt; questions, which is also the key to&amp;nbsp;reading &lt;code&gt;systemd-analyze critical-chain&lt;/code&gt; correctly.&lt;/p&gt;
&lt;p&gt;The next article in this series picks up where this one ends: systemd has a root filesystem, so now we&amp;#8217;ll look at what that root filesystem actually is. Part 2 covers the storage and memory stack on Fedora: the block layer, device mapper and &lt;span class="caps"&gt;LUKS2&lt;/span&gt; with &lt;span class="caps"&gt;TPM2&lt;/span&gt; auto-unlock&amp;nbsp;via &lt;code&gt;systemd-cryptenroll&lt;/code&gt;, Btrfs subvolumes and compression, and why Fedora replaced disk swap with zram and&amp;nbsp;lets &lt;code&gt;systemd-oomd&lt;/code&gt; step in before the kernel&amp;#8217;s &lt;span class="caps"&gt;OOM&lt;/span&gt; killer&amp;nbsp;does.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/rhboot/shim"&gt;shim on GitHub&lt;/a&gt; - including the &lt;a href="https://github.com/rhboot/shim/blob/main/SBAT.md"&gt;&lt;span class="caps"&gt;SBAT&lt;/span&gt;&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.fedoraproject.org/en-US/quick-docs/grub2-bootloader/"&gt;Fedora Docs - Working with the &lt;span class="caps"&gt;GRUB&lt;/span&gt; 2 Boot&amp;nbsp;Loader&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://uapi-group.org/specifications/specs/boot_loader_specification/"&gt;&lt;span class="caps"&gt;UAPI&lt;/span&gt; Group - Boot Loader&amp;nbsp;Specification&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/managing_monitoring_and_updating_the_kernel/index"&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 - Managing, monitoring, and updating the kernel&lt;/a&gt; - covers grubby, Secure Boot and module&amp;nbsp;signing&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man7.org/linux/man-pages/man7/dracut.cmdline.7.html"&gt;dracut.cmdline(7)&lt;/a&gt; - all&amp;nbsp;the &lt;code&gt;rd.*&lt;/code&gt; options&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/bootup.html"&gt;bootup(7)&lt;/a&gt; - the systemd boot sequence, including the initrd&amp;nbsp;part&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd.unit.html"&gt;systemd.unit(5)&lt;/a&gt; - dependencies and&amp;nbsp;ordering&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/latest/systemd-analyze.html"&gt;systemd-analyze(1)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="linux"/><category term="linux-foundationals"/><category term="fedora"/><category term="rhel"/><category term="boot"/><category term="uefi"/><category term="secure-boot"/><category term="shim"/><category term="mokutil"/><category term="grub"/><category term="dracut"/><category term="initramfs"/><category term="systemd"/><category term="systemd-analyze"/></entry><entry><title>WinBoat Broke, Windows Kept Running: dockurr/windows with Plain Podman</title><link href="https://hofstede.it/winboat-broke-windows-kept-running-dockurrwindows-with-plain-podman/" rel="alternate"/><published>2026-10-16T00:00:00+02:00</published><updated>2026-10-16T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-10-16:/winboat-broke-windows-kept-running-dockurrwindows-with-plain-podman/</id><summary type="html">&lt;p&gt;After a cleanup broke my WinBoat installation, I brought the same Windows 11 &lt;span class="caps"&gt;VM&lt;/span&gt; back with a single rootless podman run against the old storage directory. Here is what dockurr/windows actually is, what lives in its storage folder, why &lt;span class="caps"&gt;NETWORK&lt;/span&gt;=nat quietly became passt in rootless Podman, how I verified &lt;span class="caps"&gt;RDP&lt;/span&gt; without opening a window, and a Quadlet unit to keep it&amp;nbsp;around.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Fedora 44 with KDE Plasma on the left, showing fastfetch and podman ps; on the right, Windows 11 in KRDC over RDP, showing fastfetch inside Windows, the About Windows dialog and Calculator" src="https://hofstede.it/images/2026-10-16-dockurr-windows-podman.webp" title="Left: the Fedora host and the running dockur/windows:6.05 container. Right: the same Windows 11 install, reached over RDP on 127.0.0.1."&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-dockurrwindows-actually-is"&gt;What dockurr/windows Actually&amp;nbsp;Is&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-was-left-in-the-storage-directory"&gt;What Was Left in the Storage&amp;nbsp;Directory&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#bringing-it-back-with-one-command"&gt;Bringing It Back With One&amp;nbsp;Command&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#networknat-silently-became-passt"&gt;&lt;span class="caps"&gt;NETWORK&lt;/span&gt;=nat Silently Became&amp;nbsp;passt&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#checking-that-it-works-without-touching-a-mouse"&gt;Checking That It Works Without Touching a&amp;nbsp;Mouse&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#making-it-permanent-with-a-quadlet"&gt;Making It Permanent With a&amp;nbsp;Quadlet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-it-uses-ram-cpu-and-disk"&gt;What It Uses: &lt;span class="caps"&gt;RAM&lt;/span&gt;, &lt;span class="caps"&gt;CPU&lt;/span&gt; and&amp;nbsp;Disk&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#winboat-or-plain-dockurrwindows"&gt;WinBoat or Plain&amp;nbsp;dockurr/windows?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#caveats"&gt;Caveats&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;A few weeks ago I set up &lt;a href="https://winboat.app"&gt;WinBoat&lt;/a&gt; on my Fedora laptop. WinBoat is a nice Electron frontend that runs Windows applications as if they were Linux windows. It worked. Then I did some housekeeping on the machine, removed things I thought I no longer needed, and the WinBoat &lt;span class="caps"&gt;UI&lt;/span&gt; stopped finding its&amp;nbsp;Windows.&lt;/p&gt;
&lt;p&gt;The interesting part is that WinBoat never was the Windows machine. It is a frontend for &lt;a href="https://hub.docker.com/r/dockurr/windows"&gt;dockurr/windows&lt;/a&gt;, a container image that runs Windows in &lt;span class="caps"&gt;QEMU&lt;/span&gt; with &lt;span class="caps"&gt;KVM&lt;/span&gt;. The &lt;span class="caps"&gt;VM&lt;/span&gt; itself is a disk image plus a few state files in a directory. My cleanup had removed the container and the image. It had not touched the&amp;nbsp;directory.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The short version:&lt;/strong&gt;&amp;nbsp;one &lt;code&gt;podman run&lt;/code&gt; with the original storage directory mounted&amp;nbsp;at &lt;code&gt;/storage&lt;/code&gt;, and the existing Windows 11 install booted straight to the lock screen. No reinstall, no WinBoat, no&amp;nbsp;root.&lt;/p&gt;
&lt;h2 id="what-dockurrwindows-actually-is"&gt;What dockurr/windows Actually&amp;nbsp;Is&lt;/h2&gt;
&lt;p&gt;The project lives at &lt;a href="https://github.com/dockur/windows"&gt;github.com/dockur/windows&lt;/a&gt; and is published both&amp;nbsp;as &lt;code&gt;docker.io/dockurr/windows&lt;/code&gt; and&amp;nbsp;as &lt;code&gt;ghcr.io/dockur/windows&lt;/code&gt;. Note the spelling: the GitHub organisation&amp;nbsp;is &lt;code&gt;dockur&lt;/code&gt;, the Docker Hub account&amp;nbsp;is &lt;code&gt;dockurr&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Inside the image there is no Windows. There is a Debian 13 userland, &lt;span class="caps"&gt;QEMU&lt;/span&gt;, a web &lt;span class="caps"&gt;VNC&lt;/span&gt; viewer, networking helpers and a large collection of shell scripts&amp;nbsp;under &lt;code&gt;/run&lt;/code&gt;. Those scripts do the real&amp;nbsp;work:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Job&lt;/th&gt;
&lt;th&gt;How the container does it&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Get Windows&lt;/td&gt;
&lt;td&gt;Downloads the installation &lt;span class="caps"&gt;ISO&lt;/span&gt; for the&amp;nbsp;chosen &lt;code&gt;VERSION&lt;/code&gt;, or uses one you mount&amp;nbsp;at &lt;code&gt;/custom.iso&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Install Windows&lt;/td&gt;
&lt;td&gt;Builds an unattended answer file and runs Setup without interaction&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Run Windows&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;QEMU&lt;/span&gt; with &lt;span class="caps"&gt;KVM&lt;/span&gt;,&amp;nbsp;a &lt;code&gt;q35&lt;/code&gt; machine, &lt;span class="caps"&gt;UEFI&lt;/span&gt; firmware, VirtIO disk and network&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Show the screen&lt;/td&gt;
&lt;td&gt;noVNC on port 8006, in the browser&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Daily access&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;RDP&lt;/span&gt; on port 3389 inside the guest&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Network&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt; with a tap device, or user-mode networking with passt&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Files&lt;/td&gt;
&lt;td&gt;A Samba share that appears&amp;nbsp;as &lt;code&gt;Shared&lt;/code&gt; on the Windows desktop&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;On my running container, the &lt;span class="caps"&gt;QEMU&lt;/span&gt; command line confirms that&amp;nbsp;picture: &lt;code&gt;accel=kvm&lt;/code&gt;, &lt;code&gt;-cpu host&lt;/code&gt;, four cores, 14 GiB of memory, a&amp;nbsp;raw &lt;code&gt;data.img&lt;/code&gt; attached&amp;nbsp;through &lt;code&gt;virtio-scsi&lt;/code&gt;, and the pflash&amp;nbsp;pair &lt;code&gt;windows.rom&lt;/code&gt; and &lt;code&gt;windows.vars&lt;/code&gt; for &lt;span class="caps"&gt;UEFI&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;VERSION&lt;/code&gt; variable picks what gets installed. The &lt;span class="caps"&gt;README&lt;/span&gt; lists everything&amp;nbsp;from &lt;code&gt;95&lt;/code&gt; and &lt;code&gt;98&lt;/code&gt; through &lt;code&gt;xp&lt;/code&gt;, &lt;code&gt;7u&lt;/code&gt;, &lt;code&gt;10&lt;/code&gt;, &lt;code&gt;11&lt;/code&gt;, the &lt;span class="caps"&gt;LTSC&lt;/span&gt; and Enterprise variants, Windows Server 2003 to 2025, Tiny11 and even ReactOS. The default is Windows 11 Pro. &lt;span class="caps"&gt;ARM64&lt;/span&gt; Windows lives in a separate&amp;nbsp;image, &lt;code&gt;dockur/windows-arm&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The project is explicit about licensing: it does not distribute Windows, the keys in its code are Microsoft&amp;#8217;s generic installation keys, and &lt;strong&gt;you need your own valid license&lt;/strong&gt;. A running container is not an activated&amp;nbsp;Windows.&lt;/p&gt;
&lt;h2 id="what-was-left-in-the-storage-directory"&gt;What Was Left in the Storage&amp;nbsp;Directory&lt;/h2&gt;
&lt;p&gt;WinBoat had pointed the&amp;nbsp;container&amp;#8217;s &lt;code&gt;/storage&lt;/code&gt; at &lt;code&gt;~/winboat&lt;/code&gt;. This is what survived my&amp;nbsp;cleanup:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;File&lt;/th&gt;
&lt;th&gt;Size&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;data.img&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;64 GiB apparent, 18 GiB on disk&lt;/td&gt;
&lt;td&gt;The Windows system disk, a sparse raw image&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;windows.rom&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;3.5 MiB&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;UEFI&lt;/span&gt; firmware code&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;windows.vars&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;528 KiB&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;UEFI&lt;/span&gt; variables, including the boot entries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;windows.mac&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;18 bytes&lt;/td&gt;
&lt;td&gt;The guest&amp;#8217;s &lt;span class="caps"&gt;MAC&lt;/span&gt; address, so Windows sees the same &lt;span class="caps"&gt;NIC&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;windows.base&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;13 bytes&lt;/td&gt;
&lt;td&gt;The &lt;span class="caps"&gt;ISO&lt;/span&gt; the install came from,&amp;nbsp;here &lt;code&gt;win11x64.iso&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;windows.ver&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;5 bytes&lt;/td&gt;
&lt;td&gt;The image version that did the&amp;nbsp;install, &lt;code&gt;6.05&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;windows.boot&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;0 bytes&lt;/td&gt;
&lt;td&gt;Marker that installation finished&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;That&amp;nbsp;empty &lt;code&gt;windows.boot&lt;/code&gt; file matters most. The install script checks for it&amp;nbsp;(&lt;code&gt;hasCompletedInstall&lt;/code&gt; in &lt;code&gt;/run/install.sh&lt;/code&gt;) and, if it is there, skips the download and Setup and boots the disk. Delete the directory and you lose Windows. Delete the container and the image and you lose nothing but a&amp;nbsp;pull.&lt;/p&gt;
&lt;p&gt;The original WinBoat compose file was also still&amp;nbsp;in &lt;code&gt;~/.winboat/podman-compose.yml&lt;/code&gt;, so I knew the exact image tag and&amp;nbsp;settings: &lt;code&gt;ghcr.io/dockur/windows:6.05&lt;/code&gt;, &lt;code&gt;VERSION=11&lt;/code&gt;, 14G of &lt;span class="caps"&gt;RAM&lt;/span&gt;, four cores and a 64G disk. It also contained my Windows password in plain text, which is worth knowing if you back that directory up&amp;nbsp;somewhere.&lt;/p&gt;
&lt;h2 id="bringing-it-back-with-one-command"&gt;Bringing It Back With One&amp;nbsp;Command&lt;/h2&gt;
&lt;p&gt;Both &lt;code&gt;podman ps -a&lt;/code&gt; and &lt;code&gt;podman images&lt;/code&gt; were empty. I pulled the same tag WinBoat had used. The resulting image &lt;span class="caps"&gt;ID&lt;/span&gt; was identical to the one in WinBoat&amp;#8217;s install log, which is a nice confirmation that I was really getting the same&amp;nbsp;bits:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;ghcr.io/dockur/windows:6.05
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Version &lt;code&gt;6.06&lt;/code&gt; and &lt;code&gt;latest&lt;/code&gt; exist already. I deliberately stayed on the version that installed this &lt;span class="caps"&gt;VM&lt;/span&gt;, because &amp;#8220;get it working again&amp;#8221; and &amp;#8220;upgrade it&amp;#8221; are two separate&amp;nbsp;changes.&lt;/p&gt;
&lt;p&gt;Then:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;windows&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;VERSION&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;11&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;RAM_SIZE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;14G&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;CPU_CORES&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;DISK_SIZE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;64G&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--device&lt;span class="w"&gt; &lt;/span&gt;/dev/kvm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;127&lt;/span&gt;.0.0.1:8006:8006&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;127&lt;/span&gt;.0.0.1:3389:3389/tcp&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;127&lt;/span&gt;.0.0.1:3389:3389/udp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;~/winboat:/storage:Z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--stop-timeout&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;120&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;ghcr.io/dockur/windows:6.05
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few notes on the flags, because each one is there for a&amp;nbsp;reason:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;/dev/kvm&lt;/code&gt;&lt;/strong&gt; is the whole point. Without it, the image does not quietly fall back to software&amp;nbsp;emulation: &lt;code&gt;init.sh&lt;/code&gt; reports that &lt;span class="caps"&gt;KVM&lt;/span&gt; is unavailable and exits with code 88. Software emulation needs an&amp;nbsp;explicit &lt;code&gt;KVM=N&lt;/code&gt;, and the image itself warns that this is about ten times slower. On my Fedora installation the device is world-accessible&amp;nbsp;(&lt;code&gt;crw-rw-rw-&lt;/code&gt;), so my rootless container can open&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ports&amp;nbsp;on &lt;code&gt;127.0.0.1&lt;/code&gt;&lt;/strong&gt; keep the web viewer and &lt;span class="caps"&gt;RDP&lt;/span&gt; off the network. WinBoat did the same in my installation, but with random host ports. I wanted fixed&amp;nbsp;ones.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;:Z&lt;/code&gt;&lt;/strong&gt; relabels the storage directory for SELinux with a label private to this container. WinBoat&amp;nbsp;used &lt;code&gt;privileged: true&lt;/code&gt;, which switches labeling off entirely. I would rather not run a &lt;span class="caps"&gt;VM&lt;/span&gt; container privileged when a label does the&amp;nbsp;job.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No &lt;code&gt;NET_ADMIN&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;/dev/net/tun&lt;/code&gt;.&lt;/strong&gt; The &lt;span class="caps"&gt;README&lt;/span&gt;&amp;#8217;s example includes both, and so did WinBoat. In the 6.05&amp;nbsp;scripts, &lt;code&gt;/dev/net/tun&lt;/code&gt; is only used by &lt;span class="caps"&gt;NAT&lt;/span&gt; setup,&amp;nbsp;and &lt;code&gt;NET_ADMIN&lt;/code&gt; by &lt;span class="caps"&gt;NAT&lt;/span&gt;, macvtap and iptables. None of that happens in rootless mode (more on that below). I tested without them: the guest booted in passt mode, &lt;span class="caps"&gt;RDP&lt;/span&gt; login worked and Windows had network traffic. With rootful Podman or &lt;span class="caps"&gt;NAT&lt;/span&gt; networking you need them&amp;nbsp;again.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;--stop-timeout 120&lt;/code&gt;&lt;/strong&gt; gives Windows up to two minutes to shut down after the container&amp;#8217;s entry script sends an &lt;span class="caps"&gt;ACPI&lt;/span&gt; shutdown through the &lt;span class="caps"&gt;QEMU&lt;/span&gt; monitor. Podman&amp;#8217;s default is ten seconds&amp;nbsp;before &lt;code&gt;SIGKILL&lt;/code&gt;. My idle guest needed about six seconds&amp;nbsp;for &lt;code&gt;podman stop&lt;/code&gt;, but a Windows that is installing updates will need far&amp;nbsp;more.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;USERNAME&lt;/code&gt; and &lt;code&gt;PASSWORD&lt;/code&gt;&lt;/strong&gt; are missing on purpose. They feed the unattended installer. On an installed disk Windows already has its&amp;nbsp;accounts.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Twenty seconds later the log&amp;nbsp;said:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;❯ Starting Windows for Podman v6.05...
❯ Warning: you are using the BTRFS filesystem for /storage, this might introduce issues with Windows Setup!
❯ Guest: 192.168.4.12 (Windows)  |  Mode: User (passt)  |  MAC: 02:CA:4A:33:6E:F4
❯ Booting Windows using QEMU v11.1.0...
BdsDxe: starting Boot0005 &amp;quot;Windows Boot Manager&amp;quot; from HD(1,GPT,...)/\EFI\Microsoft\Boot\bootmgfw.efi
❯ Windows started successfully, visit http://127.0.0.1:8006/ to view the screen...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;BTRFS&lt;/span&gt; warning is about Setup. My install was done weeks ago on the same Btrfs filesystem and the existing disk booted fine. If you are starting fresh on Btrfs, take that warning seriously before the first&amp;nbsp;install.&lt;/p&gt;
&lt;h2 id="networknat-silently-became-passt"&gt;&lt;span class="caps"&gt;NETWORK&lt;/span&gt;=nat Silently Became&amp;nbsp;passt&lt;/h2&gt;
&lt;p&gt;The line that caught my eye&amp;nbsp;was &lt;code&gt;Mode: User (passt)&lt;/code&gt;. My first run had actually&amp;nbsp;copied &lt;code&gt;-e NETWORK=nat&lt;/code&gt; and &lt;code&gt;--cap-add NET_RAW&lt;/code&gt; straight from WinBoat&amp;#8217;s compose file, and there was even&amp;nbsp;a &lt;code&gt;podman-compose.yml.before-nat&lt;/code&gt; next to it. At some point I had switched WinBoat&amp;nbsp;from &lt;code&gt;user&lt;/code&gt; to &lt;code&gt;nat&lt;/code&gt;,&amp;nbsp;added &lt;code&gt;NET_RAW&lt;/code&gt; and &lt;code&gt;/dev/net/tun&lt;/code&gt;, and assumed that changed something. The log disagreed, so I read the scripts. The container detects rootless mode from its &lt;span class="caps"&gt;UID&lt;/span&gt;&amp;nbsp;mapping:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$uid_map&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0 0 4294967295&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nv"&gt;ROOTLESS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;N&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nv"&gt;ROOTLESS&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Y&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;NAT&lt;/span&gt; mode starts by creating a bridge&amp;nbsp;with &lt;code&gt;ip link add ... type bridge&lt;/code&gt;. In my rootless container that fails,&amp;nbsp;and &lt;code&gt;network.sh&lt;/code&gt; returns without printing the error&amp;nbsp;when &lt;code&gt;ROOTLESS&lt;/code&gt; is set. The caller then tears down what it created,&amp;nbsp;sets &lt;code&gt;NETWORK="user"&lt;/code&gt;, and only prints the &amp;#8220;falling back to user-mode networking&amp;#8221; warning if the container is &lt;strong&gt;not&lt;/strong&gt; rootless. In rootless mode the fallback is treated as expected, and the image keeps quiet about&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;So in my rootless setup, that&amp;nbsp;whole &lt;code&gt;before-nat&lt;/code&gt; change had no effect. Windows was on passt before, and it is on passt now. The screenshot at the top shows the result: fastfetch inside Windows&amp;nbsp;reports &lt;code&gt;192.168.4.12/24&lt;/code&gt; as its Ethernet address, the same address the host has&amp;nbsp;on &lt;code&gt;wlp0s20f3&lt;/code&gt;. passt hands the guest a copy of the host&amp;#8217;s address instead of putting it on a network of its own. That is fine for what I do: outbound traffic works, and the published ports reach &lt;span class="caps"&gt;RDP&lt;/span&gt; and the web&amp;nbsp;viewer.&lt;/p&gt;
&lt;p&gt;What it means: Windows has an &lt;span class="caps"&gt;IP&lt;/span&gt; address inside the guest, but not a separately reachable address on the &lt;span class="caps"&gt;LAN&lt;/span&gt;. To expose another Windows service, you need two forwarding layers: the port&amp;nbsp;in &lt;code&gt;USER_PORTS&lt;/code&gt;, so passt forwards it into the guest, and a&amp;nbsp;matching &lt;code&gt;-p&lt;/code&gt; or &lt;code&gt;PublishPort&lt;/code&gt;, so Podman forwards it from the host into the&amp;nbsp;container.&lt;/p&gt;
&lt;p&gt;If you want Windows to get its own &lt;span class="caps"&gt;DHCP&lt;/span&gt; lease from your router, the &lt;span class="caps"&gt;README&lt;/span&gt; documents a macvlan network combined&amp;nbsp;with &lt;code&gt;DHCP=Y&lt;/code&gt;, which switches the guest to macvtap. That is a substantially different setup, and in practice it means rootful container networking. It is also a poor fit for my laptop: macvlan and macvtap on a Wi-Fi client interface&amp;nbsp;like &lt;code&gt;wlp0s20f3&lt;/code&gt; generally do not work, because the access point only accepts frames from the &lt;span class="caps"&gt;MAC&lt;/span&gt; address that associated with&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="checking-that-it-works-without-touching-a-mouse"&gt;Checking That It Works Without Touching a&amp;nbsp;Mouse&lt;/h2&gt;
&lt;p&gt;The web viewer&amp;nbsp;answered:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;-w&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%{http_code}\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;http://127.0.0.1:8006/
&lt;span class="m"&gt;200&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For &lt;span class="caps"&gt;RDP&lt;/span&gt;, an open port only proves that something listens. I wanted to know that my account still logs in. FreeRDP has&amp;nbsp;an &lt;code&gt;/auth-only&lt;/code&gt; mode that runs the &lt;span class="caps"&gt;NLA&lt;/span&gt; handshake and exits without opening a&amp;nbsp;window:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;xfreerdp&lt;span class="w"&gt; &lt;/span&gt;/v:127.0.0.1&lt;span class="w"&gt; &lt;/span&gt;/u:chofsted&lt;span class="w"&gt; &lt;/span&gt;/p:&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PASS&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/cert:ignore&lt;span class="w"&gt; &lt;/span&gt;/auth-only
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;One trap here. FreeRDP 3.32.1 logs the result&amp;nbsp;as &lt;code&gt;Authentication only, exit status 1&lt;/code&gt; on success,&amp;nbsp;at &lt;code&gt;ERROR&lt;/code&gt; level, followed&amp;nbsp;by &lt;code&gt;ERRCONNECT_CONNECT_CANCELLED&lt;/code&gt;. That looks like a failure. It is not. With a deliberately wrong password, the same command fails earlier&amp;nbsp;with &lt;code&gt;CONNECTION_STATE_NLA status STATE_RUN_FAILED&lt;/code&gt; and&amp;nbsp;logs &lt;code&gt;exit status 0&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The logged number is not the process exit code. The&amp;nbsp;shell&amp;#8217;s &lt;code&gt;$?&lt;/code&gt; was the reliable&amp;nbsp;signal:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Password&lt;/th&gt;
&lt;th&gt;Log line&lt;/th&gt;
&lt;th&gt;&lt;code&gt;$?&lt;/code&gt;&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Correct&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Authentication only, exit status 1&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;0&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Wrong&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Authentication only, exit status 0&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;134&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;&lt;code&gt;134&lt;/code&gt; means FreeRDP aborted, not that it exited cleanly with an error. I saw the same result on repeated runs. These are observations with FreeRDP 3.32.1 against this one Windows 11 guest. Do not assume they mean the same thing with other FreeRDP versions or &lt;span class="caps"&gt;RDP&lt;/span&gt; security settings, and always test a wrong password first. The&amp;nbsp;Kerberos &lt;code&gt;Preauthentication failed&lt;/code&gt; lines in between are FreeRDP trying Kerberos first and falling back to &lt;span class="caps"&gt;NTLM&lt;/span&gt;, which is normal for a local&amp;nbsp;account.&lt;/p&gt;
&lt;p&gt;Also, &lt;code&gt;/p:&lt;/code&gt; puts the password on the command line, where other local users can see it in the process list while the command runs. That is fine on a single-user laptop for a quick check, but not something to put in a script on a shared&amp;nbsp;machine.&lt;/p&gt;
&lt;p&gt;For screenshots without any client, there is the &lt;span class="caps"&gt;QEMU&lt;/span&gt; monitor. The container exposes it as a Unix socket,&amp;nbsp;and &lt;code&gt;nc&lt;/code&gt; is in the&amp;nbsp;image:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;windows&lt;span class="w"&gt; &lt;/span&gt;sh&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;echo &amp;quot;screendump /tmp/screen.ppm&amp;quot; | nc -q1 -U /run/shm/monitor.sock&amp;#39;&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;cp&lt;span class="w"&gt; &lt;/span&gt;windows:/tmp/screen.ppm&lt;span class="w"&gt; &lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The same socket&amp;nbsp;accepts &lt;code&gt;sendkey&lt;/code&gt;, which is how I got past the lock screen without opening a viewer. That is a handy trick for scripted checks. It is also a reminder that anyone who&amp;nbsp;can &lt;code&gt;podman exec&lt;/code&gt; into this container can type into your Windows&amp;nbsp;session.&lt;/p&gt;
&lt;p&gt;For actual work, &lt;span class="caps"&gt;RDP&lt;/span&gt; is the way to go. The &lt;span class="caps"&gt;README&lt;/span&gt; is honest that the web viewer is mainly meant for watching the installation: it is less responsive and has no clipboard sharing. WinBoat itself launched the full desktop&amp;nbsp;with &lt;code&gt;xfreerdp ... +clipboard /sound:sys:pulse /microphone:sys:pulse /floatbar +f&lt;/code&gt;, and that works just as well without WinBoat against port 3389. On Plasma, &lt;span class="caps"&gt;KRDC&lt;/span&gt; with clipboard sharing turned on is what you see in the screenshot at the&amp;nbsp;top.&lt;/p&gt;
&lt;h2 id="making-it-permanent-with-a-quadlet"&gt;Making It Permanent With a&amp;nbsp;Quadlet&lt;/h2&gt;
&lt;p&gt;A &lt;code&gt;podman run&lt;/code&gt; in a terminal does not survive a reboot. Since I no longer want WinBoat to own the container&amp;#8217;s life cycle, a &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Quadlet&lt;/a&gt; unit is the natural home. This goes&amp;nbsp;in &lt;code&gt;~/.config/containers/systemd/windows.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Windows 11 in a dockur/windows container&lt;/span&gt;

&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;windows&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;ghcr.io/dockur/windows:6.05&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;VERSION=11 RAM_SIZE=14G CPU_CORES=4 DISK_SIZE=64G&lt;/span&gt;
&lt;span class="na"&gt;AddDevice&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/dev/kvm&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;127.0.0.1:8006:8006&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;127.0.0.1:3389:3389/tcp&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;127.0.0.1:3389:3389/udp&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;%h/winboat:/storage:Z&lt;/span&gt;
&lt;span class="na"&gt;StopTimeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;120&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;TimeoutStopSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;150&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;TimeoutStopSec&lt;/code&gt; has to be longer&amp;nbsp;than &lt;code&gt;StopTimeout&lt;/code&gt;, otherwise systemd kills the unit while Windows is still shutting down. I checked the unit with the Quadlet generator&amp;#8217;s dry run, which produces the&amp;nbsp;expected &lt;code&gt;podman run ... --stop-timeout 120 ...&lt;/code&gt; line:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/usr/libexec/podman/quadlet&lt;span class="w"&gt; &lt;/span&gt;-dryrun&lt;span class="w"&gt; &lt;/span&gt;-user
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There is already a container&amp;nbsp;called &lt;code&gt;windows&lt;/code&gt; from the&amp;nbsp;manual &lt;code&gt;podman run&lt;/code&gt;. The generated service&amp;nbsp;runs &lt;code&gt;podman run --replace&lt;/code&gt;, so starting it would remove that container, but I would not let that happen implicitly while a Windows &lt;span class="caps"&gt;VM&lt;/span&gt; is running. A controlled handover is&amp;nbsp;better:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt; &lt;/span&gt;windows
podman&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;windows

systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;daemon-reload
systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;windows.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;windows.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is safe because the &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;#8217;s state lives&amp;nbsp;in &lt;code&gt;~/winboat&lt;/code&gt;, not in the container&amp;#8217;s writable&amp;nbsp;layer.&lt;/p&gt;
&lt;p&gt;The unit has&amp;nbsp;no &lt;code&gt;[Install]&lt;/code&gt; section on purpose. I start Windows when I need it, not a 14 GiB &lt;span class="caps"&gt;VM&lt;/span&gt; every time I log in. If you want it to start automatically,&amp;nbsp;add:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;default.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The Quadlet generator reads that section&amp;nbsp;at &lt;code&gt;daemon-reload&lt;/code&gt;. You do not&amp;nbsp;run &lt;code&gt;systemctl --user enable&lt;/code&gt; on a generated unit. For a rootless Quadlet, &amp;#8220;automatically&amp;#8221; means whenever your user&amp;#8217;s systemd manager starts. That is usually at login, or at boot if lingering is enabled&amp;nbsp;with &lt;code&gt;loginctl enable-linger&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="what-it-uses-ram-cpu-and-disk"&gt;What It Uses: &lt;span class="caps"&gt;RAM&lt;/span&gt;, &lt;span class="caps"&gt;CPU&lt;/span&gt; and&amp;nbsp;Disk&lt;/h2&gt;
&lt;p&gt;Measured on my laptop, an Intel Core Ultra 7 165U with 32 GiB of &lt;span class="caps"&gt;RAM&lt;/span&gt;, shortly after&amp;nbsp;boot:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Resource&lt;/th&gt;
&lt;th&gt;Value&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Memory&lt;/td&gt;
&lt;td&gt;15.1 &lt;span class="caps"&gt;GB&lt;/span&gt;&amp;nbsp;in &lt;code&gt;podman stats&lt;/code&gt;,&amp;nbsp;with &lt;code&gt;RAM_SIZE=14G&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;Busy right after boot, as Windows always is&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Disk&lt;/td&gt;
&lt;td&gt;18 GiB actually used of a 64 GiB sparse image&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Host ports&lt;/td&gt;
&lt;td&gt;8006 and 3389, bound to localhost&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The memory figure is the one to plan around. &lt;span class="caps"&gt;QEMU&lt;/span&gt; allocates guest memory as Windows touches it, and Windows touches a lot of it. The &lt;span class="caps"&gt;README&lt;/span&gt; describes a memory ballooning option for dynamic allocation; I have not tried it&amp;nbsp;here.&lt;/p&gt;
&lt;h2 id="winboat-or-plain-dockurrwindows"&gt;WinBoat or Plain&amp;nbsp;dockurr/windows?&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;/th&gt;
&lt;th&gt;WinBoat&lt;/th&gt;
&lt;th&gt;Plain Podman&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Install&lt;/td&gt;
&lt;td&gt;Guided &lt;span class="caps"&gt;GUI&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;Write the command or Quadlet&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Windows apps as Linux windows&lt;/td&gt;
&lt;td&gt;Yes, via FreeRDP RemoteApp and a guest agent&lt;/td&gt;
&lt;td&gt;Not out of the box; full desktop by default&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Who manages the container&lt;/td&gt;
&lt;td&gt;WinBoat,&amp;nbsp;via &lt;code&gt;podman compose&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;You, or systemd&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Recovers from a broken frontend&lt;/td&gt;
&lt;td&gt;Not on its own&lt;/td&gt;
&lt;td&gt;Independent of WinBoat; direct access to the &lt;span class="caps"&gt;VM&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Ports&lt;/td&gt;
&lt;td&gt;Random localhost ports in my install&lt;/td&gt;
&lt;td&gt;Whatever you choose&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;WinBoat adds real value: an app launcher, integration of individual Windows programs into the Linux desktop, and a guest &lt;span class="caps"&gt;API&lt;/span&gt; that it installs through the&amp;nbsp;container&amp;#8217;s &lt;code&gt;/oem&lt;/code&gt; hook. If that is what you want, use it. But it is worth understanding that everything valuable, your Windows installation, sits in one directory that the base container can boot on its own. That knowledge turned &amp;#8220;reinstall Windows&amp;#8221; into a five-minute fix for&amp;nbsp;me.&lt;/p&gt;
&lt;h2 id="caveats"&gt;Caveats&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;License.&lt;/strong&gt; dockurr/windows installs Windows. It does not license&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No &lt;span class="caps"&gt;GPU&lt;/span&gt; acceleration in this configuration.&lt;/strong&gt; I use the standard virtual display and &lt;span class="caps"&gt;RDP&lt;/span&gt;, without &lt;span class="caps"&gt;GPU&lt;/span&gt; passthrough or the&amp;nbsp;image&amp;#8217;s &lt;code&gt;GPU&lt;/code&gt; option, which is off by default. Desktop applications are the target here, not games or &lt;span class="caps"&gt;GPU&lt;/span&gt;-heavy&amp;nbsp;software.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;One container per disk.&lt;/strong&gt; Never start two containers against the&amp;nbsp;same &lt;code&gt;data.img&lt;/code&gt; at the same time, for example WinBoat&amp;#8217;s and your own. &lt;span class="caps"&gt;QEMU&lt;/span&gt;&amp;#8217;s image locking should make the second one fail with a lock error, but that is not something to test on the only copy of your Windows&amp;nbsp;disk. &lt;code&gt;:Z&lt;/code&gt; adds a smaller problem: each container started&amp;nbsp;with &lt;code&gt;:Z&lt;/code&gt; relabels the directory for itself, so a previously started non-privileged container can lose access to its own&amp;nbsp;storage.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Back up offline.&lt;/strong&gt; Shut Windows down before&amp;nbsp;copying &lt;code&gt;~/winboat&lt;/code&gt;. A live copy&amp;nbsp;of &lt;code&gt;data.img&lt;/code&gt; and &lt;code&gt;windows.vars&lt;/code&gt; is not guaranteed to be a consistent &lt;span class="caps"&gt;VM&lt;/span&gt; state. A Btrfs snapshot is quick, but taken while the &lt;span class="caps"&gt;VM&lt;/span&gt; runs it is only&amp;nbsp;crash-consistent.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rootless means user-mode networking.&lt;/strong&gt; As shown&amp;nbsp;above, &lt;code&gt;NETWORK=nat&lt;/code&gt; does not do what the name suggests in rootless Podman, and the image does not tell&amp;nbsp;you.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Credentials in compose&amp;nbsp;files.&lt;/strong&gt; &lt;code&gt;USERNAME&lt;/code&gt; and &lt;code&gt;PASSWORD&lt;/code&gt; end up in plain text in whatever file starts the container. Rotate the Windows password after installation if that file gets shared or backed&amp;nbsp;up.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The monitor socket is&amp;nbsp;powerful.&lt;/strong&gt; &lt;code&gt;podman exec&lt;/code&gt; access equals keyboard access to Windows and the ability to dump its&amp;nbsp;screen.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Pin your version.&lt;/strong&gt; I&amp;nbsp;kept &lt;code&gt;6.05&lt;/code&gt; on purpose. Upgrading the container image is probably fine, since the scripts include migrations for older state files, but do it as a separate step with a backup&amp;nbsp;of &lt;code&gt;data.img&lt;/code&gt; and &lt;code&gt;windows.vars&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The fix was boring, and that is the compliment. dockurr/windows keeps the whole &lt;span class="caps"&gt;VM&lt;/span&gt; in a plain directory, uses a marker file to remember that installation is done, and boots that state with any compatible image. WinBoat is a comfortable layer on top, but when the layer breaks, the machine underneath is&amp;nbsp;one &lt;code&gt;podman run&lt;/code&gt; away.&lt;/p&gt;
&lt;p&gt;The container was disposable. The &lt;span class="caps"&gt;VM&lt;/span&gt; was not. Keeping that distinction in mind made the recovery&amp;nbsp;trivial.&lt;/p&gt;
&lt;p&gt;What I took away besides a working Windows: read the startup log line about the network mode. In my case, a configuration change I had made weeks ago had never taken effect, and only that one line said&amp;nbsp;so.&lt;/p&gt;</content><category term="Linux"/><category term="podman"/><category term="windows"/><category term="qemu"/><category term="kvm"/><category term="dockurr"/><category term="winboat"/><category term="rdp"/><category term="quadlet"/><category term="rootless"/><category term="fedora"/></entry><entry><title>Self-Hosted AI: Qwen 3.8 on a Rented Blackwell at 150 Tokens per Second</title><link href="https://hofstede.it/self-hosted-ai-qwen-38-on-a-rented-blackwell-at-150-tokens-per-second/" rel="alternate"/><published>2026-10-03T00:00:00+02:00</published><updated>2026-10-03T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-10-03:/self-hosted-ai-qwen-38-on-a-rented-blackwell-at-150-tokens-per-second/</id><summary type="html">&lt;p&gt;An open-weight Qwen 3.8 27B model, an &lt;span class="caps"&gt;RTX&lt;/span&gt; &lt;span class="caps"&gt;PRO&lt;/span&gt; 6000 Blackwell rented by the hour, and one shell command to start it. For most of my everyday coding it is good enough, it answers at roughly 150 tokens per second, and my prompts stay on infrastructure I chose. Some honest caveats about what &amp;#8220;self-hosted&amp;#8221; means on someone else&amp;#8217;s &lt;span class="caps"&gt;GPU&lt;/span&gt; are included, along with the workflow that has worked best for me: a frontier model for architecture, and a fast open-weight model for the token-heavy implementation&amp;nbsp;loop.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Terminal output showing nvidia-smi on an RTX PRO 6000 Blackwell and a haiku generated at 154 tokens per second" src="https://hofstede.it/images/2026-10-03-self-hosted-ai-qwen38-runpod.png" title="Graphics cards vanish. The pod rents one anyway."&gt;&lt;/p&gt;
&lt;p&gt;For a long time, &amp;#8220;self-hosted &lt;span class="caps"&gt;AI&lt;/span&gt;&amp;#8221; meant one of two things to me. Either a small model on a laptop, slow enough to make coffee between answers, or a lot of money spent on a &lt;span class="caps"&gt;GPU&lt;/span&gt; that then spends most of its life idling under my&amp;nbsp;desk.&lt;/p&gt;
&lt;p&gt;Somewhere in the last few months, that changed. Open-weight models got good. Not &amp;#8220;good for an open model&amp;#8221;.&amp;nbsp;Good.&lt;/p&gt;
&lt;p&gt;My current setup: &lt;strong&gt;Qwen 3.8 27B&lt;/strong&gt; on a rented &lt;strong&gt;&lt;span class="caps"&gt;NVIDIA&lt;/span&gt; &lt;span class="caps"&gt;RTX&lt;/span&gt; &lt;span class="caps"&gt;PRO&lt;/span&gt; 6000 Blackwell&lt;/strong&gt; at RunPod, started with one command, answering at around &lt;strong&gt;150 tokens per second&lt;/strong&gt;, and torn down again when I am done. The screenshot above is from a running pod. It wrote a haiku about the &lt;span class="caps"&gt;GPU&lt;/span&gt; shortage while occupying one of the GPUs in&amp;nbsp;question.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#good-enough-is-now-really-good"&gt;Good Enough Is Now Really&amp;nbsp;Good&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-stack"&gt;The Stack&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#why-this-particular-card"&gt;Why This Particular&amp;nbsp;Card&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-silent-trap-mixed-precision-checkpoints"&gt;The Silent Trap: Mixed-Precision&amp;nbsp;Checkpoints&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#one-command"&gt;One&amp;nbsp;Command&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#how-fast-is-fast"&gt;How Fast Is&amp;nbsp;Fast?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#data-sovereignty-the-part-i-care-about-most"&gt;Data Sovereignty: The Part I Care About Most&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#where-the-sovereignty-ends"&gt;Where the Sovereignty&amp;nbsp;Ends&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-it-costs"&gt;What It Costs&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-forgotten-pod-problem"&gt;The Forgotten Pod&amp;nbsp;Problem&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#frontier-architect-open-weight-builder"&gt;Frontier Architect, Open-Weight&amp;nbsp;Builder&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-i-changed"&gt;What I&amp;nbsp;Changed&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="good-enough-is-now-really-good"&gt;Good Enough Is Now Really&amp;nbsp;Good&lt;/h2&gt;
&lt;p&gt;Let me get the opinion part out of the way first, because everything else depends on&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;After testing Qwen 3.8 on real work, my conclusion is: &lt;strong&gt;for the overwhelming majority of my everyday coding tasks, it is sufficient.&lt;/strong&gt; Shell scripts, Ansible roles, Python tooling, refactoring, writing tests, explaining an unfamiliar codebase, fixing the thing that broke after a dependency update. That kind of&amp;nbsp;work.&lt;/p&gt;
&lt;p&gt;Today&amp;#8217;s frontier models, such as Claude Opus 5.5 or &lt;span class="caps"&gt;GPT&lt;/span&gt;-6, still produce better results. I notice it on hard architectural questions, on long chains of reasoning across many files, and on problems where the first plausible answer is wrong. But for most of what I do on an ordinary Tuesday, the difference has stopped being relevant. The gap is real. It just no longer sits where most of my work&amp;nbsp;happens.&lt;/p&gt;
&lt;p&gt;That is a statement about my workload, not a benchmark. Your mileage will depend on what you&amp;nbsp;build.&lt;/p&gt;
&lt;h2 id="the-stack"&gt;The&amp;nbsp;Stack&lt;/h2&gt;
&lt;p&gt;I did not write this from scratch. The foundation is &lt;a href="https://github.com/nicremo/qwen38-runpod-stack"&gt;qwen38-runpod-stack&lt;/a&gt; by nicremo, &lt;span class="caps"&gt;MIT&lt;/span&gt;-licensed, which already solved most of the hard parts. I adapted it to my needs, more on that&amp;nbsp;below.&lt;/p&gt;
&lt;p&gt;The moving&amp;nbsp;parts:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;Choice&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Model&lt;/td&gt;
&lt;td&gt;Qwen3.8-27B, the huihui-ai abliterated variant, &lt;a href="https://huggingface.co/sakamakismile/Huihui-Qwen3.8-27B-abliterated-NVFP4"&gt;quantized to &lt;span class="caps"&gt;NVFP4&lt;/span&gt; by sakamakismile&lt;/a&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inference server&lt;/td&gt;
&lt;td&gt;&lt;a href="https://github.com/sgl-project/sglang"&gt;SGLang&lt;/a&gt;, OpenAI-compatible &lt;span class="caps"&gt;API&lt;/span&gt; on port 8000&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Speculative decoding&lt;/td&gt;
&lt;td&gt;DFlash2, using&amp;nbsp;the &lt;code&gt;incoai/Qwen3.8-27B-DFlash2&lt;/code&gt; draft model&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Context&lt;/td&gt;
&lt;td&gt;262,144 tokens&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;GPU&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;RTX&lt;/span&gt; &lt;span class="caps"&gt;PRO&lt;/span&gt; 6000 Blackwell Server Edition, 96 &lt;span class="caps"&gt;GB&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Frontends&lt;/td&gt;
&lt;td&gt;OpenWebUI on port 8080, or any OpenAI-compatible coding agent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Lifecycle&lt;/td&gt;
&lt;td&gt;&lt;code&gt;runpodctl&lt;/code&gt;, wrapped in a timer-enforcing shell script&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The base model is Apache 2.0. &amp;#8220;Abliterated&amp;#8221; means the weights were modified after training to suppress much of the model&amp;#8217;s refusal behavior. The stack uses that build by default. If you prefer the stock&amp;nbsp;alignment, &lt;code&gt;--model&lt;/code&gt; accepts any &lt;span class="caps"&gt;NVFP4&lt;/span&gt; checkpoint in the right format (see the trap&amp;nbsp;below).&lt;/p&gt;
&lt;h3 id="why-this-particular-card"&gt;Why This Particular&amp;nbsp;Card&lt;/h3&gt;
&lt;p&gt;Three things drive the &lt;span class="caps"&gt;GPU&lt;/span&gt;&amp;nbsp;choice:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;NVFP4&lt;/span&gt; needs Blackwell.&lt;/strong&gt; The 4-bit floating point format runs on Blackwell&amp;#8217;s &lt;span class="caps"&gt;FP4&lt;/span&gt; tensor cores. An H100 or H200 does not have native &lt;span class="caps"&gt;FP4&lt;/span&gt; tensor cores, so SGLang cannot use the Blackwell &lt;span class="caps"&gt;NVFP4&lt;/span&gt; execution path and falls back to a different kernel path, no matter how expensive the card is per&amp;nbsp;hour.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;96 &lt;span class="caps"&gt;GB&lt;/span&gt; of &lt;span class="caps"&gt;VRAM&lt;/span&gt;.&lt;/strong&gt; An &lt;span class="caps"&gt;RTX&lt;/span&gt; 5090 shares the same &lt;span class="caps"&gt;GB202&lt;/span&gt; chip and similar memory bandwidth, but with 32 &lt;span class="caps"&gt;GB&lt;/span&gt; the context has to be cut down significantly. The &lt;span class="caps"&gt;PRO&lt;/span&gt; 6000 fits the full 262K&amp;nbsp;context.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;It is the validated configuration.&lt;/strong&gt; The SGLang cookbook for Qwen3.8-27B covers this card end to&amp;nbsp;end.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;In the screenshot, SGLang occupies about 87 &lt;span class="caps"&gt;GB&lt;/span&gt; of the 96 &lt;span class="caps"&gt;GB&lt;/span&gt;. Most of it is &lt;span class="caps"&gt;KV&lt;/span&gt; cache and state, reserved up front&amp;nbsp;by &lt;code&gt;--mem-fraction-static 0.85&lt;/code&gt;. The weights themselves are only about 20.6 &lt;span class="caps"&gt;GB&lt;/span&gt;, plus 3.8 &lt;span class="caps"&gt;GB&lt;/span&gt; for the draft&amp;nbsp;model.&lt;/p&gt;
&lt;h3 id="the-silent-trap-mixed-precision-checkpoints"&gt;The Silent Trap: Mixed-Precision&amp;nbsp;Checkpoints&lt;/h3&gt;
&lt;p&gt;This one is worth knowing even if you never touch RunPod. One commonly used abliterated &lt;span class="caps"&gt;NVFP4&lt;/span&gt; build of this model uses the compressed-tensors&amp;nbsp;format &lt;code&gt;mixed-precision&lt;/code&gt;. SGLang loads such checkpoints &lt;strong&gt;completely unquantized, without a warning&lt;/strong&gt; (&lt;a href="https://github.com/sgl-project/sglang/issues/32736"&gt;sgl-project/sglang#32736&lt;/a&gt;). The model runs, the answers are fine, and the entire &lt;span class="caps"&gt;FP4&lt;/span&gt; speedup is&amp;nbsp;gone.&lt;/p&gt;
&lt;p&gt;The stack therefore uses a&amp;nbsp;uniform &lt;code&gt;nvfp4-pack-quantized&lt;/code&gt; checkpoint. If your &lt;span class="caps"&gt;NVFP4&lt;/span&gt; model feels as slow as &lt;span class="caps"&gt;BF16&lt;/span&gt;, check&amp;nbsp;the &lt;code&gt;format&lt;/code&gt; field in&amp;nbsp;its &lt;code&gt;config.json&lt;/code&gt; first.&lt;/p&gt;
&lt;h2 id="one-command"&gt;One&amp;nbsp;Command&lt;/h2&gt;
&lt;p&gt;Starting a session looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;qwen38fast&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="c1"&gt;# RTX PRO 6000, DFlash2, 4h window, OpenWebUI&lt;/span&gt;
qwen38fast&lt;span class="w"&gt; &lt;/span&gt;2h&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# shorter window&lt;/span&gt;
qwen38fast&lt;span class="w"&gt; &lt;/span&gt;--pi&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="c1"&gt;# API only, no web UI, ready sooner&lt;/span&gt;
qwen38fast&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt;       &lt;/span&gt;&lt;span class="c1"&gt;# what is running, current spend per hour&lt;/span&gt;
qwen38fast&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="c1"&gt;# terminate everything&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Under the hood, the&amp;nbsp;script:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Generates a random &lt;span class="caps"&gt;API&lt;/span&gt; key on the first run and stores it&amp;nbsp;in &lt;code&gt;~/.runpod/qwen38.key&lt;/code&gt; with&amp;nbsp;mode &lt;code&gt;0600&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Base64-encodes a bootstrap script and passes it, together with model &lt;span class="caps"&gt;ID&lt;/span&gt;, speculative decoding method, context length, and &lt;span class="caps"&gt;API&lt;/span&gt; key, as environment variables to a RunPod&amp;nbsp;template.&lt;/li&gt;
&lt;li&gt;Creates the pod&amp;nbsp;through &lt;code&gt;rp&lt;/code&gt;,&amp;nbsp;a &lt;code&gt;runpodctl&lt;/code&gt; wrapper that refuses to create a pod without a termination&amp;nbsp;timer.&lt;/li&gt;
&lt;li&gt;Waits until the &lt;span class="caps"&gt;API&lt;/span&gt; lists the model, then waits for OpenWebUI, then opens it in the&amp;nbsp;browser.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Inside the pod, the bootstrap script downloads the weights from Hugging Face, writes an SGLang start script, and launches&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/opt/sglang/bin/python3&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;sglang.launch_server&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--model-path&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$MODEL_PATH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--served-model-name&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$SERVED_NAME&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--context-length&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$MAX_LEN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--mem-fraction-static&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$MEM_FRAC&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--attention-backend&lt;span class="w"&gt; &lt;/span&gt;flashinfer&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--reasoning-parser&lt;span class="w"&gt; &lt;/span&gt;qwen3&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--tool-call-parser&lt;span class="w"&gt; &lt;/span&gt;qwen3_coder&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--mamba-full-memory-ratio&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$MAMBA_RATIO&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--mamba-ssm-dtype&lt;span class="w"&gt; &lt;/span&gt;float32&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--speculative-algorithm&lt;span class="w"&gt; &lt;/span&gt;DFLASH&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--speculative-draft-model-path&lt;span class="w"&gt; &lt;/span&gt;incoai/Qwen3.8-27B-DFlash2&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--speculative-num-draft-tokens&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--api-key&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$API_KEY&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--host&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;.0.0.0&lt;span class="w"&gt; &lt;/span&gt;--port&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8000&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(Shortened. The real script assembles the speculative decoding flags depending&amp;nbsp;on &lt;code&gt;--spec&lt;/code&gt;.)&lt;/p&gt;
&lt;p&gt;From &lt;code&gt;qwen38fast&lt;/code&gt; to a usable model takes roughly 8 to 15 minutes. Most of that is the container image pull and 20 &lt;span class="caps"&gt;GB&lt;/span&gt; of weights. Not instant, but it fits comfortably into &amp;#8220;start it, then read the ticket&amp;nbsp;properly&amp;#8221;.&lt;/p&gt;
&lt;h2 id="how-fast-is-fast"&gt;How Fast Is&amp;nbsp;Fast?&lt;/h2&gt;
&lt;p&gt;The numbers in the screenshot: &lt;strong&gt;154 tokens per second&lt;/strong&gt;, 837 tokens in 5.44 seconds, and &lt;strong&gt;0.11 seconds to the first token&lt;/strong&gt;, with reasoning enabled. Most of those 837 tokens were thinking. The haiku itself is short. It is also, I would argue,&amp;nbsp;accurate.&lt;/p&gt;
&lt;p&gt;The upstream project measured more systematically: an input of 8,192 tokens, 1,024 output tokens, a single stream, five runs after a warmup. The median was &lt;strong&gt;150.7 tokens per second&lt;/strong&gt;, with individual runs between 122 and&amp;nbsp;156.&lt;/p&gt;
&lt;p&gt;For comparison, the same model on a MacBook Pro M2 Max manages around 14 tokens per second according to the upstream &lt;span class="caps"&gt;README&lt;/span&gt;. That is a perfectly usable chat speed. For an agent that reads twenty files, plans, edits, runs tests, and iterates, 14 tokens per second gets old&amp;nbsp;quickly.&lt;/p&gt;
&lt;p&gt;A substantial part of the speed comes from speculative decoding. A small draft model proposes several tokens ahead, and the large model verifies them in a single pass. The target model can verify several proposed tokens in a single step, and every accepted guess is a token it did not have to generate on its own. DFlash2 drafts eight tokens per step here. The draft model was trained on the base model, not on the abliterated one, so the accept rate is probably a bit below the cookbook figures. It is still the fastest of the three methods the stack&amp;nbsp;supports.&lt;/p&gt;
&lt;p&gt;At 150 tokens per second, the model is faster than I can read. In an agentic loop, the bottleneck shifts from token generation to everything else: running tests, waiting for compilers, and reading what the agent&amp;nbsp;did.&lt;/p&gt;
&lt;h2 id="data-sovereignty-the-part-i-care-about-most"&gt;Data Sovereignty: The Part I Care About&amp;nbsp;Most&lt;/h2&gt;
&lt;p&gt;Speed is nice. Cost is nice. The real reason I bother with any of this is a different&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;When I run the inference server myself, I get to choose which infrastructure receives my data and which software handles&amp;nbsp;it.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;With a hosted model &lt;span class="caps"&gt;API&lt;/span&gt;, every prompt, every file the agent reads, and every snippet of proprietary code leaves infrastructure I operate the moment I press Enter. What happens next is governed by the provider&amp;#8217;s terms of service, retention policies, abuse monitoring, and, depending on the plan, sometimes training opt-outs. Those may all be perfectly reasonable. But they are someone else&amp;#8217;s decisions, enforced by someone else&amp;#8217;s systems, and they can&amp;nbsp;change.&lt;/p&gt;
&lt;p&gt;With my own inference server, the picture&amp;nbsp;changes:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;No separate model &lt;span class="caps"&gt;API&lt;/span&gt; provider sees my prompts.&lt;/strong&gt; The infrastructure provider still sits underneath the workload, but there is no third-party inference &lt;span class="caps"&gt;API&lt;/span&gt; receiving, retaining, or moderating my&amp;nbsp;requests.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;I choose the exact weights.&lt;/strong&gt; The model is a specific checkpoint I selected. It does not get silently swapped, re-aligned, or deprecated overnight. If it behaves a certain way today, it behaves that way&amp;nbsp;tomorrow.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The lifecycle is mine.&lt;/strong&gt; When I&amp;nbsp;run &lt;code&gt;qwen38fast stop&lt;/code&gt;, the pod and its ephemeral container storage are deleted. I do not attach a persistent network volume, so OpenWebUI&amp;#8217;s local chat history does not survive into my next&amp;nbsp;session.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Access is mine.&lt;/strong&gt; The &lt;span class="caps"&gt;API&lt;/span&gt; key is generated locally and only exists on my machine and in the&amp;nbsp;pod.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No dependency on a vendor&amp;#8217;s &lt;span class="caps"&gt;API&lt;/span&gt;.&lt;/strong&gt; No rate limits, no outage of a third-party &lt;span class="caps"&gt;API&lt;/span&gt;, no quota surprises halfway through a refactoring&amp;nbsp;session.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A note on the elephant in the room, since Qwen comes from Alibaba: these are &lt;strong&gt;open weights&lt;/strong&gt;. Running the model does not involve calling an Alibaba inference service. The checkpoint is just weights, loaded locally by SGLang on a &lt;span class="caps"&gt;GPU&lt;/span&gt; I rented. Like every model, Qwen reflects characteristics and biases of its training and alignment. For writing a systemd unit, I have not noticed&amp;nbsp;any.&lt;/p&gt;
&lt;h3 id="where-the-sovereignty-ends"&gt;Where the Sovereignty&amp;nbsp;Ends&lt;/h3&gt;
&lt;p&gt;I want to be honest here, because &amp;#8220;self-hosted&amp;#8221; on a rented &lt;span class="caps"&gt;GPU&lt;/span&gt; is not the same as self-hosted in my own&amp;nbsp;rack.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;RunPod is still the infrastructure provider.&lt;/strong&gt; The pod runs on their platform, and anybody with root on the physical host could, in principle, inspect the container or &lt;span class="caps"&gt;GPU&lt;/span&gt; memory. I trust them more than I would trust an arbitrary endpoint. But I trust them as an infrastructure provider, not as a cryptographic&amp;nbsp;guarantee.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The standard &lt;span class="caps"&gt;HTTPS&lt;/span&gt; endpoint goes through RunPod&amp;#8217;s proxy.&lt;/strong&gt; I access the service&amp;nbsp;as &lt;code&gt;https://&amp;lt;pod-id&amp;gt;-8000.proxy.runpod.net&lt;/code&gt;, so I do not treat that connection as end-to-end &lt;span class="caps"&gt;TLS&lt;/span&gt; terminating inside my&amp;nbsp;container.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Community Cloud is not Secure Cloud.&lt;/strong&gt; They have different infrastructure and compliance characteristics, and I treat Community Cloud as the less appropriate option for sensitive workloads. The script tries Community first because it is cheaper. Where policy permits the workload and its sensitivity warrants it, I select Secure Cloud&amp;nbsp;explicitly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;RUNPOD_CLOUD&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;SECURE&lt;span class="w"&gt; &lt;/span&gt;qwen38fast
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;The region is not pinned.&lt;/strong&gt; The script currently takes whatever data center has a card available. For workloads that require &lt;span class="caps"&gt;EU&lt;/span&gt; data residency, pinning an &lt;span class="caps"&gt;EU&lt;/span&gt; data center would be the next&amp;nbsp;step.&lt;/p&gt;
&lt;p&gt;So I would place this setup on a&amp;nbsp;ladder:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Setup&lt;/th&gt;
&lt;th&gt;Who sees your prompts&lt;/th&gt;
&lt;th&gt;Effort&lt;/th&gt;
&lt;th&gt;Cost model&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Hosted model &lt;span class="caps"&gt;API&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;the &lt;span class="caps"&gt;API&lt;/span&gt; provider&lt;/td&gt;
&lt;td&gt;none&lt;/td&gt;
&lt;td&gt;per token&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Rented &lt;span class="caps"&gt;GPU&lt;/span&gt;, own inference&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;the infrastructure provider, in principle&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;one script&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;per hour&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Own hardware&lt;/td&gt;
&lt;td&gt;no external infrastructure provider&lt;/td&gt;
&lt;td&gt;considerable&lt;/td&gt;
&lt;td&gt;upfront&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The middle rung is where I am. The point is not that RunPod becomes invisible. The point is that the questions about my data change from &amp;#8220;what does this &lt;span class="caps"&gt;AI&lt;/span&gt; vendor do with my prompts?&amp;#8221; to &amp;#8220;do I trust this hosting provider?&amp;#8221;. That is the same question I already answer for every &lt;span class="caps"&gt;VPS&lt;/span&gt; and cloud instance I run. I know how to evaluate&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="what-it-costs"&gt;What It&amp;nbsp;Costs&lt;/h2&gt;
&lt;p&gt;Prices from &lt;a href="https://www.runpod.io/pricing"&gt;RunPod&amp;#8217;s pricing page&lt;/a&gt; as of October 3, 2026, on-demand per&amp;nbsp;hour:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;span class="caps"&gt;GPU&lt;/span&gt;&lt;/th&gt;
&lt;th&gt;&lt;span class="caps"&gt;VRAM&lt;/span&gt;&lt;/th&gt;
&lt;th&gt;Community Cloud&lt;/th&gt;
&lt;th&gt;Secure Cloud&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;RTX&lt;/span&gt; 5090&lt;/td&gt;
&lt;td&gt;32 &lt;span class="caps"&gt;GB&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;$0.69&lt;/td&gt;
&lt;td&gt;$0.99&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;RTX&lt;/span&gt; &lt;span class="caps"&gt;PRO&lt;/span&gt; 6000 &lt;span class="caps"&gt;MIG&lt;/span&gt; slice&lt;/td&gt;
&lt;td&gt;48 &lt;span class="caps"&gt;GB&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;not offered&lt;/td&gt;
&lt;td&gt;$1.09&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;span class="caps"&gt;RTX&lt;/span&gt; &lt;span class="caps"&gt;PRO&lt;/span&gt; 6000&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;96 &lt;span class="caps"&gt;GB&lt;/span&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$1.69&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;&lt;strong&gt;$2.09&lt;/strong&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B200&lt;/td&gt;
&lt;td&gt;180 &lt;span class="caps"&gt;GB&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;$5.98&lt;/td&gt;
&lt;td&gt;$6.79&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;B300&lt;/td&gt;
&lt;td&gt;288 &lt;span class="caps"&gt;GB&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;$6.94&lt;/td&gt;
&lt;td&gt;$7.89&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;A typical four-hour session on the &lt;span class="caps"&gt;PRO&lt;/span&gt; 6000 costs &lt;strong&gt;$6.76&lt;/strong&gt; on Community Cloud or &lt;strong&gt;$8.36&lt;/strong&gt; on Secure Cloud. The 150 &lt;span class="caps"&gt;GB&lt;/span&gt; container disk adds a few cents at $0.10 per &lt;span class="caps"&gt;GB&lt;/span&gt; per month,&amp;nbsp;prorated.&lt;/p&gt;
&lt;p&gt;Running it eight hours a day, every working day, would add up to roughly $335 a month on Secure Cloud. That is not the point of this setup. I start it for focused coding sessions, and the timer ends&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The half-card &lt;span class="caps"&gt;MIG&lt;/span&gt; slice at $1.09 per hour is the budget option I added. It offers roughly half the compute and half the memory bandwidth, so I expect a clearly lower decode speed, though not necessarily exactly half. The context is limited to 131,072 tokens because the &lt;span class="caps"&gt;KV&lt;/span&gt; cache for the full 262K does not fit next to the weights in 48 &lt;span class="caps"&gt;GB&lt;/span&gt;. I have not benchmarked it yet, but even half of 150 beats every laptop I&amp;nbsp;own.&lt;/p&gt;
&lt;h3 id="the-forgotten-pod-problem"&gt;The Forgotten Pod&amp;nbsp;Problem&lt;/h3&gt;
&lt;p&gt;Per-hour billing has one classic failure mode: the pod you forgot about. The upstream author learned it the hard way, with two idle pods running for 24 hours. The stack therefore has three&amp;nbsp;layers:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;--terminate-after&lt;/code&gt;&lt;/strong&gt;: a server-side timer at RunPod. It fires even when my laptop is&amp;nbsp;closed.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;rp&lt;/code&gt;&lt;/strong&gt;: the wrapper that adds this timer to&amp;nbsp;every &lt;code&gt;pod create&lt;/code&gt;, and refuses to create a pod if it cannot calculate the&amp;nbsp;deadline.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;runpod-reaper&lt;/code&gt;&lt;/strong&gt;: a periodic job that terminates every pod older than one hour, unless its name starts&amp;nbsp;with &lt;code&gt;keep-&lt;/code&gt;. &lt;code&gt;rp&lt;/code&gt; adds that prefix automatically when the requested window is longer than one&amp;nbsp;hour.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The third layer exists because the first one cannot be&amp;nbsp;verified: &lt;code&gt;runpodctl pod get&lt;/code&gt; does not show whether a termination timer was stored. Trust, but&amp;nbsp;reap.&lt;/p&gt;
&lt;h2 id="frontier-architect-open-weight-builder"&gt;Frontier Architect, Open-Weight&amp;nbsp;Builder&lt;/h2&gt;
&lt;p&gt;The workflow that has worked best for me combines both&amp;nbsp;worlds.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A frontier model writes the specification.&lt;/strong&gt; For a non-trivial change, I let Claude Opus 5.5 or a comparable model do what it is best at: understanding the problem, weighing the trade-offs, defining the architecture, and writing a precise spec with interfaces, edge cases, and acceptance&amp;nbsp;criteria.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Qwen 3.8 implements it in an agentic loop.&lt;/strong&gt; The spec goes to a coding agent backed by my own pod. It writes the code, runs the tests, reads the failures, and iterates until the acceptance criteria&amp;nbsp;pass.&lt;/p&gt;
&lt;p&gt;This works well for three&amp;nbsp;reasons:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The hard thinking happens once.&lt;/strong&gt; The expensive model spends its tokens on the part where its advantage matters. The iterative grind of edit, test, fix, and repeat consumes most of the tokens in an agentic session, and that runs at a flat hourly&amp;nbsp;rate.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Speed compounds in loops.&lt;/strong&gt; An agent makes many sequential calls. At 150 tokens per second with a 0.1 second time to first token, the iterations are short enough that I can watch the agent work instead of switching to something&amp;nbsp;else.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Less code leaves my environment.&lt;/strong&gt; The frontier model sees the problem description and the parts of the codebase needed to design a solution. The full implementation loop, with every file the agent reads and every test output, stays on the pod. That is not zero exposure, but it is considerably&amp;nbsp;less.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A good spec matters more here than with a frontier model doing the whole job. A 27B model executes a clear plan well. It is less good at noticing that the plan itself is wrong. That is precisely the division of&amp;nbsp;labor.&lt;/p&gt;
&lt;h2 id="what-i-changed"&gt;What I&amp;nbsp;Changed&lt;/h2&gt;
&lt;p&gt;Most of my work was not &lt;span class="caps"&gt;AI&lt;/span&gt; work at all. It was the familiar collection of portability bugs, lifecycle races, dependency collisions, and misleading diagnostics that appears whenever a useful prototype meets daily use. The upstream project is built for macOS, and I run Fedora. My changes, roughly in order of how much time they saved&amp;nbsp;me:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Linux portability.&lt;/strong&gt; &lt;span class="caps"&gt;BSD&lt;/span&gt; &lt;code&gt;date -v +4H&lt;/code&gt; became &lt;span class="caps"&gt;GNU&lt;/span&gt; &lt;code&gt;date -d "+4 hours"&lt;/code&gt; with a fallback, so the scripts work on&amp;nbsp;both. &lt;code&gt;open&lt;/code&gt; became &lt;code&gt;xdg-open&lt;/code&gt;. The reaper runs as a systemd user timer instead of a&amp;nbsp;LaunchAgent:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Timer]&lt;/span&gt;
&lt;span class="na"&gt;OnBootSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5min&lt;/span&gt;
&lt;span class="na"&gt;OnUnitActiveSec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;10min&lt;/span&gt;
&lt;span class="na"&gt;Persistent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;A retry loop for scarce GPUs.&lt;/strong&gt; Blackwell availability changes by the second. The same card type can be refused on one request and available on the next. The original script tried Community Cloud once, then Secure Cloud once, then gave up. Mine alternates between both for up to twelve attempts, 15 seconds apart, and suppresses the expected &amp;#8220;no instances available&amp;#8221; error message while it&amp;nbsp;does.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Honest price output.&lt;/strong&gt; The script used to print only the Community Cloud price. Since Community Cloud is exactly the pool that runs dry first, the price I actually paid was usually the Secure Cloud one. It now prints&amp;nbsp;both.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The &lt;span class="caps"&gt;MIG&lt;/span&gt; option&lt;/strong&gt; described&amp;nbsp;above, &lt;code&gt;--gpu mig48&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A bootstrap that does not depend on &lt;span class="caps"&gt;SSH&lt;/span&gt;.&lt;/strong&gt; OpenWebUI caches the model list at startup, so it has to start after SGLang is ready. The original launcher solved that by connecting to the pod via &lt;span class="caps"&gt;SSH&lt;/span&gt; and restarting OpenWebUI. That never worked: the template overrides the container start command, so the image never&amp;nbsp;started &lt;code&gt;sshd&lt;/code&gt;, and the failed &lt;span class="caps"&gt;SSH&lt;/span&gt; call disappeared&amp;nbsp;into &lt;code&gt;/dev/null&lt;/code&gt;. The bootstrap script now waits for the &lt;span class="caps"&gt;API&lt;/span&gt; itself in a background subshell before starting&amp;nbsp;OpenWebUI.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;OpenWebUI in its own venv.&lt;/strong&gt;&amp;nbsp;Installing &lt;code&gt;open-webui&lt;/code&gt; into SGLang&amp;#8217;s Python environment pulls in dependencies that reinstall PyTorch and &lt;span class="caps"&gt;NCCL&lt;/span&gt;, which&amp;nbsp;removed &lt;code&gt;libnccl.so&lt;/code&gt; from underneath the running inference server. An isolated venv fixes that and allows OpenWebUI to install in parallel with the model&amp;nbsp;download.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Faster&amp;nbsp;downloads.&lt;/strong&gt; &lt;code&gt;hf_transfer&lt;/code&gt; is no longer used by&amp;nbsp;current &lt;code&gt;huggingface_hub&lt;/code&gt; versions, which only log a warning and fall back to the slow path. The bootstrap now&amp;nbsp;uses &lt;code&gt;hf_xet&lt;/code&gt; with &lt;code&gt;HF_XET_HIGH_PERFORMANCE=1&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A network probe that measures the network.&lt;/strong&gt; The original download check&amp;nbsp;fetched &lt;code&gt;config.json&lt;/code&gt; without following redirects and therefore measured the body of an &lt;span class="caps"&gt;HTTP&lt;/span&gt; 307 response. It reported 0.0 &lt;span class="caps"&gt;MB&lt;/span&gt;/s on perfectly healthy pods. The probe now requests a 20 &lt;span class="caps"&gt;MB&lt;/span&gt; range from the largest actual weight file and reports the &lt;span class="caps"&gt;HTTP&lt;/span&gt; status alongside the&amp;nbsp;speed.&lt;/p&gt;
&lt;p&gt;None of these are dramatic. Each one cost somewhere between ten minutes and an&amp;nbsp;evening.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;A year ago, I would have dismissed self-hosted models for serious coding. They were either too slow or too limited, usually&amp;nbsp;both.&lt;/p&gt;
&lt;p&gt;Today, a 27B open-weight model on a rented Blackwell &lt;span class="caps"&gt;GPU&lt;/span&gt; handles most of my daily work, at a speed faster than I can read, for the price of a decent lunch per session. And my prompts and my code go to infrastructure I selected and can reason about, instead of an &lt;span class="caps"&gt;API&lt;/span&gt; whose terms I merely&amp;nbsp;accepted.&lt;/p&gt;
&lt;p&gt;Frontier models still have their place, and I still use them. But &amp;#8220;good enough&amp;#8221; has quietly turned into &amp;#8220;good&amp;#8221;, and for a lot of my work, the question is no longer whether an open model can do it. The question is whether I want to send it somewhere&amp;nbsp;else.&lt;/p&gt;
&lt;p&gt;Mostly, I do&amp;nbsp;not.&lt;/p&gt;</content><category term="Linux"/><category term="ai"/><category term="llm"/><category term="self-hosting"/><category term="qwen"/><category term="runpod"/><category term="sglang"/><category term="nvidia"/><category term="blackwell"/><category term="data-sovereignty"/><category term="linux"/></entry><entry><title>No Proxmox Backup Client for RHEL? Hold My Container Runtime</title><link href="https://hofstede.it/pbs-client-rhel-container/" rel="alternate"/><published>2026-09-17T00:00:00+02:00</published><updated>2026-09-17T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-09-17:/pbs-client-rhel-container/</id><summary type="html">&lt;p&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 on &lt;span class="caps"&gt;ARM64&lt;/span&gt;, a Proxmox Backup Server, and no convenient native client package. I borrowed the client from an old server container, mounted the host root filesystem read-only, and gave SELinux a very specific explanation. Five minutes later, I had an encrypted, incremental backup. There are caveats. Updated 2026-10-18 with a maintained client image for x86_64 and aarch64 on&amp;nbsp;ghcr.io.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="An ARM64 server wearing a red hat sends backup data through an improvised shipping container to a backup appliance" src="https://hofstede.it/images/2026-09-17-pbs-client-rhel-container.webp" title="No native client? The shipping container has volunteered."&gt;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Update (2026-10-18):&lt;/strong&gt; The dedicated client image promised at the end of this article now exists. &lt;a href="https://github.com/chofstede/pbs-client"&gt;&lt;code&gt;ghcr.io/chofstede/pbs-client&lt;/code&gt;&lt;/a&gt; contains&amp;nbsp;only &lt;code&gt;proxmox-backup-client&lt;/code&gt;, for x86_64 and aarch64, installed from Proxmox&amp;#8217;s&amp;nbsp;official &lt;code&gt;pbs-client&lt;/code&gt; repository for Debian trixie, which now carries arm64 packages as well. The new section &lt;a href="#the-replacement-a-dedicated-client-image"&gt;The Replacement: A Dedicated Client Image&lt;/a&gt; shows the updated command and the output of a run on server01. The&amp;nbsp;original &lt;code&gt;ayufan/proxmox-backup-server:v3.4&lt;/code&gt; invocation below is left as it was&amp;nbsp;run.&lt;/p&gt;
&lt;p&gt;I have a server running &lt;strong&gt;Red Hat Enterprise Linux 10 on aarch64&lt;/strong&gt;, because apparently choosing an enterprise distribution was not enough. I also wanted the architecture to make package availability interesting. I will call it &lt;strong&gt;server01&lt;/strong&gt; here. Hostnames and repository addresses in the examples and log excerpts are&amp;nbsp;anonymized.&lt;/p&gt;
&lt;p&gt;I also have a Proxmox Backup Server. The goal was simple: back up the machine to &lt;span class="caps"&gt;PBS&lt;/span&gt;, with encryption and incremental uploads, using the infrastructure I already&amp;nbsp;have.&lt;/p&gt;
&lt;p&gt;The missing piece was a convenient native Proxmox Backup Client package for this particular combination of operating system and&amp;nbsp;architecture.&lt;/p&gt;
&lt;p&gt;There was, however, a container image with the client inside&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;You can probably see where this is&amp;nbsp;going.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-packaging-problem-with-a-container-shaped-solution"&gt;A Packaging Problem With a Container-Shaped&amp;nbsp;Solution&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-command"&gt;The&amp;nbsp;Command&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-selinux-incantation"&gt;The SELinux&amp;nbsp;Incantation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-stable-name-and-an-ephemeral-container"&gt;A Stable Name and an Ephemeral&amp;nbsp;Container&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-part-where-it-actually-works"&gt;The Part Where It Actually&amp;nbsp;Works&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#full-machine-needs-an-asterisk"&gt;“Full Machine” Needs an&amp;nbsp;Asterisk&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#postgresql-does-not-get-a-free-pass"&gt;PostgreSQL Does Not Get a Free&amp;nbsp;Pass&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#an-old-image-with-a-useful-passenger"&gt;An Old Image With a Useful&amp;nbsp;Passenger&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-replacement-a-dedicated-client-image"&gt;The Replacement: A Dedicated Client&amp;nbsp;Image&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-other-half-of-a-backup"&gt;The Other Half of a&amp;nbsp;Backup&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="a-packaging-problem-with-a-container-shaped-solution"&gt;A Packaging Problem With a Container-Shaped&amp;nbsp;Solution&lt;/h2&gt;
&lt;p&gt;The &lt;a href="https://pbs.proxmox.com/docs/installation.html#client-installation"&gt;official client installation instructions&lt;/a&gt; cover Debian packages and a static client for &lt;strong&gt;x86-64&lt;/strong&gt; Linux. That static binary does not solve my &lt;span class="caps"&gt;ARM64&lt;/span&gt; problem. This is not a claim that nobody has ever built an &lt;span class="caps"&gt;RPM&lt;/span&gt; or an alternative client. It is the absence of the straightforward, supported package I wanted on this&amp;nbsp;machine.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://github.com/ayufan/pve-backup-server-dockerfiles"&gt;Ayufan&amp;#8217;s unofficial &lt;span class="caps"&gt;PBS&lt;/span&gt; builds&lt;/a&gt; provide another route, including &lt;span class="caps"&gt;ARM64&lt;/span&gt; support. For this experiment, I used the client already present in&amp;nbsp;the &lt;code&gt;v3.4&lt;/code&gt; server&amp;nbsp;image.&lt;/p&gt;
&lt;p&gt;The arrangement is wonderfully&amp;nbsp;inelegant:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Start the image with rootful&amp;nbsp;Podman.&lt;/li&gt;
&lt;li&gt;Replace its entrypoint&amp;nbsp;with &lt;code&gt;proxmox-backup-client&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;Make the&amp;nbsp;host&amp;#8217;s &lt;code&gt;/&lt;/code&gt; available&amp;nbsp;at &lt;code&gt;/mnt/root&lt;/code&gt;,&amp;nbsp;read-only.&lt;/li&gt;
&lt;li&gt;Pass in the encryption key and inject the passwords using Podman&amp;nbsp;secrets.&lt;/li&gt;
&lt;li&gt;Upload the host&amp;#8217;s files to the actual &lt;span class="caps"&gt;PBS&lt;/span&gt;&amp;nbsp;server.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;I am borrowing the image&amp;#8217;s userspace and executable. The backup server still lives elsewhere. No &lt;span class="caps"&gt;PBS&lt;/span&gt; service needs to start inside this container. It is the same packaging idea behind &lt;a href="https://hofstede.it/running-citrix-workspace-on-fedora-with-rootless-podman/"&gt;running Citrix Workspace in a Podman container&lt;/a&gt;: give an awkward application the userspace it expects, then expose the host resources it&amp;nbsp;needs.&lt;/p&gt;
&lt;h2 id="the-command"&gt;The&amp;nbsp;Command&lt;/h2&gt;
&lt;p&gt;This assumes Podman is installed, the repository is reachable, the &lt;span class="caps"&gt;PBS&lt;/span&gt; account can write backups,&amp;nbsp;and &lt;code&gt;/root/server01-backup.key&lt;/code&gt; already contains my passphrase-protected &lt;span class="caps"&gt;PBS&lt;/span&gt; encryption key. The passwords must match that account and that&amp;nbsp;key.&lt;/p&gt;
&lt;p&gt;The two secrets live in &lt;strong&gt;root&amp;#8217;s Podman secret store&lt;/strong&gt;, because the backup container also runs through rootful Podman. My &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman production guide&lt;/a&gt; covers secrets and Quadlets in more detail. To create the secrets interactively from&amp;nbsp;Bash:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;read&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-rsp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;PBS repository password: &amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pbs_repo_password
&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;
&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%s&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$pbs_repo_password&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;pbs_repo_password&lt;span class="w"&gt; &lt;/span&gt;-
&lt;span class="nb"&gt;unset&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pbs_repo_password

&lt;span class="nb"&gt;read&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-rsp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;PBS encryption key passphrase: &amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pbs_key_passphrase
&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;
&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%s&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$pbs_key_passphrase&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;pbs_key_passphrase&lt;span class="w"&gt; &lt;/span&gt;-
&lt;span class="nb"&gt;unset&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pbs_key_passphrase
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;These are one-time creation commands. Skip them if those secrets already exist. The prompt keeps the values out of the command text and shell&amp;nbsp;history.&lt;/p&gt;
&lt;p&gt;Then the actual&amp;nbsp;backup:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;-it&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;pbs-backup-job&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--security-opt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;type:spc_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--hostname&lt;span class="w"&gt; &lt;/span&gt;server01&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--entrypoint&lt;span class="w"&gt; &lt;/span&gt;/usr/bin/proxmox-backup-client&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PBS_REPOSITORY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;backup@pbs@pbs.example.net:data&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/:/mnt/root:ro&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/root/server01-backup.key:/etc/proxmox/server01-backup.key:ro,Z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--secret&lt;span class="w"&gt; &lt;/span&gt;pbs_repo_password,type&lt;span class="o"&gt;=&lt;/span&gt;env,target&lt;span class="o"&gt;=&lt;/span&gt;PBS_PASSWORD&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--secret&lt;span class="w"&gt; &lt;/span&gt;pbs_key_passphrase,type&lt;span class="o"&gt;=&lt;/span&gt;env,target&lt;span class="o"&gt;=&lt;/span&gt;PBS_ENCRYPTION_PASSWORD&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;docker.io/ayufan/proxmox-backup-server:v3.4&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;backup&lt;span class="w"&gt; &lt;/span&gt;root.pxar:/mnt/root&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--keyfile&lt;span class="w"&gt; &lt;/span&gt;/etc/proxmox/server01-backup.key
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The repository string identifies&amp;nbsp;the &lt;code&gt;backup@pbs&lt;/code&gt; account, the&amp;nbsp;server &lt;code&gt;pbs.example.net&lt;/code&gt;, and the&amp;nbsp;datastore &lt;code&gt;data&lt;/code&gt;. Replace those, the hostname, and the key paths with your own&amp;nbsp;values.&lt;/p&gt;
&lt;p&gt;This is the command from my successful run with two redundant environment variables&amp;nbsp;removed: &lt;code&gt;PBS_PASSWORD_SECRET&lt;/code&gt; and &lt;code&gt;PBS_KEY_PASSPHRASE_SECRET&lt;/code&gt;. I had pointed those&amp;nbsp;at &lt;code&gt;/run/secrets/...&lt;/code&gt;, but this invocation overrides the image entrypoint and injects the secrets directly into environment variables. There are no corresponding secret-file mounts&amp;nbsp;here.&lt;/p&gt;
&lt;p&gt;The native client&amp;nbsp;consumes &lt;code&gt;PBS_PASSWORD&lt;/code&gt; and &lt;code&gt;PBS_ENCRYPTION_PASSWORD&lt;/code&gt;. Those are the names that matter. See the &lt;a href="https://pbs.proxmox.com/docs-3/backup-client.html#environment-variables"&gt;client&amp;#8217;s environment-variable documentation&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;And yes: &lt;strong&gt;Podman secrets&lt;/strong&gt;, not Podman sockets. The &lt;a href="https://docs.podman.io/en/latest/markdown/podman-run.1.html#secret-secret-opt-opt"&gt;secret options&lt;/a&gt; map each stored secret to the named environment variable&amp;nbsp;with &lt;code&gt;type=env,target=...&lt;/code&gt;. No container-engine &lt;span class="caps"&gt;API&lt;/span&gt; socket is&amp;nbsp;involved.&lt;/p&gt;
&lt;h2 id="the-selinux-incantation"&gt;The SELinux&amp;nbsp;Incantation&lt;/h2&gt;
&lt;p&gt;This is the line doing the interesting&amp;nbsp;work:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;--security-opt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;type:spc_t
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A normal confined container is not supposed to wander through an entire &lt;span class="caps"&gt;RHEL&lt;/span&gt; host filesystem. Backing up that filesystem makes this restriction rather&amp;nbsp;inconvenient.&lt;/p&gt;
&lt;p&gt;If process domains and file labels are unfamiliar, my &lt;a href="https://hofstede.it/selinux-a-practical-guide-for-fedora-and-rhel/"&gt;practical SELinux guide for Fedora and &lt;span class="caps"&gt;RHEL&lt;/span&gt;&lt;/a&gt; explains how those checks work and how to investigate&amp;nbsp;denials.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;spc_t&lt;/code&gt; is the broadly privileged container domain in &lt;a href="https://github.com/containers/container-selinux/blob/main/container.te"&gt;container-selinux policy&lt;/a&gt;. Selecting it relaxes SELinux confinement for this process. It does not switch SELinux off globally, nor does it grant every capability associated with&amp;nbsp;Podman&amp;#8217;s &lt;code&gt;--privileged&lt;/code&gt; option.&lt;/p&gt;
&lt;p&gt;Still, this is a trusted backup tool with access to host data and credentials. The container is a packaging solution, not a place to run a binary I&amp;nbsp;distrust.&lt;/p&gt;
&lt;p&gt;Notice the different volume&amp;nbsp;options:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/:/mnt/root:ro
-v&lt;span class="w"&gt; &lt;/span&gt;/root/server01-backup.key:/etc/proxmox/server01-backup.key:ro,Z
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The host root is mounted read-only without relabeling. The dedicated key file&amp;nbsp;has &lt;code&gt;:Z&lt;/code&gt;, which requests private container labeling and changes that file&amp;#8217;s host label. &lt;strong&gt;Do not&amp;nbsp;add &lt;code&gt;:Z&lt;/code&gt; to the host root mount&lt;/strong&gt;: recursively relabeling the operating system is a very different sort of afternoon. Podman documents this distinction under &lt;a href="https://docs.podman.io/en/latest/markdown/podman-run.1.html#volume-v-source-volume-host-dir-container-dir-options"&gt;volume labeling&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;Read-only also does not mean frozen. The host keeps writing while the backup&amp;nbsp;reads.&lt;/p&gt;
&lt;h2 id="a-stable-name-and-an-ephemeral-container"&gt;A Stable Name and an Ephemeral&amp;nbsp;Container&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;--hostname server01&lt;/code&gt; gives the client a stable host identity. In the run below, the backup lands&amp;nbsp;under &lt;code&gt;host/server01/...&lt;/code&gt;, rather than acquiring an identity derived from a disposable container&amp;#8217;s&amp;nbsp;hostname.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;--rm&lt;/code&gt; removes the container after it exits. The image remains cached, and the backup remains on &lt;span class="caps"&gt;PBS&lt;/span&gt;. &lt;code&gt;root.pxar:/mnt/root&lt;/code&gt; tells the client which directory to archive. Backing up the container&amp;#8217;s&amp;nbsp;own &lt;code&gt;/&lt;/code&gt; would be a successful backup of entirely the wrong&amp;nbsp;machine.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;-it&lt;/code&gt; flags suit this manual invocation. A scheduled job would drop them and needs its own logging, failure reporting, and overlap&amp;nbsp;handling.&lt;/p&gt;
&lt;h2 id="the-part-where-it-actually-works"&gt;The Part Where It Actually&amp;nbsp;Works&lt;/h2&gt;
&lt;p&gt;On September 17, the client started&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Starting backup: host/server01/2026-09-17T17:43:30Z
Client name: server01
Using encryption key from &amp;#39;/etc/proxmox/server01-backup.key&amp;#39;..
storing login ticket failed: $XDG_RUNTIME_DIR must be set
Downloading previous manifest (Sat May 30 15:18:26 2026)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The ticket-cache warning did not prevent this run from authenticating and completing. It is visible in the transcript. I am not quietly editing it into a cleaner success&amp;nbsp;story.&lt;/p&gt;
&lt;p&gt;After a procession of skipped mounts and progress&amp;nbsp;messages:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;root.pxar: had to backup 22.778 GiB of 40.588 GiB (compressed 10.378 GiB) in 304.93 s (average 76.49 MiB/s)
root.pxar: backup was done incrementally, reused 17.811 GiB (43.9%)
Uploaded backup catalog (13.255 MiB)
Duration: 306.42s
End Time: Thu Sep 17 17:48:37 2026
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Roughly &lt;strong&gt;40.6 GiB processed in five minutes&lt;/strong&gt;, with &lt;strong&gt;43.9% reused&lt;/strong&gt; from existing data. The 22.778 GiB figure is before compression. The client reports 10.378 GiB compressed for the archive&amp;#8217;s new data, with catalog and other overhead&amp;nbsp;separate.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; on &lt;span class="caps"&gt;ARM64&lt;/span&gt;. Client borrowed from a server container. Encrypted, incremental backup accepted by &lt;span class="caps"&gt;PBS&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;I would like the record to show that the computer agreed with&amp;nbsp;me.&lt;/p&gt;
&lt;h2 id="full-machine-needs-an-asterisk"&gt;“Full Machine” Needs an&amp;nbsp;Asterisk&lt;/h2&gt;
&lt;p&gt;The goal was a full-machine backup. This particular command produces a &lt;strong&gt;file archive of the root filesystem&lt;/strong&gt;, with mount boundaries that need attention before calling it&amp;nbsp;complete.&lt;/p&gt;
&lt;p&gt;My output explicitly&amp;nbsp;includes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;skipping mount point: &amp;quot;boot/efi&amp;quot;
skipping mount point: &amp;quot;dev&amp;quot;
skipping mount point: &amp;quot;proc&amp;quot;
skipping mount point: &amp;quot;run&amp;quot;
skipping mount point: &amp;quot;sys&amp;quot;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It also skips container overlay mounts, container shared-memory mounts,&amp;nbsp;and &lt;code&gt;var/lib/nfs/rpc_pipefs&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;PBS&lt;/span&gt; deliberately skips additional mount points unless asked to include them. For an &lt;span class="caps"&gt;EFI&lt;/span&gt; filesystem visible inside the container, the backup arguments could instead&amp;nbsp;be:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;backup&lt;span class="w"&gt; &lt;/span&gt;root.pxar:/mnt/root&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--include-dev&lt;span class="w"&gt; &lt;/span&gt;/mnt/root/boot/efi&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--keyfile&lt;span class="w"&gt; &lt;/span&gt;/etc/proxmox/server01-backup.key
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Alternatively, add separate archives for additional filesystems. The &lt;a href="https://pbs.proxmox.com/docs-3/backup-client.html#creating-backups"&gt;backup creation documentation&lt;/a&gt; covers both approaches. Paths here must be paths &lt;strong&gt;inside the container&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Audit the host&amp;#8217;s mounts first, for example&amp;nbsp;with &lt;code&gt;findmnt -R /&lt;/code&gt;, and verify that every required filesystem is visible to the container and included in the archive.&amp;nbsp;Separate &lt;code&gt;/home&lt;/code&gt;, &lt;code&gt;/var&lt;/code&gt;, database storage, and container volumes deserve particular attention. Skipping an overlay mount does not prove the persistent application data elsewhere was&amp;nbsp;captured.&lt;/p&gt;
&lt;p&gt;This is also not a bootable disk image. Partition layout, bootloader installation, and a tested route back to a booting host remain part of recovery&amp;nbsp;planning.&lt;/p&gt;
&lt;h2 id="postgresql-does-not-get-a-free-pass"&gt;PostgreSQL Does Not Get a Free&amp;nbsp;Pass&lt;/h2&gt;
&lt;p&gt;The machine stays online during the backup. The log is quite explicit about the&amp;nbsp;consequences:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;warning: file size increased while reading: &amp;quot;var/log/messages&amp;quot;, file will be truncated!
warning: file size increased while reading: &amp;quot;var/log/traefik/access.log&amp;quot;, file will be truncated!
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For these log files, I can live with that. My first reaction to the databases was similarly relaxed: PostgreSQL is crash-resistant enough,&amp;nbsp;surely?&lt;/p&gt;
&lt;p&gt;There is an important catch. &lt;strong&gt;Crash recovery does not make an arbitrary live file copy consistent.&lt;/strong&gt; A filesystem walk collects different files at different times. That is not the same state as a machine losing power at one&amp;nbsp;instant.&lt;/p&gt;
&lt;p&gt;PostgreSQL&amp;#8217;s &lt;a href="https://www.postgresql.org/docs/current/backup-file.html"&gt;filesystem backup documentation&lt;/a&gt; explains the requirement: stop the database for an ordinary file copy, or use a consistent snapshot covering all the necessary data and &lt;span class="caps"&gt;WAL&lt;/span&gt;. For online backups, use a database-aware method such&amp;nbsp;as &lt;code&gt;pg_basebackup&lt;/code&gt; with the required &lt;span class="caps"&gt;WAL&lt;/span&gt;, or take logical dumps and include those in the&amp;nbsp;backup.&lt;/p&gt;
&lt;p&gt;I can accept the risk for my own machine. I cannot turn that preference into a promise that the copied database files will restore. This run proves the upload worked. It does not prove database&amp;nbsp;recovery.&lt;/p&gt;
&lt;h2 id="an-old-image-with-a-useful-passenger"&gt;An Old Image With a Useful&amp;nbsp;Passenger&lt;/h2&gt;
&lt;p&gt;The image tag is&amp;nbsp;deliberate:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;docker.io/ayufan/proxmox-backup-server:v3.4
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In my testing, newer server images after 3.4.0 no longer shipped the client at the path used here. Moving&amp;nbsp;to &lt;code&gt;latest&lt;/code&gt; therefore broke the premise of this trick. That is an observation about the images I tried, not a claim that the client has disappeared from every distribution&amp;nbsp;channel.&lt;/p&gt;
&lt;p&gt;The upstream project also describes these builds as unofficial and unmaintained, and offers separate client artifacts. This working image is a convenient starting point, not a maintenance strategy. A maintained &lt;span class="caps"&gt;ARM64&lt;/span&gt; client image would be the natural&amp;nbsp;replacement.&lt;/p&gt;
&lt;p&gt;When this article was first published, I was working on exactly that: a container image for &lt;strong&gt;x86_64 and aarch64&lt;/strong&gt; containing&amp;nbsp;only &lt;code&gt;proxmox-backup-client&lt;/code&gt; and its runtime dependencies. It is now available; see the next&amp;nbsp;section.&lt;/p&gt;
&lt;p&gt;For repeatability, record the working image digest and pin that exact artifact after verifying it. A version tag is more descriptive&amp;nbsp;than &lt;code&gt;latest&lt;/code&gt;, but it is still mutable. An old binary also carries its own security and future compatibility questions. Successful communication with today&amp;#8217;s server is the evidence I&amp;nbsp;have.&lt;/p&gt;
&lt;h2 id="the-replacement-a-dedicated-client-image"&gt;The Replacement: A Dedicated Client&amp;nbsp;Image&lt;/h2&gt;
&lt;p&gt;Since this article was published, Proxmox&amp;#8217;s &lt;a href="http://download.proxmox.com/debian/pbs-client/"&gt;&lt;code&gt;pbs-client&lt;/code&gt; repository&lt;/a&gt; for Debian trixie has gained &lt;strong&gt;arm64 packages&lt;/strong&gt;. That removes the need to borrow a server image, or to build the client from source. &lt;a href="https://github.com/chofstede/pbs-client"&gt;&lt;code&gt;ghcr.io/chofstede/pbs-client&lt;/code&gt;&lt;/a&gt; is&amp;nbsp;a &lt;code&gt;debian:trixie-slim&lt;/code&gt; image&amp;nbsp;with &lt;code&gt;proxmox-backup-client&lt;/code&gt; installed from that repository, verified against Proxmox&amp;#8217;s published archive key, and nothing else besides its runtime&amp;nbsp;dependencies.&lt;/p&gt;
&lt;p&gt;One wrinkle: the arm64 packages currently live only in the&amp;nbsp;repository&amp;#8217;s &lt;code&gt;test&lt;/code&gt; component, while x86_64 builds are&amp;nbsp;in &lt;code&gt;main&lt;/code&gt;. The arm64 image therefore pulls&amp;nbsp;from &lt;code&gt;test&lt;/code&gt; and trails x86_64 by a few point releases: 4.2.4 versus 4.2.8 at the time of writing. When Proxmox promotes arm64&amp;nbsp;to &lt;code&gt;main&lt;/code&gt;, a rebuild picks that&amp;nbsp;up.&lt;/p&gt;
&lt;p&gt;The backup command barely&amp;nbsp;changes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;-it&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;pbs-backup-job&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--security-opt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;type:spc_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--hostname&lt;span class="w"&gt; &lt;/span&gt;server01&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PBS_REPOSITORY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;backup@pbs@pbs.example.net:data&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/:/mnt/root:ro&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/root/server01-backup.key:/etc/proxmox/server01-backup.key:ro,Z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--secret&lt;span class="w"&gt; &lt;/span&gt;pbs_repo_password,type&lt;span class="o"&gt;=&lt;/span&gt;env,target&lt;span class="o"&gt;=&lt;/span&gt;PBS_PASSWORD&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--secret&lt;span class="w"&gt; &lt;/span&gt;pbs_key_passphrase,type&lt;span class="o"&gt;=&lt;/span&gt;env,target&lt;span class="o"&gt;=&lt;/span&gt;PBS_ENCRYPTION_PASSWORD&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;ghcr.io/chofstede/pbs-client:latest&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;backup&lt;span class="w"&gt; &lt;/span&gt;root.pxar:/mnt/root&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--keyfile&lt;span class="w"&gt; &lt;/span&gt;/etc/proxmox/server01-backup.key
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The differences from the original&amp;nbsp;invocation:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;No &lt;code&gt;--entrypoint&lt;/code&gt; override.&lt;/strong&gt; The client is the image&amp;#8217;s entrypoint, so everything after the image name is passed straight&amp;nbsp;to &lt;code&gt;proxmox-backup-client&lt;/code&gt;. &lt;code&gt;snapshot list&lt;/code&gt;, &lt;code&gt;restore&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;key create&lt;/code&gt; work the same&amp;nbsp;way.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No ticket-cache warning.&lt;/strong&gt; The image&amp;nbsp;sets &lt;code&gt;XDG_RUNTIME_DIR&lt;/code&gt;, which the client complained about in the transcript above.&amp;nbsp;With &lt;code&gt;--rm&lt;/code&gt;, the cached ticket disappears with the container anyway; the point is a quieter&amp;nbsp;log.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A multi-arch manifest.&lt;/strong&gt; Podman picks the x86_64 or aarch64 variant&amp;nbsp;automatically.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;On October 18, server01 ran the command above against the same &lt;span class="caps"&gt;PBS&lt;/span&gt;&amp;nbsp;datastore:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Starting backup: host/server01/2026-10-18T12:21:59Z
Client name: server01
Starting backup protocol: Sun Oct 18 12:21:59 2026
Using encryption key from &amp;#39;/etc/proxmox/server01-backup.key&amp;#39;..
Downloading previous manifest (Thu Oct 15 16:25:47 2026)
Upload directory &amp;#39;/mnt/root&amp;#39; to &amp;#39;backup@pbs@pbs.example.net:8007:data&amp;#39; as root.pxar.didx
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;storing login ticket failed&lt;/code&gt; line is gone. The same mount points are skipped as before, and the log file still grows underneath the&amp;nbsp;reader:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;warning: file size increased while reading: &amp;quot;var/log/messages&amp;quot;, file will be truncated!
root.pxar: had to backup 1.913 GiB of 28.785 GiB (compressed 857.358 MiB) in 189.56 s (average 10.333 MiB/s)
root.pxar: backup was done incrementally, reused 26.872 GiB (93.4%)
Uploaded backup catalog (8.157 MiB)
Duration: 190.79s
End Time: Sun Oct 18 12:25:10 2026
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;About &lt;strong&gt;28.8 GiB processed in just over three minutes&lt;/strong&gt;, with &lt;strong&gt;93.4% reused&lt;/strong&gt;. The previous snapshot was three days old this time, not months. Same host, same datastore, same encryption key, new&amp;nbsp;client.&lt;/p&gt;
&lt;p&gt;GitHub Actions rebuilds both architectures weekly to pick up new client releases and Debian security updates. Each build is published&amp;nbsp;as &lt;code&gt;latest&lt;/code&gt;, as a dated tag such&amp;nbsp;as &lt;code&gt;20261018&lt;/code&gt;, and&amp;nbsp;as &lt;code&gt;sha-&amp;lt;commit&amp;gt;&lt;/code&gt;. The advice from the previous section still applies: pin a dated tag or, better, a digest you have verified, rather than&amp;nbsp;letting &lt;code&gt;latest&lt;/code&gt; move underneath a backup&amp;nbsp;job.&lt;/p&gt;
&lt;p&gt;One compatibility note: this is a &lt;strong&gt;4.x client&lt;/strong&gt;. My &lt;span class="caps"&gt;PBS&lt;/span&gt; runs the current release. If your server still runs &lt;span class="caps"&gt;PBS&lt;/span&gt; 3.x, run a backup and a test restore before relying on that&amp;nbsp;combination.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;proxmox-backup-client mount&lt;/code&gt; and &lt;code&gt;map&lt;/code&gt; additionally need &lt;span class="caps"&gt;FUSE&lt;/span&gt; inside the container, for&amp;nbsp;example &lt;code&gt;--device /dev/fuse --cap-add SYS_ADMIN&lt;/code&gt;. The plain backup above does&amp;nbsp;not.&lt;/p&gt;
&lt;h2 id="the-other-half-of-a-backup"&gt;The Other Half of a&amp;nbsp;Backup&lt;/h2&gt;
&lt;p&gt;Keep the encryption key and its passphrase recoverable somewhere independent of server01 and this encrypted backup. A copy of the key inside the archive it unlocks is an especially elegant circular&amp;nbsp;dependency.&lt;/p&gt;
&lt;p&gt;Before depending on this arrangement, restore files into an isolated scratch directory, inspect ownership and permissions, and test the applications that matter. The run shown here is an upload result, not a completed disaster-recovery&amp;nbsp;exercise.&lt;/p&gt;
&lt;p&gt;The wider question of where this copy belongs, and what survives losing a host or a backup destination, is the subject of my &lt;a href="https://hofstede.it/my-multi-stage-backup-strategy-zfs-proxmox-and-paranoia/"&gt;multi-stage backup strategy&lt;/a&gt;. This container solves one transfer step in that larger&amp;nbsp;problem.&lt;/p&gt;
&lt;p&gt;But the original packaging problem? Solved. &lt;span class="caps"&gt;RHEL&lt;/span&gt; did not need a native client package for the client to read its files and talk to &lt;span class="caps"&gt;PBS&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;A server image donated an executable, Podman supplied the userspace, SELinux received an unusually permissive explanation, and five minutes later the backup was&amp;nbsp;there.&lt;/p&gt;
&lt;p&gt;Cursed? A little. Working? Yes, within those&amp;nbsp;boundaries.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="rhel"/><category term="aarch64"/><category term="arm64"/><category term="podman"/><category term="containers"/><category term="proxmox"/><category term="backup"/><category term="selinux"/></entry><entry><title>Ansible Automation Platform, Powered by Arch Linux. BTW.</title><link href="https://hofstede.it/ansible-execution-environment-arch-btw/" rel="alternate"/><published>2026-09-12T00:00:00+02:00</published><updated>2026-09-12T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-09-12:/ansible-execution-environment-arch-btw/</id><summary type="html">&lt;p&gt;I built an Arch Linux execution environment, taught pacman to answer to microdnf, and let Ansible Automation Platform use it to patch &lt;span class="caps"&gt;RHEL&lt;/span&gt; systems. The job succeeded. This is evidence, not a&amp;nbsp;recommendation.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="AAP execution environment list showing ee-unsupported-arch_btw and its Arch, btw description" src="https://hofstede.it/images/2026-09-12-arch-ee-environments.png" title="The description is Arch, btw. The image name already contains the disclaimer."&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-proof-it-works-for-one-job-at-least"&gt;The Proof It Works (For One Job, At&amp;nbsp;Least)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-this-is-even-possible"&gt;Why This Is Even&amp;nbsp;Possible&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-experiment-a-containerfile-with-intent"&gt;The Experiment: A Containerfile With Intent&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#step-1-install-the-tools-remove-the-inhibitions"&gt;Step 1: Install The Tools, Remove The&amp;nbsp;Inhibitions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-2-give-pacman-a-fake-identity"&gt;Step 2: Give Pacman A Fake&amp;nbsp;Identity&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-3-let-pip-think-about-what-weve-done"&gt;Step 3: Let Pip Think About What We&amp;#8217;ve&amp;nbsp;Done&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-4-put-on-the-runner-uniform"&gt;Step 4: Put On The Runner&amp;nbsp;Uniform&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-this-is-a-bad-idea"&gt;Why This Is A Bad&amp;nbsp;Idea&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-proper-approach"&gt;The Proper&amp;nbsp;Approach&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-honest-assessment"&gt;The Honest&amp;nbsp;Assessment&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;Back in February, I &lt;a href="https://hofstede.it/rhel-on-zfs-root-an-unholy-experiment/"&gt;put &lt;span class="caps"&gt;RHEL&lt;/span&gt; on a &lt;span class="caps"&gt;ZFS&lt;/span&gt; root filesystem&lt;/a&gt;. Apparently, getting that to boot did not satisfy my need to make perfectly respectable enterprise software participate in questionable&amp;nbsp;activities.&lt;/p&gt;
&lt;p&gt;This time, the question was: &amp;#8220;An &lt;a href="https://hofstede.it/reproducible-ansible-with-execution-environments/"&gt;execution environment&lt;/a&gt; is just a container with Ansible in it. What happens if the container is Arch&amp;nbsp;Linux?&amp;#8221;&lt;/p&gt;
&lt;p&gt;The answer is that it can run a job in Red Hat Ansible Automation Platform. The follow-up question should have been whether that was a good idea. Instead, I gave it &lt;span class="caps"&gt;SSH&lt;/span&gt; credentials and selected &lt;strong&gt;Patch &lt;span class="caps"&gt;RHEL&lt;/span&gt; Systems&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Let me be absolutely clear: &lt;strong&gt;this is unsupported. Please don&amp;#8217;t try this at home, and definitely don&amp;#8217;t turn it into something at work.&lt;/strong&gt; This is a record of a homelab experiment, not an alternative deployment guide. Getting a green job status does not turn Arch Linux, some PyPI packages, and a shell script wearing a fake moustache into a supported &lt;span class="caps"&gt;AAP&lt;/span&gt; execution&amp;nbsp;environment.&lt;/p&gt;
&lt;p&gt;The image is&amp;nbsp;called &lt;code&gt;ee-unsupported-arch_btw&lt;/code&gt;. I feel the naming convention provides adequate evidence of&amp;nbsp;premeditation.&lt;/p&gt;
&lt;h2 id="the-proof-it-works-for-one-job-at-least"&gt;The Proof It Works (For One Job, At&amp;nbsp;Least)&lt;/h2&gt;
&lt;p&gt;The execution environment list at the top of this article shows it between entries that made better life&amp;nbsp;choices.&lt;/p&gt;
&lt;p&gt;The registry reference&amp;nbsp;is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;aap.intra.hofstede.it/ee-unsupported-arch_btw:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Yes, &lt;code&gt;latest&lt;/code&gt;. On a rolling distribution. We will return to that particular aggravating&amp;nbsp;circumstance.&lt;/p&gt;
&lt;p&gt;The job output shows &lt;strong&gt;Success&lt;/strong&gt;, with one play, four tasks, seven hosts, and an elapsed time of &lt;strong&gt;56 seconds&lt;/strong&gt;:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Patch RHEL Systems job output showing Success, seven hosts, four tasks and an elapsed time of 56 seconds" src="https://hofstede.it/images/2026-09-12-arch-ee-job-output.png" title="Arch Linux supervising RHEL maintenance. Nobody involved raised an architectural objection."&gt;&lt;/p&gt;
&lt;p&gt;And because a green bar alone doesn&amp;#8217;t prove which image ran the job, here are the&amp;nbsp;details:&lt;/p&gt;
&lt;p&gt;&lt;img alt="Details of successful job 281 showing ee-unsupported-arch_btw as the execution environment, Satellite Hosts as the inventory and patch_systems.yml as the playbook" src="https://hofstede.it/images/2026-09-12-arch-ee-job-details.png" title="Job 281. Execution environment: ee-unsupported-arch_btw. Status: Success."&gt;&lt;/p&gt;
&lt;p&gt;Job &lt;strong&gt;281&lt;/strong&gt;&amp;nbsp;ran &lt;code&gt;patch_systems.yml&lt;/code&gt; against the &lt;strong&gt;Satellite Hosts&lt;/strong&gt; inventory using &lt;strong&gt;ee-unsupported-arch_btw&lt;/strong&gt;. The play was named &amp;#8220;Patch all systems and reboot if&amp;nbsp;required.&amp;#8221;&lt;/p&gt;
&lt;p&gt;The screenshots establish that this job completed successfully using that execution environment. They don&amp;#8217;t establish that every host needed updates or rebooted, that every collection works, or that this arrangement will survive the next&amp;nbsp;rebuild.&lt;/p&gt;
&lt;p&gt;But yes: I used Arch, btw, to run my &lt;span class="caps"&gt;RHEL&lt;/span&gt; patching job through &lt;span class="caps"&gt;AAP&lt;/span&gt;.&lt;/p&gt;
&lt;h2 id="why-this-is-even-possible"&gt;Why This Is Even&amp;nbsp;Possible&lt;/h2&gt;
&lt;p&gt;At its core, an Ansible execution environment is an &lt;span class="caps"&gt;OCI&lt;/span&gt; container image&amp;nbsp;carrying &lt;code&gt;ansible-core&lt;/code&gt;, &lt;code&gt;ansible-runner&lt;/code&gt;, collections, and the Python and system dependencies needed by the automation. &lt;a href="https://docs.ansible.com/projects/builder/en/stable/"&gt;Ansible Builder documents these ingredients and the stages used to assemble them&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;There is no tiny Red Hat engineer inside the container&amp;nbsp;checking &lt;code&gt;/etc/os-release&lt;/code&gt; before allowing Python to&amp;nbsp;start.&lt;/p&gt;
&lt;p&gt;In this experiment, the container supplies the control-side Ansible runtime. The managed &lt;span class="caps"&gt;RHEL&lt;/span&gt; hosts still have their own operating systems, package managers, and Python interpreters. Running Ansible from Arch doesn&amp;#8217;t make a remote &lt;span class="caps"&gt;RHEL&lt;/span&gt; machine start installing &lt;span class="caps"&gt;RPM&lt;/span&gt; updates&amp;nbsp;with &lt;code&gt;pacman&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The interpreter discovery warnings visible in the job output concern Python on the managed hosts. They aren&amp;#8217;t &lt;span class="caps"&gt;AAP&lt;/span&gt; discovering Arch in the execution environment and attempting to contact an&amp;nbsp;adult.&lt;/p&gt;
&lt;p&gt;The interesting part is how little the runtime needs to care about the distribution name once the required tools and dependencies are present. The terrible part is deciding that this observation constitutes an operational&amp;nbsp;strategy.&lt;/p&gt;
&lt;h2 id="the-experiment-a-containerfile-with-intent"&gt;The Experiment: A Containerfile With&amp;nbsp;Intent&lt;/h2&gt;
&lt;p&gt;What follows is a reconstruction of this particular experiment. The snippets document the shortcuts that got one image through one job; they are not a supported build procedure or a reusable Arch execution environment&amp;nbsp;template.&lt;/p&gt;
&lt;p&gt;The build context contained&amp;nbsp;a &lt;code&gt;Containerfile&lt;/code&gt; and the familiar Ansible&amp;nbsp;Builder &lt;code&gt;_build&lt;/code&gt; directory: requirements files plus scripts such&amp;nbsp;as &lt;code&gt;assemble&lt;/code&gt;, &lt;code&gt;check_ansible&lt;/code&gt;, &lt;code&gt;install-from-bindep&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;entrypoint&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The Containerfile kept the four&amp;nbsp;stages: &lt;code&gt;base&lt;/code&gt;, &lt;code&gt;galaxy&lt;/code&gt;, &lt;code&gt;builder&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;final&lt;/code&gt;. So the overall structure looked reassuringly ordinary, provided you didn&amp;#8217;t read the first&amp;nbsp;line:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;ARG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;EE_BASE_IMAGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;docker.io/archlinux:latest&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;These are excerpts from the experiment, not a complete standalone recipe. The build also depended on&amp;nbsp;those &lt;code&gt;_build&lt;/code&gt; files. Copying the snippets into an empty directory will mostly reproduce&amp;nbsp;disappointment.&lt;/p&gt;
&lt;h3 id="step-1-install-the-tools-remove-the-inhibitions"&gt;Step 1: Install The Tools, Remove The&amp;nbsp;Inhibitions&lt;/h3&gt;
&lt;p&gt;Naturally, the first run didn&amp;#8217;t immediately go green. &lt;span class="caps"&gt;AAP&lt;/span&gt; launched the execution environment, only&amp;nbsp;for &lt;code&gt;dumb-init&lt;/code&gt; to&amp;nbsp;report &lt;code&gt;ssh-agent: No such file or directory&lt;/code&gt;. We had successfully packaged an automation runtime that couldn&amp;#8217;t start the &lt;span class="caps"&gt;SSH&lt;/span&gt; agent needed for its credentials. Excellent work,&amp;nbsp;everyone.&lt;/p&gt;
&lt;p&gt;For this job, the credential setup&amp;nbsp;needed &lt;code&gt;ssh-agent&lt;/code&gt; inside the execution environment to load the supplied &lt;span class="caps"&gt;SSH&lt;/span&gt; key. The initial image didn&amp;#8217;t have it: the minimal Arch base had left OpenSSH as an exercise for the reader.&amp;nbsp;Once &lt;code&gt;openssh&lt;/code&gt; was dragged into the base stage&amp;nbsp;alongside &lt;code&gt;tar&lt;/code&gt; and &lt;code&gt;rsync&lt;/code&gt;, the job could finally get past credential setup and talk to the managed&amp;nbsp;hosts.&lt;/p&gt;
&lt;p&gt;The corrected base stage initialized the Arch package signing keyring and installed the baseline&amp;nbsp;packages:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pacman-key&lt;span class="w"&gt; &lt;/span&gt;--init&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;pacman-key&lt;span class="w"&gt; &lt;/span&gt;--populate&lt;span class="w"&gt; &lt;/span&gt;archlinux&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-Syu&lt;span class="w"&gt; &lt;/span&gt;--noconfirm&lt;span class="w"&gt; &lt;/span&gt;--needed&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;base-devel&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;git&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;openssh&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;rsync&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;tar&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;python&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;python-pip&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;python-setuptools&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;python-wheel&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;ansible-core&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;ca-certificates&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-Scc&lt;span class="w"&gt; &lt;/span&gt;--noconfirm
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That build installed&amp;nbsp;Arch&amp;#8217;s &lt;code&gt;ansible-core&lt;/code&gt; &lt;strong&gt;2.21.4&lt;/strong&gt; and Python &lt;strong&gt;3.14.7&lt;/strong&gt;. Those are the versions in this experiment&amp;#8217;s log, not a promise about what a future build will&amp;nbsp;pull.&lt;/p&gt;
&lt;p&gt;For the remaining Python packages, the Containerfile announced its intentions&amp;nbsp;early:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PIP_BREAK_SYSTEM_PACKAGES&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;An environment variable whose name doubles as a&amp;nbsp;confession.&lt;/p&gt;
&lt;p&gt;Then it installed Runner and the init helper into the base&amp;nbsp;stage:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$PYCMD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--no-cache-dir&lt;span class="w"&gt; &lt;/span&gt;ansible-runner&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;dumb-init==1.2.5&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;nbsp;produced &lt;code&gt;ansible-runner&lt;/code&gt; &lt;strong&gt;2.4.3&lt;/strong&gt;. It also ensured that the&amp;nbsp;later &lt;code&gt;check_ansible&lt;/code&gt; script could find both Ansible and Runner and import their Python&amp;nbsp;modules.&lt;/p&gt;
&lt;p&gt;The build repeatedly printed pip&amp;#8217;s warning about running as root and mixing with the system package manager. The warning was doing its job. I was doing something&amp;nbsp;else.&lt;/p&gt;
&lt;h3 id="step-2-give-pacman-a-fake-identity"&gt;Step 2: Give Pacman A Fake&amp;nbsp;Identity&lt;/h3&gt;
&lt;p&gt;The most incriminating file in the image&amp;nbsp;is &lt;code&gt;/usr/local/bin/microdnf&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;It is not&amp;nbsp;microdnf.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Experimental shim, not a compatibility layer:&lt;/strong&gt; the following is the exact impersonator used in this build. Do not copy it into a real execution environment and assume you have added Arch support to Ansible&amp;nbsp;Builder.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;
&lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;clean&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-Scc&lt;span class="w"&gt; &lt;/span&gt;--noconfirm
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;install&lt;span class="p"&gt;|&lt;/span&gt;update&lt;span class="p"&gt;|&lt;/span&gt;upgrade&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;arg&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$arg&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;-y&lt;span class="p"&gt;|&lt;/span&gt;--nodocs&lt;span class="p"&gt;|&lt;/span&gt;--setopt&lt;span class="o"&gt;=&lt;/span&gt;*&lt;span class="p"&gt;|&lt;/span&gt;--enablerepo&lt;span class="o"&gt;=&lt;/span&gt;*&lt;span class="p"&gt;|&lt;/span&gt;--disablerepo&lt;span class="o"&gt;=&lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;args&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$args&lt;/span&gt;&lt;span class="s2"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$arg&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;esac&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;--noconfirm&lt;span class="w"&gt; &lt;/span&gt;--needed&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$args&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;repoquery&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;shift&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;-Si&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pacman&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="k"&gt;esac&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The build argument pointed the helper scripts at this&amp;nbsp;impersonator:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;ARG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PKGMGR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/usr/local/bin/microdnf&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;When the final dependency installation&amp;nbsp;ran:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/usr/local/bin/microdnf install -y curl
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Pacman answered&amp;nbsp;that &lt;code&gt;curl&lt;/code&gt; was already installed. The build continued. Nobody asked to see microdnf&amp;#8217;s&amp;nbsp;passport.&lt;/p&gt;
&lt;p&gt;This wrapper handles a small set of invocations well enough for this build. It does not translate &lt;span class="caps"&gt;RPM&lt;/span&gt; package names into Arch package names, reproduce repository selection, or implement &lt;span class="caps"&gt;DNF&lt;/span&gt;&amp;#8217;s dependency semantics.&amp;nbsp;Its &lt;code&gt;update&lt;/code&gt; and &lt;code&gt;upgrade&lt;/code&gt; branches simply&amp;nbsp;run &lt;code&gt;pacman -S&lt;/code&gt;, and&amp;nbsp;its &lt;code&gt;repoquery&lt;/code&gt; branch discards the output&amp;nbsp;entirely.&lt;/p&gt;
&lt;p&gt;It is roughly the package management equivalent of writing &amp;#8220;ambulance&amp;#8221; on a shopping trolley. Useful resemblance, limited&amp;nbsp;guarantees.&lt;/p&gt;
&lt;h3 id="step-3-let-pip-think-about-what-weve-done"&gt;Step 3: Let Pip Think About What We&amp;#8217;ve&amp;nbsp;Done&lt;/h3&gt;
&lt;p&gt;The Galaxy stage&amp;nbsp;installed &lt;code&gt;amazon.aws&lt;/code&gt; &lt;strong&gt;11.4.0&lt;/strong&gt;. Dependency introspection then combined the collection requirements with the user requirements,&amp;nbsp;including &lt;code&gt;boto3&lt;/code&gt;, &lt;code&gt;botocore&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;aiobotocore&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;Pip responded by reviewing a substantial portion of their release&amp;nbsp;history.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;INFO: pip is looking at multiple versions of aiobotocore to determine which version is compatible with other requirements. This could take a while.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It did take a while. Eventually the resolver settled&amp;nbsp;on &lt;code&gt;boto3&lt;/code&gt; and &lt;code&gt;botocore&lt;/code&gt; &lt;strong&gt;1.43.75&lt;/strong&gt;,&amp;nbsp;with &lt;code&gt;aiobotocore&lt;/code&gt; &lt;strong&gt;3.9.1&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Then the final stage resolved the requirements again. The cache saved downloads, but apparently not the opportunity for&amp;nbsp;reflection.&lt;/p&gt;
&lt;p&gt;The build technically worked, but not quite for the reasons the stage names&amp;nbsp;suggested.&lt;/p&gt;
&lt;p&gt;Despite the directory&amp;nbsp;name &lt;code&gt;/output/wheels&lt;/code&gt;, the scripts found zero&amp;nbsp;top-level &lt;code&gt;*.whl&lt;/code&gt; files there. Instead of installing a prepared set of wheel files from the builder stage, the final stage installed from the requirements file again. The extra fallback install then found those requirements already satisfied. The image built, but this run did not demonstrate an offline handoff of a fixed set of Python packages between&amp;nbsp;stages.&lt;/p&gt;
&lt;p&gt;The Containerfile also&amp;nbsp;included:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/output/scripts/install-from-bindep&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;true
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That particular invocation succeeded in the supplied build.&amp;nbsp;The &lt;code&gt;|| true&lt;/code&gt; would also have allowed a failure to pass unnoticed at that layer, which is a rather generous interpretation of dependency&amp;nbsp;management.&lt;/p&gt;
&lt;h3 id="step-4-put-on-the-runner-uniform"&gt;Step 4: Put On The Runner&amp;nbsp;Uniform&lt;/h3&gt;
&lt;p&gt;The final stage&amp;nbsp;created &lt;code&gt;/runner&lt;/code&gt;, adjusted permissions, and created&amp;nbsp;a &lt;code&gt;runner&lt;/code&gt; user with &lt;span class="caps"&gt;UID&lt;/span&gt; 1000 and group 0. It finished&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;LABEL&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ansible-execution-environment&lt;span class="o"&gt;=&lt;/span&gt;true
&lt;span class="k"&gt;USER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;1000&lt;/span&gt;
&lt;span class="k"&gt;ENTRYPOINT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/opt/builder/bin/entrypoint&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;dumb-init&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;CMD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bash&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The label is metadata. It is not a certificate of good&amp;nbsp;behaviour.&lt;/p&gt;
&lt;p&gt;The image built successfully and pushed to the private registry. An interactive run confirmed what was inside. Abbreviated from the terminal&amp;nbsp;output:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[runner@60a6dd752943 runner]$ cat /etc/os-release
NAME=&amp;quot;Arch Linux&amp;quot;
PRETTY_NAME=&amp;quot;Arch Linux&amp;quot;
ID=arch
BUILD_ID=rolling
VERSION_ID=20260906.0.587075

[runner@60a6dd752943 runner]$ ansible --version
ansible [core 2.21.4]
  python version = 3.14.7 (main, Aug 10 2026, 07:46:56) [GCC 16.1.1 20260728] (/usr/bin/python)

[runner@60a6dd752943 runner]$ ansible-runner --version
2.4.3
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Arch Linux. Ansible. Runner. A working image and no remaining excuse to pretend this happened&amp;nbsp;accidentally.&lt;/p&gt;
&lt;h2 id="why-this-is-a-bad-idea"&gt;Why This Is A Bad&amp;nbsp;Idea&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;The support boundary hasn&amp;#8217;t moved.&lt;/strong&gt; This is an unsupported experiment. Registering the image in &lt;span class="caps"&gt;AAP&lt;/span&gt;, storing it in a private registry, and running a successful job do not make its contents Red Hat-delivered or supported.&amp;nbsp;The &lt;code&gt;unsupported&lt;/code&gt; in the name continues to apply after the green tick&amp;nbsp;appears.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Rebuildable does not mean&amp;nbsp;reproducible.&lt;/strong&gt; &lt;code&gt;archlinux:latest&lt;/code&gt; can select a different base&amp;nbsp;image, &lt;code&gt;pacman -Syu&lt;/code&gt; pulls the rolling repositories&amp;#8217; current packages, and loosely constrained Python dependencies leave pip to make fresh decisions. On a later uncached rebuild, the same Git commit can produce a different Python, a different Ansible, and a different dependency set. The change request says &amp;#8220;rebuild the image.&amp;#8221; The result may include an unplanned runtime upgrade. A built image doesn&amp;#8217;t spontaneously update itself; the surprise arrives when you try to recreate&amp;nbsp;it. &lt;code&gt;latest&lt;/code&gt; is a tag, not a reproducibility&amp;nbsp;strategy.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The package manager is lying about its qualifications.&lt;/strong&gt; The wrapper was sufficient for the calls seen here. Another collection might need differently named packages, repository options, or query output that it doesn&amp;#8217;t provide. A&amp;nbsp;successful &lt;code&gt;curl&lt;/code&gt; check is not a package manager compatibility&amp;nbsp;suite.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The build contains several escape hatches.&lt;/strong&gt; System-wide pip installs are explicitly enabled, collection &lt;span class="caps"&gt;GPG&lt;/span&gt; verification is disabled in the Galaxy command, and some commands&amp;nbsp;have &lt;code&gt;|| true&lt;/code&gt; attached. These are properties of this Containerfile, not requirements for building execution environments. Arch&amp;#8217;s package keyring was still initialized and used; that is separate from collection signature&amp;nbsp;verification.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The final image inherits the toolchain.&lt;/strong&gt;&amp;nbsp;Because &lt;code&gt;final&lt;/code&gt; starts&amp;nbsp;from &lt;code&gt;base&lt;/code&gt;, it&amp;nbsp;inherits &lt;code&gt;base-devel&lt;/code&gt; and the other baseline packages. Four build stages don&amp;#8217;t automatically produce a minimal runtime image. In this case, the compiler came along to witness the&amp;nbsp;offence.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;One successful job is one successful job.&lt;/strong&gt; It doesn&amp;#8217;t validate all collections, other inventories, future Python versions, upgrades, or a different execution node. The seven hosts in the screenshot were test subjects, not a certification&amp;nbsp;programme.&lt;/p&gt;
&lt;h2 id="the-proper-approach"&gt;The Proper&amp;nbsp;Approach&lt;/h2&gt;
&lt;p&gt;For actual &lt;span class="caps"&gt;AAP&lt;/span&gt; work, use the documented execution environment base and component versions for your &lt;span class="caps"&gt;AAP&lt;/span&gt; release. Start at the &lt;a href="https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/"&gt;Red Hat Ansible Automation Platform documentation&lt;/a&gt;, select the version you operate, and follow its execution environment guide. Use the&amp;nbsp;appropriate &lt;code&gt;ee-minimal&lt;/code&gt; image where that guide recommends it as the foundation for custom content, keeping control over the dependencies you&amp;nbsp;add.&lt;/p&gt;
&lt;p&gt;Keep the definition and dependencies under version control, build with Ansible Builder, test the resulting image with your automation, and promote the tested artifact deliberately. Adding custom content still means paying attention to that content&amp;#8217;s compatibility and support&amp;nbsp;boundaries.&lt;/p&gt;
&lt;p&gt;I even wrote about &lt;a href="https://hofstede.it/sane-development-environments-for-ansible-automation-code/"&gt;sane Ansible development environments&lt;/a&gt; recently. The existence of that article makes this one harder to explain as an honest&amp;nbsp;misunderstanding.&lt;/p&gt;
&lt;h2 id="the-honest-assessment"&gt;The Honest&amp;nbsp;Assessment&lt;/h2&gt;
&lt;p&gt;The experiment worked because the essential runtime was there: an &lt;span class="caps"&gt;OCI&lt;/span&gt; container&amp;nbsp;with &lt;code&gt;ansible-core&lt;/code&gt;, &lt;code&gt;ansible-runner&lt;/code&gt;, and the dependencies needed by this job. Ansible could run, reach the managed hosts, and execute the playbook. The distribution inside the container did not prevent that particular&amp;nbsp;workflow.&lt;/p&gt;
&lt;p&gt;That is an interesting technical observation. It provides absolutely no reason to adopt this&amp;nbsp;configuration.&lt;/p&gt;
&lt;p&gt;Please don&amp;#8217;t use this article as a deployment guide. Use the documented &lt;span class="caps"&gt;AAP&lt;/span&gt; execution environments for the infrastructure you care about, and leave the counterfeit microdnf in the evidence&amp;nbsp;locker.&lt;/p&gt;
&lt;p&gt;My &lt;span class="caps"&gt;RHEL&lt;/span&gt; systems have now been maintained by Arch Linux through Red Hat Ansible Automation&amp;nbsp;Platform.&lt;/p&gt;
&lt;p&gt;The job took 56 seconds. Explaining myself will take considerably&amp;nbsp;longer.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="rhel"/><category term="archlinux"/><category term="ansible"/><category term="ansible-automation-platform"/><category term="podman"/><category term="containers"/><category term="unsupported"/></entry><entry><title>Running Citrix Workspace on Fedora with Rootless Podman</title><link href="https://hofstede.it/running-citrix-workspace-on-fedora-with-rootless-podman/" rel="alternate"/><published>2026-09-03T00:00:00+02:00</published><updated>2026-09-03T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-09-03:/running-citrix-workspace-on-fedora-with-rootless-podman/</id><summary type="html">&lt;p&gt;Citrix Workspace is an application I need but do not want woven into my workstation. I put its official Debian package in a disposable rootless Podman container, then pass through just enough X11, PipeWire, &lt;span class="caps"&gt;GPU&lt;/span&gt;, and webcam plumbing to open downloaded &lt;span class="caps"&gt;ICA&lt;/span&gt; files from Fedora&amp;#8217;s file&amp;nbsp;manager.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A remote desktop client contained inside a rootless application container, with controlled connections to host devices" src="https://hofstede.it/images/2026-09-03-citrix-workspace-rootless-podman-fedora.webp" title="Citrix Workspace contained in rootless Podman with selected desktop resources passed through"&gt;&lt;/p&gt;
&lt;p&gt;There is a class of software I need for work but do not particularly want to &lt;em&gt;adopt&lt;/em&gt;. Citrix Workspace belongs firmly in that&amp;nbsp;class.&lt;/p&gt;
&lt;p&gt;The Linux client itself works. The less appealing part is everything around it: a large proprietary package, supporting libraries, certificate-store modifications, optional &lt;span class="caps"&gt;USB&lt;/span&gt; components, configuration below my home directory, and another application whose lifecycle becomes entangled with the workstation. I need it to open&amp;nbsp;an &lt;code&gt;.ica&lt;/code&gt; file. I do not need it to become part of&amp;nbsp;Fedora.&lt;/p&gt;
&lt;p&gt;So I put the whole thing in an Ubuntu 24.04 image and launch each session as a disposable rootless Podman container. Fedora 44 keeps only Podman, a short wrapper script, and a desktop entry. The Citrix binaries and their Debian dependencies stay in container storage, where deleting them later has a satisfying, unambiguous&amp;nbsp;meaning.&lt;/p&gt;
&lt;p&gt;The result has been pleasantly boring. Audio works through the host&amp;#8217;s PipeWire PulseAudio socket, hardware rendering&amp;nbsp;uses &lt;code&gt;/dev/dri&lt;/code&gt;, webcams are discovered dynamically, clipboard text crosses the session, and double-clicking a downloaded &lt;span class="caps"&gt;ICA&lt;/span&gt; file starts the connection like any native file&amp;nbsp;association.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-boundary-i-actually-want"&gt;The Boundary I Actually&amp;nbsp;Want&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#downloading-the-package"&gt;Downloading the&amp;nbsp;Package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-container-image"&gt;The Container&amp;nbsp;Image&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-launcher-for-one-ica-file"&gt;A Launcher for One &lt;span class="caps"&gt;ICA&lt;/span&gt;&amp;nbsp;File&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#making-ica-files-behave-like-normal-files"&gt;Making &lt;span class="caps"&gt;ICA&lt;/span&gt; Files Behave Like Normal&amp;nbsp;Files&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-remains-on-the-host"&gt;What Remains on the&amp;nbsp;Host&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-boundary-i-actually-want"&gt;The Boundary I Actually&amp;nbsp;Want&lt;/h2&gt;
&lt;p&gt;This is primarily a &lt;strong&gt;packaging boundary&lt;/strong&gt;, not a high-security&amp;nbsp;sandbox.&lt;/p&gt;
&lt;p&gt;That distinction matters. The container runs rootless and receives no home-directory mount, which is already a much cleaner arrangement than installing the client directly. It is also removed after every session. But a desktop client cannot do useful desktop-client things without access to the desktop. This one receives the X11 socket, the audio socket, host networking and &lt;span class="caps"&gt;IPC&lt;/span&gt;, plus &lt;span class="caps"&gt;GPU&lt;/span&gt; and video devices. I also disable SELinux label separation for the container because those host resources carry labels that are not intended for container&amp;nbsp;access.&lt;/p&gt;
&lt;p&gt;In other words: the proprietary filesystem clutter is well contained. The running process is deliberately integrated with the local session and should not be mistaken for hostile-code isolation. If I did not trust the binary at all, I would run it in a &lt;span class="caps"&gt;VM&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;For the actual goal, keeping Fedora&amp;#8217;s &lt;span class="caps"&gt;RPM&lt;/span&gt;&amp;nbsp;database, &lt;code&gt;/opt&lt;/code&gt;, &lt;code&gt;/etc&lt;/code&gt;, and my home directory free of Citrix state, the container is exactly the right size of&amp;nbsp;solution.&lt;/p&gt;
&lt;h2 id="downloading-the-package"&gt;Downloading the&amp;nbsp;Package&lt;/h2&gt;
&lt;p&gt;Citrix provides the Linux packages on its &lt;a href="https://www.citrix.com/downloads/workspace-app/linux/workspace-app-for-linux-latest.html"&gt;Workspace app download page&lt;/a&gt;. At the time of writing, the downloads do not require an account, although accepting the licence terms is still part of the download&amp;nbsp;flow.&lt;/p&gt;
&lt;p&gt;I download the x86-64 Debian &lt;strong&gt;Full Package (Self-Service Support)&lt;/strong&gt; into an otherwise empty build&amp;nbsp;directory:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;citrix-workspace-container/
├── Containerfile
└── icaclient_&amp;lt;version&amp;gt;_amd64.deb
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I verify the published &lt;span class="caps"&gt;SHA&lt;/span&gt;-256 sum before building. This is a proprietary package subject to Citrix&amp;#8217;s licence, so the result is a local image, not something I publish to a&amp;nbsp;registry.&lt;/p&gt;
&lt;p&gt;Citrix also offers a&amp;nbsp;separate &lt;code&gt;ctxusb&lt;/code&gt; package for generic &lt;span class="caps"&gt;USB&lt;/span&gt; redirection. I deliberately leave it out. &lt;a href="https://docs.citrix.com/en-us/citrix-workspace-app-for-linux/usb.html"&gt;Citrix&amp;#8217;s &lt;span class="caps"&gt;USB&lt;/span&gt; documentation&lt;/a&gt; describes its own device-policy layer;&amp;nbsp;exposing &lt;code&gt;/dev/bus/usb&lt;/code&gt; alone does not reproduce that setup. The optimised webcam path used below does not require generic &lt;span class="caps"&gt;USB&lt;/span&gt;&amp;nbsp;redirection.&lt;/p&gt;
&lt;h2 id="the-container-image"&gt;The Container&amp;nbsp;Image&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;Containerfile&lt;/code&gt; is deliberately&amp;nbsp;uninteresting:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ubuntu:24.04&lt;/span&gt;

&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;DEBIAN_FRONTEND&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;noninteractive

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;apt-get&lt;span class="w"&gt; &lt;/span&gt;update&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;apt-get&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;--no-install-recommends&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;ca-certificates&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;curl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;gstreamer1.0-libav&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;gstreamer1.0-plugins-bad&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;gstreamer1.0-plugins-base&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;gstreamer1.0-plugins-good&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;gstreamer1.0-plugins-ugly&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;gstreamer1.0-vaapi&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;libasound2t64&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;libgtk-3-0t64&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;libpulse0&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;libv4l-0t64&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;libx11-6&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;libxss1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;v4l-utils&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;xclip&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;xdg-utils&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;xsel&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-rf&lt;span class="w"&gt; &lt;/span&gt;/var/lib/apt/lists/*

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;icaclient-*.deb&lt;span class="w"&gt; &lt;/span&gt;/tmp/

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;apt-get&lt;span class="w"&gt; &lt;/span&gt;update&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;apt-get&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;--no-install-recommends&lt;span class="w"&gt; &lt;/span&gt;/tmp/*.deb&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/tmp/*.deb&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-rf&lt;span class="w"&gt; &lt;/span&gt;/var/lib/apt/lists/*

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;-rf&lt;span class="w"&gt; &lt;/span&gt;/opt/Citrix/ICAClient/keystore/cacerts&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;ln&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;/etc/ssl/certs&lt;span class="w"&gt; &lt;/span&gt;/opt/Citrix/ICAClient/keystore/cacerts

&lt;span class="k"&gt;CMD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/opt/Citrix/ICAClient/selfservice&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Ubuntu is not a philosophical statement here. Citrix ships a Debian package, &lt;a href="https://docs.citrix.com/en-us/citrix-workspace-app-for-linux/system-requirements.html"&gt;Ubuntu 24.04 is among the distributions it supports&lt;/a&gt;, and using the package on its expected userspace is less exciting than trying to unpack it into Fedora. Less exciting is exactly what I want from remote-desktop&amp;nbsp;infrastructure.&lt;/p&gt;
&lt;p&gt;The GStreamer packages are for &lt;span class="caps"&gt;HDX&lt;/span&gt; webcam redirection. Citrix lists the &lt;a href="https://docs.citrix.com/en-us/citrix-workspace-app-for-linux/system-requirements.html"&gt;base, good, bad, ugly, and libav plug-ins&lt;/a&gt; for that path, while its &lt;a href="https://docs.citrix.com/en-us/citrix-workspace-app-for-linux/webcams.html"&gt;webcam documentation&lt;/a&gt; also calls out &lt;span class="caps"&gt;VA&lt;/span&gt;-&lt;span class="caps"&gt;API&lt;/span&gt; for H.264 acceleration. I name them explicitly&amp;nbsp;because &lt;code&gt;--no-install-recommends&lt;/code&gt; should not quietly turn a documented feature into an accidental property of the dependency&amp;nbsp;graph.&lt;/p&gt;
&lt;p&gt;The last build step replaces Citrix&amp;#8217;s private &lt;span class="caps"&gt;CA&lt;/span&gt; directory with Ubuntu&amp;#8217;s maintained certificate store. That avoids the familiar situation where an otherwise valid corporate or public certificate works in every application except the &lt;span class="caps"&gt;ICA&lt;/span&gt; client. The certificates are still those from the image, so rebuilding the image is how they receive&amp;nbsp;updates.&lt;/p&gt;
&lt;p&gt;From the build&amp;nbsp;directory:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;localhost/citrix-workspace&lt;span class="w"&gt; &lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The image contains the client and all of its userspace dependencies. Nothing has been installed into&amp;nbsp;Fedora.&lt;/p&gt;
&lt;h2 id="a-launcher-for-one-ica-file"&gt;A Launcher for One &lt;span class="caps"&gt;ICA&lt;/span&gt;&amp;nbsp;File&lt;/h2&gt;
&lt;p&gt;I use the client at the lower level rather than configuring a persistent store&amp;nbsp;in &lt;code&gt;selfservice&lt;/code&gt;: a browser downloads an &lt;span class="caps"&gt;ICA&lt;/span&gt; file,&amp;nbsp;and &lt;code&gt;wfica&lt;/code&gt; opens precisely that file. The wrapper lives&amp;nbsp;at &lt;code&gt;~/.local/bin/citrix-open&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-euo&lt;span class="w"&gt; &lt;/span&gt;pipefail

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;((&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Usage: %s FILE.ica\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;0&lt;/span&gt;&lt;span class="p"&gt;##*/&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="nv"&gt;ICA_FILE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;realpath&lt;span class="w"&gt; &lt;/span&gt;--&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$ICA_FILE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ICA file not found: %s\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ICA_FILE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="nv"&gt;HOST_USER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;id&lt;span class="w"&gt; &lt;/span&gt;-un&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;RUNTIME_DIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;XDG_RUNTIME_DIR&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="p"&gt;/run/user/&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;id&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;
&lt;span class="nv"&gt;PULSE_SOCKET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$RUNTIME_DIR&lt;/span&gt;/pulse/native

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;DISPLAY&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;DISPLAY is not set; launch this from an X11 or XWayland session\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;!&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$PULSE_SOCKET&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;PulseAudio socket not found: %s\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PULSE_SOCKET&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;

&lt;span class="c1"&gt;# The rootless container process is this local user from X11&amp;#39;s perspective.&lt;/span&gt;
xhost&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;+si:localuser:&lt;/span&gt;&lt;span class="nv"&gt;$HOST_USER&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null
cleanup&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;xhost&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-si:localuser:&lt;/span&gt;&lt;span class="nv"&gt;$HOST_USER&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="nb"&gt;trap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;cleanup&lt;span class="w"&gt; &lt;/span&gt;EXIT

&lt;span class="c1"&gt;# Keep every device argument separate and correctly quoted.&lt;/span&gt;
&lt;span class="nv"&gt;DEVICE_ARGS&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;--group-add&lt;span class="w"&gt; &lt;/span&gt;keep-groups&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="nb"&gt;shopt&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;nullglob
&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;dev&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/dev/dri/card*&lt;span class="w"&gt; &lt;/span&gt;/dev/dri/renderD*&lt;span class="w"&gt; &lt;/span&gt;/dev/video*&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;DEVICE_ARGS&lt;/span&gt;&lt;span class="o"&gt;+=(&lt;/span&gt;--device&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$dev&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="nv"&gt;$dev&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="nb"&gt;shopt&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;nullglob

podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--net&lt;span class="o"&gt;=&lt;/span&gt;host&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ipc&lt;span class="o"&gt;=&lt;/span&gt;host&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--security-opt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;disable&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;DEVICE_ARGS&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;DISPLAY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$DISPLAY&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;QT_X11_NO_MITSHM&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/tmp/.X11-unix:/tmp/.X11-unix:ro&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PULSE_SOCKET&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="nv"&gt;$PULSE_SOCKET&lt;/span&gt;&lt;span class="s2"&gt;:ro&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PULSE_SERVER&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;unix:&lt;/span&gt;&lt;span class="nv"&gt;$PULSE_SOCKET&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$ICA_FILE&lt;/span&gt;&lt;span class="s2"&gt;:/tmp/session.ica:ro&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;localhost/citrix-workspace&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/bin/bash&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
&lt;span class="s1"&gt;        mkdir -p /root/.ICAClient&lt;/span&gt;

&lt;span class="s1"&gt;        # These defaults prevent wfica from starting with an incomplete&lt;/span&gt;
&lt;span class="s1"&gt;        # per-user configuration in a fresh --rm container.&lt;/span&gt;
&lt;span class="s1"&gt;        cp -n /opt/Citrix/ICAClient/config/All_Regions.ini /root/.ICAClient/&lt;/span&gt;
&lt;span class="s1"&gt;        cp -n /opt/Citrix/ICAClient/config/canonicalization.ini /root/.ICAClient/&lt;/span&gt;

&lt;span class="s1"&gt;        # Keep clipboard integration enabled but restrict this setup to text.&lt;/span&gt;
&lt;span class="s1"&gt;        cat &amp;gt; /root/.ICAClient/wfclient.ini &amp;lt;&amp;lt; &amp;quot;EOF&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;[WFClient]&lt;/span&gt;
&lt;span class="s1"&gt;Version=2&lt;/span&gt;
&lt;span class="s1"&gt;ClientClipboard=On&lt;/span&gt;
&lt;span class="s1"&gt;HashtagClipboard=On&lt;/span&gt;

&lt;span class="s1"&gt;[Virtual Channels\Clipboard]&lt;/span&gt;
&lt;span class="s1"&gt;VirtualChannelAllowed=true&lt;/span&gt;
&lt;span class="s1"&gt;ClientClipboard=true&lt;/span&gt;
&lt;span class="s1"&gt;MaxFileSize=0&lt;/span&gt;
&lt;span class="s1"&gt;EOF&lt;/span&gt;

&lt;span class="s1"&gt;        exec /opt/Citrix/ICAClient/wfica /tmp/session.ica&lt;/span&gt;
&lt;span class="s1"&gt;    &amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then make it&amp;nbsp;executable:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chmod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0755&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;~/.local/bin/citrix-open
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There are a few details hiding in the&amp;nbsp;long &lt;code&gt;podman run&lt;/code&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;--rm&lt;/code&gt; means the writable container layer disappears when the session ends. The client configuration is recreated on every&amp;nbsp;launch.&lt;/li&gt;
&lt;li&gt;The &lt;span class="caps"&gt;ICA&lt;/span&gt; file is the only ordinary host file mounted into the container. It is read-only and always appears&amp;nbsp;as &lt;code&gt;/tmp/session.ica&lt;/code&gt;, so unusual characters in the downloaded filename never enter the inner shell&amp;nbsp;command.&lt;/li&gt;
&lt;li&gt;Citrix&amp;#8217;s Linux client is fundamentally an &lt;span class="caps"&gt;ICA&lt;/span&gt;-to-X11 application. On my Wayland desktop it runs through XWayland, hence the X11 socket&amp;nbsp;and &lt;code&gt;DISPLAY&lt;/code&gt; rather than a Wayland&amp;nbsp;socket.&lt;/li&gt;
&lt;li&gt;Modern Fedora provides PulseAudio compatibility through PipeWire. Passing the per-user Pulse socket is sufficient for session audio; no sound server needs to run in the&amp;nbsp;container.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/dev/dri&lt;/code&gt; provides hardware graphics acceleration.&amp;nbsp;Any &lt;code&gt;/dev/video*&lt;/code&gt; nodes present at launch are added for webcam&amp;nbsp;redirection.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;--group-add keep-groups&lt;/code&gt; is important in a rootless container. It retains the host user&amp;#8217;s supplementary group access to devices; adding a group merely&amp;nbsp;named &lt;code&gt;video&lt;/code&gt; inside an Ubuntu image does not guarantee that its numeric &lt;span class="caps"&gt;GID&lt;/span&gt; matches Fedora&amp;#8217;s. This is the mechanism recommended by the &lt;a href="https://docs.podman.io/en/latest/markdown/podman-run.1.html"&gt;Podman run documentation&lt;/a&gt; for group-owned host devices. It requires&amp;nbsp;Podman&amp;#8217;s &lt;code&gt;crun&lt;/code&gt; &lt;span class="caps"&gt;OCI&lt;/span&gt; runtime, which is Fedora&amp;#8217;s&amp;nbsp;default.&lt;/li&gt;
&lt;li&gt;The two files copied&amp;nbsp;from &lt;code&gt;/opt/Citrix/ICAClient/config&lt;/code&gt; avoid&amp;nbsp;the &lt;code&gt;ConfigurationManager&lt;/code&gt; error I otherwise get from an entirely&amp;nbsp;fresh &lt;code&gt;~/.ICAClient&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;My Citrix environment rejects file transfer.&amp;nbsp;Setting &lt;code&gt;MaxFileSize=0&lt;/code&gt; while leaving clipboard support enabled gives me reliable plain-text copy and paste without the client attempting file-copy&amp;nbsp;behaviour.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I intentionally do not&amp;nbsp;use &lt;code&gt;-it&lt;/code&gt;. This script is launched by a graphical file manager, not a terminal, and asking Podman to allocate an interactive &lt;span class="caps"&gt;TTY&lt;/span&gt; there is at best unnecessary and at worst an immediate&amp;nbsp;failure.&lt;/p&gt;
&lt;p&gt;Webcam redirection&amp;nbsp;through &lt;code&gt;/dev/video*&lt;/code&gt; is the only device redirection I rely on. If it is not required,&amp;nbsp;remove &lt;code&gt;/dev/video*&lt;/code&gt; from the loop to give the client a meaningfully smaller view of the host. The same applies to&amp;nbsp;the &lt;code&gt;/dev/dri&lt;/code&gt; patterns when software rendering is&amp;nbsp;acceptable.&lt;/p&gt;
&lt;h2 id="making-ica-files-behave-like-normal-files"&gt;Making &lt;span class="caps"&gt;ICA&lt;/span&gt; Files Behave Like Normal&amp;nbsp;Files&lt;/h2&gt;
&lt;p&gt;The last part is desktop integration. I&amp;nbsp;create &lt;code&gt;~/.local/share/applications/citrix-open.desktop&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Desktop Entry]&lt;/span&gt;
&lt;span class="na"&gt;Name&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Citrix Workspace (Podman)&lt;/span&gt;
&lt;span class="na"&gt;Comment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Launch Citrix ICA sessions in Podman&lt;/span&gt;
&lt;span class="na"&gt;Exec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;citrix-open %f&lt;/span&gt;
&lt;span class="na"&gt;TryExec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;citrix-open&lt;/span&gt;
&lt;span class="na"&gt;Terminal&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;false&lt;/span&gt;
&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Application&lt;/span&gt;
&lt;span class="na"&gt;MimeType&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;application/x-ica&lt;/span&gt;&lt;span class="c1"&gt;;&lt;/span&gt;
&lt;span class="na"&gt;Icon&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;application-x-ica&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This&amp;nbsp;assumes &lt;code&gt;~/.local/bin&lt;/code&gt; is in the desktop&amp;nbsp;session&amp;#8217;s &lt;code&gt;PATH&lt;/code&gt;, as it is on my Fedora installation. If it is&amp;nbsp;not, &lt;code&gt;Exec&lt;/code&gt; must contain the literal absolute path, for&amp;nbsp;example &lt;code&gt;Exec=/home/alice/.local/bin/citrix-open %f&lt;/code&gt;. Tilde and shell-variable expansion do not happen in desktop&amp;nbsp;entries.&lt;/p&gt;
&lt;p&gt;It is tempting to&amp;nbsp;write &lt;code&gt;/home/%U/.local/bin/citrix-open&lt;/code&gt;,&amp;nbsp;but &lt;code&gt;%U&lt;/code&gt; is not a username placeholder. In the &lt;a href="https://specifications.freedesktop.org/desktop-entry/latest/exec-variables.html"&gt;freedesktop.org desktop-entry specification&lt;/a&gt;, it means a list of&amp;nbsp;URLs. &lt;code&gt;%f&lt;/code&gt; is the correct field code for one local&amp;nbsp;file.&lt;/p&gt;
&lt;p&gt;I register the launcher as the default&amp;nbsp;handler:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;update-desktop-database&lt;span class="w"&gt; &lt;/span&gt;~/.local/share/applications
xdg-mime&lt;span class="w"&gt; &lt;/span&gt;default&lt;span class="w"&gt; &lt;/span&gt;citrix-open.desktop&lt;span class="w"&gt; &lt;/span&gt;application/x-ica
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The association can be checked without launching a&amp;nbsp;session:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;xdg-mime&lt;span class="w"&gt; &lt;/span&gt;query&lt;span class="w"&gt; &lt;/span&gt;default&lt;span class="w"&gt; &lt;/span&gt;application/x-ica
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The expected answer&amp;nbsp;is &lt;code&gt;citrix-open.desktop&lt;/code&gt;. From then on, downloading an &lt;span class="caps"&gt;ICA&lt;/span&gt; file and opening it causes the desktop to call the wrapper, Podman creates the container,&amp;nbsp;and &lt;code&gt;wfica&lt;/code&gt; connects to the remote session. When the window closes, the container goes with&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="what-remains-on-the-host"&gt;What Remains on the&amp;nbsp;Host&lt;/h2&gt;
&lt;p&gt;The final host footprint is wonderfully&amp;nbsp;small:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the &lt;code&gt;citrix-workspace&lt;/code&gt; image in rootless container&amp;nbsp;storage&lt;/li&gt;
&lt;li&gt;&lt;code&gt;~/.local/bin/citrix-open&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;~/.local/share/applications/citrix-open.desktop&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Podman itself, which was already&amp;nbsp;there&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There is no Citrix &lt;span class="caps"&gt;RPM&lt;/span&gt;, no vendor repository, no service enabled as root, and no archaeological dig&amp;nbsp;through &lt;code&gt;/opt&lt;/code&gt; when I stop needing the client. Updating is equally explicit: download the new Debian package, verify it, rebuild the image, and replace the old image once I have tested a&amp;nbsp;session.&lt;/p&gt;
&lt;p&gt;This is the part of containers on the desktop I find most useful. It is not about pretending every graphical application is a cloud-native microservice. It is about ownership. All Citrix-shaped state belongs to one image and two tiny integration files. Fedora remains Fedora, and the proprietary thing I occasionally need stays a thing I can remove in one&amp;nbsp;piece.&lt;/p&gt;</content><category term="Linux"/><category term="citrix"/><category term="citrix workspace"/><category term="ica"/><category term="podman"/><category term="containers"/><category term="fedora"/><category term="linux"/><category term="rootless"/><category term="x11"/></entry><entry><title>Crawlfest: Building a Terminal Roguelike in Rust</title><link href="https://hofstede.it/crawlfest-building-a-terminal-roguelike-in-rust/" rel="alternate"/><published>2026-08-16T00:00:00+02:00</published><updated>2026-08-16T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-08-16:/crawlfest-building-a-terminal-roguelike-in-rust/</id><summary type="html">&lt;p&gt;For a few months now I have been building a terminal roguelike in Rust in the spirit of &lt;span class="caps"&gt;ADOM&lt;/span&gt; and NetHack. This article is about the technical groundwork: why ratatui and legion instead of a game engine, how the four map generators work (rooms, cellular-automata caverns, hand-drawn prefabs, and a hybrid of the last two), how field of view and A* pathfinding hook into my own map type through two traits, how content lives in &lt;span class="caps"&gt;RON&lt;/span&gt; files instead of code, and how save games avoid serialising the &lt;span class="caps"&gt;ECS&lt;/span&gt; at&amp;nbsp;all.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="The crawlfest title screen in a terminal, ASCII art logo above a menu" src="https://hofstede.it/images/2026-08-16-crawlfest-title-screen.png" title="A dungeon crawl of statistically doomed away missions"&gt;&lt;/p&gt;
&lt;p&gt;I have wanted to write a roguelike since roughly the first time &lt;span class="caps"&gt;ADOM&lt;/span&gt; killed me in the Small Cave. A few months ago I finally started one, mostly because I wanted a Rust project that was large enough to be interesting and small enough to finish a piece of it in an&amp;nbsp;evening.&lt;/p&gt;
&lt;p&gt;The result is &lt;strong&gt;crawlfest&lt;/strong&gt;: a turn-based &lt;span class="caps"&gt;ASCII&lt;/span&gt; roguelike that runs in a real terminal. Procedurally generated multi-floor dungeons, permadeath, field of view, monsters that pathfind to you, loot, equipment, status effects, and a tombstone at the end. The setting is a fantasy world with a wrecked starship scattered through its lower levels, which is why the dungeon contains both kobolds and&amp;nbsp;hyposprays.&lt;/p&gt;
&lt;p&gt;This is not a release announcement. The game is unfinished, the source is not public, and I might publish it at some point. What follows is a write-up of the technical groundwork that already exists, because the architecture came together cleanly enough to be worth documenting: the framework choices, the map generation, the field of view implementation, the &lt;span class="caps"&gt;ECS&lt;/span&gt; layout, and the parts I got wrong on the first&amp;nbsp;attempt.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-rust-and-why-not-a-game-engine"&gt;Why Rust, and why not a game engine&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-bracket-lib-detour"&gt;The bracket-lib&amp;nbsp;detour&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#architecture-in-one-paragraph"&gt;Architecture in one paragraph&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-turn-state-machine"&gt;The turn state&amp;nbsp;machine&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-map"&gt;The map&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#two-traits-and-other-peoples-algorithms-work-on-my-data"&gt;Two traits, and other people&amp;#8217;s algorithms work on my&amp;nbsp;data&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#map-generation"&gt;Map generation&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-contract-that-keeps-the-spawner-shape-agnostic"&gt;The contract that keeps the spawner&amp;nbsp;shape-agnostic&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rooms-and-corridors"&gt;Rooms and&amp;nbsp;corridors&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#caverns-via-cellular-automata"&gt;Caverns via cellular&amp;nbsp;automata&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#prefabs-levels-drawn-by-hand"&gt;Prefabs: levels drawn by&amp;nbsp;hand&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#hybrid-levels-and-a-test-that-proves-a-design-claim"&gt;Hybrid levels, and a test that proves a design&amp;nbsp;claim&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#which-generator-does-a-floor-use"&gt;Which generator does a floor&amp;nbsp;use&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#field-of-view"&gt;Field of view&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#field-of-view-as-a-game-rule"&gt;Field of view as a game&amp;nbsp;rule&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-camera"&gt;The&amp;nbsp;camera&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#monsters-that-think"&gt;Monsters that&amp;nbsp;think&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#combat-rules-as-pure-functions"&gt;Combat rules as pure&amp;nbsp;functions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#content-lives-in-ron-not-in-rust"&gt;Content lives in &lt;span class="caps"&gt;RON&lt;/span&gt;, not in Rust&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#danger-instead-of-depth-cutoffs"&gt;Danger instead of depth&amp;nbsp;cutoffs&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#status-effects"&gt;Status&amp;nbsp;effects&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#save-games-without-serialising-the-ecs"&gt;Save games without serialising the &lt;span class="caps"&gt;ECS&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#testing-something-this-random"&gt;Testing something this&amp;nbsp;random&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-is-next"&gt;What is&amp;nbsp;next&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="why-rust-and-why-not-a-game-engine"&gt;Why Rust, and why not a game&amp;nbsp;engine&lt;/h2&gt;
&lt;p&gt;The project started as a Rust learning exercise, so the language was the point rather than a decision. The interesting decisions were everything around&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;A roguelike in a terminal needs surprisingly little: a way to paint characters into a grid, a way to read keys, random numbers, and two classic algorithms (field of view and pathfinding). It does not need a renderer, a physics engine, an asset pipeline, or a scene graph. Bevy or Godot would have been a lot of machinery around a game whose entire visual output is a few thousand coloured&amp;nbsp;characters.&lt;/p&gt;
&lt;p&gt;The stack I ended up&amp;nbsp;with:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concern&lt;/th&gt;
&lt;th&gt;Choice&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Language&lt;/td&gt;
&lt;td&gt;Rust, edition 2024&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Terminal &lt;span class="caps"&gt;UI&lt;/span&gt; and input&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ratatui&lt;/code&gt; 0.30&amp;nbsp;with &lt;code&gt;crossterm&lt;/code&gt; 0.29&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;span class="caps"&gt;ECS&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;legion&lt;/code&gt; 0.4&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Roguelike algorithms&lt;/td&gt;
&lt;td&gt;&lt;code&gt;bracket-pathfinding&lt;/code&gt;, &lt;code&gt;bracket-random&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Data-driven content&lt;/td&gt;
&lt;td&gt;&lt;code&gt;serde&lt;/code&gt; and &lt;code&gt;ron&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Everything is pure&amp;nbsp;Rust. &lt;code&gt;cargo build&lt;/code&gt; needs no system libraries, which matters more than it sounds like, as the next section&amp;nbsp;explains.&lt;/p&gt;
&lt;h3 id="the-bracket-lib-detour"&gt;The bracket-lib&amp;nbsp;detour&lt;/h3&gt;
&lt;p&gt;The obvious starting point for a Rust roguelike&amp;nbsp;is &lt;code&gt;bracket-lib&lt;/code&gt;, the library that accompanies Herbert Wolverson&amp;#8217;s book &lt;em&gt;Hands-on Rust&lt;/em&gt;. It bundles terminal emulation, field of view, pathfinding, dice, and a colour system. I planned to use it and abandoned it within an&amp;nbsp;hour.&lt;/p&gt;
&lt;p&gt;The problem is&amp;nbsp;packaging. &lt;code&gt;bracket-terminal 0.8.7&lt;/code&gt; does offer several backends, including&amp;nbsp;a &lt;code&gt;cross_term&lt;/code&gt; one that draws into the terminal you are already in.&amp;nbsp;But &lt;code&gt;winit&lt;/code&gt; is not an optional dependency of the crate, so you pay for the graphical stack whichever backend you select. Here is the reverse dependency chain, in the&amp;nbsp;direction &lt;code&gt;cargo tree -i&lt;/code&gt; prints it, from the offending system library up to the crate that pulled it&amp;nbsp;in:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;servo-fontconfig-sys &amp;lt;- servo-fontconfig &amp;lt;- crossfont &amp;lt;- sctk-adwaita &amp;lt;- winit &amp;lt;- bracket-terminal
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On Linux that means&amp;nbsp;system &lt;code&gt;fontconfig&lt;/code&gt;, &lt;code&gt;freetype&lt;/code&gt; and X development packages, plus a CMake policy workaround, in order to build a game that draws &lt;span class="caps"&gt;ASCII&lt;/span&gt;. The default backend is OpenGL, which also opens a graphical window that emulates a terminal rather than using the terminal I am already sitting&amp;nbsp;in.&lt;/p&gt;
&lt;p&gt;So I split the difference. The rendering and input layer&amp;nbsp;became &lt;code&gt;ratatui&lt;/code&gt; plus &lt;code&gt;crossterm&lt;/code&gt;, which are pure Rust, build in seconds, and paint into the actual terminal. The algorithms came&amp;nbsp;from &lt;code&gt;bracket-pathfinding&lt;/code&gt; and &lt;code&gt;bracket-random&lt;/code&gt;, which are standalone sub-crates with no &lt;span class="caps"&gt;GUI&lt;/span&gt; dependencies at all. I get Wolverson&amp;#8217;s well-tested shadowcasting and A* implementations without any of the packaging pain, and the game runs in Konsole, tmux, or over &lt;span class="caps"&gt;SSH&lt;/span&gt; like a roguelike&amp;nbsp;should.&lt;/p&gt;
&lt;h2 id="architecture-in-one-paragraph"&gt;Architecture in one&amp;nbsp;paragraph&lt;/h2&gt;
&lt;p&gt;Entities are bundles of components. Behaviour lives in systems, which are functions over component queries. Shared singletons such as the map and the message log live&amp;nbsp;in &lt;code&gt;legion::Resources&lt;/code&gt;. Content lives in &lt;span class="caps"&gt;RON&lt;/span&gt; files. Adding a feature is almost always a new component plus a system, or a new &lt;span class="caps"&gt;RON&lt;/span&gt; entry, and rarely a change to existing&amp;nbsp;code.&lt;/p&gt;
&lt;p&gt;That last sentence is the actual design goal. A hobby project that gets touched in bursts has to be additive, otherwise every session starts with an hour of remembering how things fit&amp;nbsp;together.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="sd"&gt;/// Where an entity sits on the map, in grid cells.&lt;/span&gt;
&lt;span class="cp"&gt;#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Position&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="sd"&gt;/// How an entity is drawn, and who wins when several share a tile.&lt;/span&gt;
&lt;span class="cp"&gt;#[derive(Clone, Copy, Debug)]&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Renderable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;glyph&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;fg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Color&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;render_order&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="sd"&gt;/// A marker component: no data, it just tags an entity as a monster.&lt;/span&gt;
&lt;span class="cp"&gt;#[derive(Clone, Copy, Debug)]&lt;/span&gt;
&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Monster&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The player is an entity&amp;nbsp;with &lt;code&gt;Position&lt;/code&gt;, &lt;code&gt;Renderable&lt;/code&gt;, &lt;code&gt;CombatStats&lt;/code&gt;, &lt;code&gt;Attributes&lt;/code&gt; and &lt;code&gt;Level&lt;/code&gt;. A monster is the same&amp;nbsp;minus &lt;code&gt;Level&lt;/code&gt;,&amp;nbsp;plus &lt;code&gt;Monster&lt;/code&gt; and &lt;code&gt;BlocksTile&lt;/code&gt;. An item on the floor has&amp;nbsp;a &lt;code&gt;Position&lt;/code&gt;; the same item in your backpack&amp;nbsp;has &lt;code&gt;InBackpack&lt;/code&gt; instead. Picking something up is literally swapping one component for&amp;nbsp;another.&lt;/p&gt;
&lt;h3 id="the-turn-state-machine"&gt;The turn state&amp;nbsp;machine&lt;/h3&gt;
&lt;p&gt;Roguelikes are turn-based, which in code means the world only advances when the player acts. That is one&amp;nbsp;enum:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;AwaitingInput -&amp;gt; PlayerTurn -&amp;gt; MonsterTurn -&amp;gt; AwaitingInput
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;RunState&lt;/code&gt; holds those three plus every screen that is really just a different input mode: the main menu, name entry, the load menu, the inventory, the drop menu, look mode, the cheat menu, and the game-over screen. The main loop redraws every iteration, then either waits for a key (input states) or runs systems and immediately advances (turn&amp;nbsp;states).&lt;/p&gt;
&lt;p&gt;Redrawing &lt;em&gt;between&lt;/em&gt; the two turn states is a small detail with a large effect on feel: you see your own move land before the monsters answer it, instead of both happening in the same&amp;nbsp;frame.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;PlayerTurn&lt;/code&gt; counts the turn, resolves melee, removes the dead, and recomputes field of&amp;nbsp;view. &lt;code&gt;MonsterTurn&lt;/code&gt; runs the &lt;span class="caps"&gt;AI&lt;/span&gt;, resolves melee again, removes the dead, ticks status effects, and removes the dead once more, because poison can kill after the fight that inflicted it is&amp;nbsp;over.&lt;/p&gt;
&lt;h2 id="the-map"&gt;The&amp;nbsp;map&lt;/h2&gt;
&lt;p&gt;The map is the least clever part of the codebase and deliberately&amp;nbsp;so:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tiles&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;TileType&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;revealed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;visible&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A 2D grid stored as one flat vector in row-major order. It is cache-friendly, trivial to clone, and trivial to serialise into a save file. One helper converts coordinates into an index, and after that nobody in the codebase&amp;nbsp;writes &lt;code&gt;y * width + x&lt;/code&gt; by&amp;nbsp;hand:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;xy_idx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;usize&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A fresh map is entirely wall. Generators carve floor into a solid block of rock, which makes &amp;#8220;the default is wall&amp;#8221; a convenient invariant: anything a generator forgets to touch stays safely&amp;nbsp;solid.&lt;/p&gt;
&lt;h3 id="two-traits-and-other-peoples-algorithms-work-on-my-data"&gt;Two traits, and other people&amp;#8217;s algorithms work on my&amp;nbsp;data&lt;/h3&gt;
&lt;p&gt;This is the part of the design I like most, and it is very Rust. I do not inherit from a framework map class. I implement two traits on my own type, and the library&amp;#8217;s algorithms start working on&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;BaseMap&lt;/code&gt; answers three questions: is this tile opaque, what are its walkable neighbours and at what cost, and what is the heuristic distance between two&amp;nbsp;tiles.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;impl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;BaseMap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Map&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;is_opaque&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tiles&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;is_none_or&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;TileType&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;Wall&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;get_available_exits&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;SmallVec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;f32&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;exits&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SmallVec&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;%&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ny&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;is_blocked&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ny&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="n"&gt;exits&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;push&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;xy_idx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;nx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ny&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;1.0&lt;/span&gt;&lt;span class="p"&gt;));&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;exits&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;get_pathing_distance&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idx1&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idx2&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;f32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;// straight-line distance: a good guess on open floor&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;impl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Algorithm2D&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Map&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;dimensions&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Point&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;Point&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the entire integration&amp;nbsp;surface. &lt;code&gt;is_opaque&lt;/code&gt; gives me shadowcasting field of&amp;nbsp;view. &lt;code&gt;get_available_exits&lt;/code&gt; plus &lt;code&gt;get_pathing_distance&lt;/code&gt; give me A*. And because my own map generators&amp;nbsp;reuse &lt;code&gt;get_available_exits&lt;/code&gt; for their connectivity checks, &amp;#8220;reachable during generation&amp;#8221; and &amp;#8220;walkable during play&amp;#8221; can never drift apart. There is exactly one definition of what counts as a step, and everything else asks&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;Movement is cardinal only, so the exits list is at most four&amp;nbsp;entries. &lt;code&gt;SmallVec&lt;/code&gt; keeps those inline on the stack instead of heap-allocating for every node A*&amp;nbsp;expands.&lt;/p&gt;
&lt;h2 id="map-generation"&gt;Map&amp;nbsp;generation&lt;/h2&gt;
&lt;p&gt;Levels are built by anything implementing one&amp;nbsp;method:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;trait&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MapBuilder&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&lt;/span&gt;&lt;span class="k"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kp"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nc"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;RandomNumberGenerator&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;BuiltMap&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;struct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;BuiltMap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;player_start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Position&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;spawn_regions&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;SpawnRegion&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flavour&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Option&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&amp;amp;&amp;#39;&lt;/span&gt;&lt;span class="nb"&gt;static&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;str&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There are four implementations today, and the floor decides which one it wants at runtime through&amp;nbsp;a &lt;code&gt;Box&amp;lt;dyn MapBuilder&amp;gt;&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="the-contract-that-keeps-the-spawner-shape-agnostic"&gt;The contract that keeps the spawner&amp;nbsp;shape-agnostic&lt;/h3&gt;
&lt;p&gt;The first version of the spawner took a list of rooms. That worked fine until the second generator produced caves, which have no rooms, and the third produced one enormous hall, which has exactly&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;The fix was to change what generators hand back.&amp;nbsp;A &lt;code&gt;SpawnRegion&lt;/code&gt; is just a group of walkable tiles, and the contract is that &lt;strong&gt;region 0 contains the player&amp;#8217;s start&lt;/strong&gt;, so the spawner skips it and you never materialise nose to nose with a Borg drone. A rooms level reports one region per room. A cave or a prefab level dices its open floor into rectangular chunks of roughly room size and reports those. The spawner does not know or care which it got: it walks the regions, rolls a monster for some of them, and places&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;flavour&lt;/code&gt; is an optional line for the message log, so a distinctive level style can announce itself in the game&amp;#8217;s voice when you&amp;nbsp;arrive.&lt;/p&gt;
&lt;h3 id="rooms-and-corridors"&gt;Rooms and&amp;nbsp;corridors&lt;/h3&gt;
&lt;p&gt;The classic, and still the backbone of the game. Try to place thirty random rectangles, keep each one only if it does not intersect an already placed room, and connect each new room to the previous one with an L-shaped tunnel, flipping a coin for which leg comes first so the result does not look too&amp;nbsp;regular.&lt;/p&gt;
&lt;p&gt;Two details are worth noting.&amp;nbsp;First, &lt;code&gt;carve_room&lt;/code&gt; floors the cells &lt;em&gt;inside&lt;/em&gt; the rectangle, which leaves a one-tile rock wall between rooms that happen to sit next to each other. Second, the carving core takes a bounds&amp;nbsp;rectangle:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;super&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;rooms_and_corridors&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kp"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nc"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kp"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nc"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;RandomNumberGenerator&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;bounds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Rect&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="n"&gt;Rect&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Every carved tile stays strictly inside those bounds. That single parameter is what makes the hybrid generator further down possible, because it can run the whole classic algorithm on half a map and trust that nothing leaks into the other&amp;nbsp;half.&lt;/p&gt;
&lt;h3 id="caverns-via-cellular-automata"&gt;Caverns via cellular&amp;nbsp;automata&lt;/h3&gt;
&lt;p&gt;Organic caves, in three&amp;nbsp;moves.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Noise.&lt;/strong&gt; Fill the interior with random rubble, roughly 55 percent floor. The border stays solid so nothing can leak off the map&amp;nbsp;edge.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Smoothing.&lt;/strong&gt; Apply a local rule five times: a tile becomes wall if five or more of the nine cells in its neighbourhood (itself plus its eight neighbours) are wall, otherwise floor. Each pass erodes lone pillars and fills lone potholes, and static dissolves into caves. This is a cellular automaton in the Game of Life sense: same machinery, different&amp;nbsp;rule.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;_pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;..&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;smoothed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;..&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;..&lt;/span&gt;&lt;span class="bp"&gt;self&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;walls&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;..=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dx&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="o"&gt;..=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tile&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dy&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TileType&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;Floor&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                        &lt;/span&gt;&lt;span class="n"&gt;walls&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;                    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;walls&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="n"&gt;smoothed&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set_floor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;smoothed&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The double buffer matters. Each pass reads the old map and writes a fresh one. Mutating in place lets early writes in a pass influence later reads, which smears the caves diagonally. I know this because my first version did exactly that and produced caves that all leaned the same&amp;nbsp;way.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Connectivity.&lt;/strong&gt; A cellular automaton reliably produces one big cave plus a handful of sealed pockets. I flood-fill from every not-yet-seen floor tile using a breadth-first search, keep the largest connected area, and fill every unreachable pocket back in with rock. Everything you can see, you can walk&amp;nbsp;to.&lt;/p&gt;
&lt;p&gt;The same &lt;span class="caps"&gt;BFS&lt;/span&gt; then places the stairs on the reachable tile &lt;em&gt;farthest&lt;/em&gt; from the entrance, measured in steps rather than straight-line distance. That one line is what makes a shapeless cave play like a journey instead of a lucky&amp;nbsp;spawn.&lt;/p&gt;
&lt;p&gt;One &lt;span class="caps"&gt;BFS&lt;/span&gt; helper serves three jobs across the whole codebase: connectivity checks, farthest-tile queries, and the&amp;nbsp;tests.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;bfs_distances&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kp"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Position&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Vec&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="nb"&gt;Option&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="kt"&gt;u32&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="fm"&gt;vec!&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="nb"&gt;None&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;usize&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;start_idx&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;xy_idx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;dist&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;start_idx&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;queue&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;VecDeque&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;from&lt;/span&gt;&lt;span class="p"&gt;([&lt;/span&gt;&lt;span class="n"&gt;start_idx&lt;/span&gt;&lt;span class="p"&gt;]);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;queue&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;pop_front&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;here&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dist&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;expect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;queued tiles always have a distance&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;_cost&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;get_available_exits&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dist&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;].&lt;/span&gt;&lt;span class="n"&gt;is_none&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="n"&gt;dist&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;here&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="n"&gt;queue&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;push_back&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;dist&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Cellular automata occasionally roll a cramped dud, so generation is generate-and-check: accept the first candidate whose main cave covers a quarter of the map, and after ten attempts take whatever came out, because a small cave still&amp;nbsp;plays.&lt;/p&gt;
&lt;h3 id="prefabs-levels-drawn-by-hand"&gt;Prefabs: levels drawn by&amp;nbsp;hand&lt;/h3&gt;
&lt;p&gt;Some places should not be random. A set piece is &lt;em&gt;designed&lt;/em&gt;, it is the same every visit, and that is the point. So it lives as a text drawing&amp;nbsp;in &lt;code&gt;assets/prefabs/&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;######################
#....................#
#..#....#....#....#..#
#....................#
#..#....#....#....#..#
+..................&amp;gt;.#
#..#....#....#....#..#
#....................#
#..#....#....#....#..#
#....................#
######################
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The glyph language is close to what the renderer&amp;nbsp;shows: &lt;code&gt;#&lt;/code&gt; wall, &lt;code&gt;.&lt;/code&gt; floor, &lt;code&gt;&amp;gt;&lt;/code&gt; down&amp;nbsp;staircase, &lt;code&gt;@&lt;/code&gt; the player&amp;nbsp;start, &lt;code&gt;+&lt;/code&gt; the vault&amp;#8217;s single doorway, and a space meaning &amp;#8220;leave whatever the map already had&amp;#8221;, which lets vaults have irregular&amp;nbsp;edges.&lt;/p&gt;
&lt;p&gt;The files are pulled in&amp;nbsp;with &lt;code&gt;include_str!&lt;/code&gt;, so they are embedded at compile time. That is a deliberate contrast with the &lt;span class="caps"&gt;RON&lt;/span&gt; content files, which are read at runtime specifically so I can rebalance a monster without recompiling. Monster stats are balance data. A prefab is level structure, which is code by another name, and embedding it means a missing or broken drawing is a compile or test failure instead of a panic three floors into a&amp;nbsp;run.&lt;/p&gt;
&lt;p&gt;Two things consume&amp;nbsp;prefabs. &lt;code&gt;PrefabLevelBuilder&lt;/code&gt; turns one drawing into a whole fixed level (the Big Room, a NetHack homage: one vast hall, nowhere to hide, for&amp;nbsp;anyone). &lt;code&gt;HybridBuilder&lt;/code&gt; stamps one into a corner of an otherwise procedural&amp;nbsp;map.&lt;/p&gt;
&lt;h3 id="hybrid-levels-and-a-test-that-proves-a-design-claim"&gt;Hybrid levels, and a test that proves a design&amp;nbsp;claim&lt;/h3&gt;
&lt;p&gt;Depth 3 is the level I am happiest with. The west half is the ordinary rooms-and-corridors algorithm, confined to a bounds rectangle. The east half is the temple vault stamped into untouched rock. Between the two sits a two-column moat of guaranteed rock, and exactly one L-shaped corridor connecting a room to the vault&amp;#8217;s&amp;nbsp;single &lt;code&gt;+&lt;/code&gt; doorway. The stairs are inside the vault, so to go deeper you have to walk into the set&amp;nbsp;piece.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;+--------------------------------+------------+
|  rooms &amp;amp; corridors (random)    :  ######### |
|                                :  #.......# |
|   [room]----[room]             :  +...&amp;gt;...# |
|     |         ^................:..^######## |
|   [room]      &amp;#39;-- the ONE corridor          |
+--------------------------------+------------+
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Exactly one way in&amp;#8221; is a design claim, and design claims rot. So it is a test. The vault sits against the east edge behind the moat, which means every path into it must cross the column just west of its wall. Counting the walkable tiles in that column has to yield exactly&amp;nbsp;one:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;vault_x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;78&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Prefab&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;parse&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;TEMPLE&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;crossings&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="o"&gt;..&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;|&amp;amp;&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="o"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;is_blocked&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;vault_x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;count&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;
&lt;span class="fm"&gt;assert_eq!&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;crossings&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;expected exactly one corridor into the vault&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Because generation is random, the test runs several seeded iterations and asserts invariants rather than any particular shape. The cavern test does the same: every walkable tile must be reachable from the entrance, and every cave must have&amp;nbsp;stairs.&lt;/p&gt;
&lt;h3 id="which-generator-does-a-floor-use"&gt;Which generator does a floor&amp;nbsp;use&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;builder_for_depth&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;depth&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kp"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nc"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;RandomNumberGenerator&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Box&lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="k"&gt;dyn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MapBuilder&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;match&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;depth&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SimpleMapBuilder&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MAP_WIDTH&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MAP_HEIGHT&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;HybridBuilder&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;temple&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MAP_WIDTH&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MAP_HEIGHT&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;PrefabLevelBuilder&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;big_room&lt;/span&gt;&lt;span class="p"&gt;()),&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;range&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nb"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SimpleMapBuilder&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MAP_WIDTH&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MAP_HEIGHT&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nb"&gt;Box&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;CavernBuilder&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;MAP_WIDTH&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;MAP_HEIGHT&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Floor 1 is always the familiar rooms, because a first impression should be legible. Floors 3 and 5 are fixed landmarks, so runs share a shape of memory. Everything else is a coin flip between rooms and&amp;nbsp;caves.&lt;/p&gt;
&lt;h2 id="field-of-view"&gt;Field of&amp;nbsp;view&lt;/h2&gt;
&lt;p&gt;&lt;img alt="The crawlfest dungeon at depth 2, showing lit and remembered tiles, a kobold and an item" src="https://hofstede.it/images/2026-08-16-crawlfest-dungeon.png" title="Bright tiles are visible now, dim blue tiles are remembered, black is unexplored"&gt;&lt;/p&gt;
&lt;p&gt;The screenshot above is really a screenshot of the &lt;span class="caps"&gt;FOV&lt;/span&gt; implementation. Bright grey walls and floor are what the player can see right now. The dim blue-grey structure below is what the player &lt;em&gt;remembers&lt;/em&gt; from earlier. Everything else is black, because it has never been&amp;nbsp;seen.&lt;/p&gt;
&lt;p&gt;Those are exactly the two boolean vectors on the&amp;nbsp;map:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;revealed[i]&lt;/code&gt;: has the player ever seen&amp;nbsp;tile &lt;code&gt;i&lt;/code&gt;? It accumulates and is never cleared, and it lives as long as the level&amp;nbsp;does.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;visible[i]&lt;/code&gt;: can the player see&amp;nbsp;tile &lt;code&gt;i&lt;/code&gt; right now? Cleared and recomputed on every&amp;nbsp;move.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The geometry itself is symmetric shadowcasting&amp;nbsp;from &lt;code&gt;bracket-pathfinding&lt;/code&gt;, which works on my map because of&amp;nbsp;the &lt;code&gt;is_opaque&lt;/code&gt; implementation shown earlier. My own &lt;span class="caps"&gt;FOV&lt;/span&gt; module is a dozen lines of&amp;nbsp;glue:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;update_fov&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kp"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nc"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Position&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;range&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clear_visible&lt;/span&gt;&lt;span class="p"&gt;();&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;center&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Point&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;origin&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pt&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;FieldOfViewAlg&lt;/span&gt;&lt;span class="p"&gt;::&lt;/span&gt;&lt;span class="n"&gt;SymmetricShadowcasting&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;field_of_view&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;center&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;range&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;*&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;in_bounds&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;xy_idx&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;pt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;visible&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;revealed&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Writing this myself would have meant implementing octant transforms and slope bookkeeping, which is a well-solved problem I had no interest in solving again. Delegating the geometry and owning the two flags is the right&amp;nbsp;split.&lt;/p&gt;
&lt;p&gt;Choosing the algorithm explicitly matters more than it looks, and I got this wrong at first. The crate&amp;#8217;s&amp;nbsp;bare &lt;code&gt;field_of_view&lt;/code&gt; function is not the symmetric implementation. It is an alias for the older recursive one, kept as the default for backwards compatibility, and the symmetric variant has to be asked for by name. I had written &amp;#8220;symmetric shadowcasting&amp;#8221; in a comment and then called the function that does not do it, which is an easy mistake to make and an invisible one to play, because both algorithms produce a perfectly plausible-looking lit area. It only matters because of what the next section does with the&amp;nbsp;result.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;&amp;amp;*map&lt;/code&gt; in there is a small piece of borrow-checker choreography that is worth understanding: it reborrows&amp;nbsp;the &lt;code&gt;&amp;amp;mut Map&lt;/code&gt; as the shared&amp;nbsp;reference &lt;code&gt;field_of_view&lt;/code&gt; wants. The borrow ends when the call returns its owned vector, which leaves the function free to mutate the map immediately&amp;nbsp;afterwards.&lt;/p&gt;
&lt;h3 id="field-of-view-as-a-game-rule"&gt;Field of view as a game&amp;nbsp;rule&lt;/h3&gt;
&lt;p&gt;Once you have visibility as data, other systems can just read it, and two of them&amp;nbsp;do.&lt;/p&gt;
&lt;p&gt;The renderer picks per tile between full colour, a dim remembered blue, and nothing at all. Entities are only drawn if they stand on a currently visible tile, so no monster is ever visible lurking in the&amp;nbsp;dark.&lt;/p&gt;
&lt;p&gt;The monster &lt;span class="caps"&gt;AI&lt;/span&gt; uses &lt;strong&gt;&lt;span class="caps"&gt;FOV&lt;/span&gt; symmetry&lt;/strong&gt; for aggro. Field of view is mutual: if the player can see the tile a monster is standing on, that monster can see the player. So the entire aggro check is one array&amp;nbsp;lookup:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;visible&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;continue&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is cheap, it needs no per-monster &lt;span class="caps"&gt;FOV&lt;/span&gt; computation, and it has a pleasant side effect: nothing shuffles around off-screen. The dungeon only comes alive where you are looking at it, which is both a performance property and a design&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;The inference is only sound if visibility really is mutual, which is the whole reason the algorithm choice above is not a detail. Symmetric shadowcasting guarantees it for transparent tiles, and a monster always stands on a walkable, non-opaque tile, so the case the &lt;span class="caps"&gt;AI&lt;/span&gt; asks about is exactly the case the guarantee covers. With the recursive algorithm the rule was quietly approximate: a creature could aggro from a tile it could not actually see you from, or fail to notice you from one it could. That is now a test, which seeds a room full of pillars and asserts that every visible floor tile can see the origin back. It fails immediately if anyone swaps the call back to the convenient&amp;nbsp;default.&lt;/p&gt;
&lt;h2 id="the-camera"&gt;The&amp;nbsp;camera&lt;/h2&gt;
&lt;p&gt;The map is 78 by 40, usually larger than the visible terminal area, so the renderer draws a window into it. The camera holds exactly one piece of state, the world coordinate of the viewport&amp;#8217;s top-left corner, and everything else is two coordinate&amp;nbsp;conversions.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;new&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kp"&gt;&amp;amp;&lt;/span&gt;&lt;span class="nc"&gt;Map&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;view_w&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;view_h&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;focus&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Position&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nc"&gt;Self&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="bp"&gt;Self&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;left&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;focus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;x&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;view_w&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;clamp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;width&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;view_w&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;top&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;focus&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;y&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;view_h&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;clamp&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;height&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;view_h&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;max&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Subtracting half the viewport puts the focus dead centre, and the clamp pulls the window back onto the map near the edges. Clamping rather than showing void beyond the border is the classic roguelike feel: the view stops scrolling when you approach an edge and&amp;nbsp;your &lt;code&gt;@&lt;/code&gt; drifts off centre.&amp;nbsp;The &lt;code&gt;.max(0)&lt;/code&gt; handles maps smaller than the viewport, where clamp&amp;#8217;s upper bound would otherwise fall below its lower bound and&amp;nbsp;panic.&lt;/p&gt;
&lt;p&gt;The map layer is drawn by walking &lt;em&gt;screen&lt;/em&gt; cells and asking the camera which world tile each one shows. Going in that direction means touching exactly the cells that fit on screen and never iterating over the parts of the map that are scrolled out of view. Entities go the other way, from world to screen, because there are few of them and most are&amp;nbsp;off-screen.&lt;/p&gt;
&lt;p&gt;Drawing happens in three layers: map, then entities sorted&amp;nbsp;by &lt;code&gt;render_order&lt;/code&gt; descending so the lowest number is painted last and ends up on top, then the look cursor. The player is order 0 and can therefore never be hidden by&amp;nbsp;anything.&lt;/p&gt;
&lt;h2 id="monsters-that-think"&gt;Monsters that&amp;nbsp;think&lt;/h2&gt;
&lt;p&gt;Monster &lt;span class="caps"&gt;AI&lt;/span&gt; is a three-phase pass, and the phases exist because of the borrow checker rather than in spite of&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Read.&lt;/strong&gt; Collect every monster, its position, and whether it is confused. Build a set of occupied tiles (all monsters plus the player) and a map from tile to&amp;nbsp;occupant.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Decide.&lt;/strong&gt; Borrow the map immutably. Each monster either attacks (Manhattan distance 1 from the player), or paths one step closer with A*, or stumbles randomly if it is confused. Nothing is mutated here, only decisions collected into&amp;nbsp;vectors.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Write.&lt;/strong&gt; Drop the map borrow, then apply. Attackers get&amp;nbsp;a &lt;code&gt;WantsToMelee&lt;/code&gt; component, movers get&amp;nbsp;their &lt;code&gt;Position&lt;/code&gt; updated, and the occupancy set is kept in sync as moves land so two monsters cannot claim the same tile within one&amp;nbsp;turn.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a_star_search&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;idx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;player_idx&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;*&lt;/span&gt;&lt;span class="n"&gt;map&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;success&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;len&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;next&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;path&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;steps&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;];&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// steps[0] is the monster&amp;#39;s own tile&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note that the &lt;span class="caps"&gt;AI&lt;/span&gt; never applies damage. It only &lt;em&gt;tags&lt;/em&gt; an attacker with an intent component, and a&amp;nbsp;separate &lt;code&gt;resolve_melee&lt;/code&gt; pass turns intents into damage and log lines. That is the classic &lt;span class="caps"&gt;ECS&lt;/span&gt; intent pattern, and it pays off immediately: the player&amp;#8217;s bump attack, a monster&amp;#8217;s attack, and a confused monster stumbling into another monster all produce the same intent and are resolved by the same code path. Monster versus monster brawls came out of that for free, which is the sort of thing that makes an &lt;span class="caps"&gt;ECS&lt;/span&gt; worth the&amp;nbsp;ceremony.&lt;/p&gt;
&lt;h2 id="combat-rules-as-pure-functions"&gt;Combat rules as pure&amp;nbsp;functions&lt;/h2&gt;
&lt;p&gt;The combat module has no &lt;span class="caps"&gt;ECS&lt;/span&gt; types in it at all. It is arithmetic, which makes it trivially&amp;nbsp;testable.&lt;/p&gt;
&lt;p&gt;The model is a light d20 system in &lt;span class="caps"&gt;ADOM&lt;/span&gt;&amp;#8217;s clothing. Every combatant has an attack rating and a defense value. A swing rolls 1d20 and connects if the roll plus attack reaches the defense value, with a natural 1 always missing and a natural 20 always hitting, so no fight is ever a mathematical&amp;nbsp;certainty.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;pub&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;hit_connects&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;roll&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;att&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;i32&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;-&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;bool&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;match&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;roll&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;true&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;_&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;roll&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;att&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dv&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Taking the roll as a parameter instead of an &lt;span class="caps"&gt;RNG&lt;/span&gt; reference is what makes this a pure function of its inputs, and the tests can then probe every edge exactly instead of rolling dice and&amp;nbsp;hoping.&lt;/p&gt;
&lt;p&gt;Damage is dice notation, the lingua franca of the genre, parsed from strings&amp;nbsp;like &lt;code&gt;2d6+1&lt;/code&gt; into a&amp;nbsp;three-integer &lt;code&gt;Copy&lt;/code&gt; struct. Degenerate dice such&amp;nbsp;as &lt;code&gt;0d6&lt;/code&gt; are rejected at parse time, and the &lt;span class="caps"&gt;RON&lt;/span&gt; loader validates every dice string in the content files at startup, before the terminal goes into raw mode. A typo in the bestiary fails with a readable message on a normal screen rather than a panic behind an alternate screen&amp;nbsp;buffer.&lt;/p&gt;
&lt;h2 id="content-lives-in-ron-not-in-rust"&gt;Content lives in &lt;span class="caps"&gt;RON&lt;/span&gt;, not in&amp;nbsp;Rust&lt;/h2&gt;
&lt;p&gt;Monsters and items started as Rust source. They are now&amp;nbsp;data:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;(&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Giant Newt&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;glyph&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;#39;&lt;/span&gt;&lt;span class="na"&gt;n&lt;/span&gt;&lt;span class="o"&gt;&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;color&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;lightgreen&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;max_hp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;att&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;dv&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;8&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;damage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;1d3&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;xp&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;danger&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;weight&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;10&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;poison_turns&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;poison_damage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;Some&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;1d2&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;),&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;description&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;A slick, twitching amphibian the size of a house cat. Slow, stupid, and entirely willing to bite the hand that pokes it, and its bite festers.&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;),&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;RON&lt;/span&gt; is a good fit here because it reads almost like Rust&amp;nbsp;literals, &lt;code&gt;Option&lt;/code&gt; fields survive round trips honestly, and it supports comments, so the bestiary carries its own field&amp;nbsp;documentation. &lt;code&gt;serde&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s derive does all the parsing. Optional effect fields default&amp;nbsp;to &lt;code&gt;None&lt;/code&gt;, so a monster that just bites stays a short&amp;nbsp;block.&lt;/p&gt;
&lt;p&gt;The split I try to hold: &lt;strong&gt;data is stats, flavour and spawn weights; logic is behaviour&lt;/strong&gt;. Adding a creature is editing a text file. Adding a new &lt;em&gt;kind&lt;/em&gt; of behaviour still needs Rust, specifically a new component plus the system that acts on it, plus a new optional &lt;span class="caps"&gt;RON&lt;/span&gt; field to switch it&amp;nbsp;on.&lt;/p&gt;
&lt;h3 id="danger-instead-of-depth-cutoffs"&gt;Danger instead of depth&amp;nbsp;cutoffs&lt;/h3&gt;
&lt;p&gt;The first spawn table had hard depth bands, and it felt exactly as mechanical as it sounds. Now every monster has a&amp;nbsp;baseline &lt;code&gt;danger&lt;/code&gt;, meaning the depth it is at home at, and each &lt;em&gt;encounter&lt;/em&gt; rolls its own target around the floor&amp;#8217;s&amp;nbsp;depth:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="kd"&gt;let&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;mut&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;depth&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;roll_dice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;2&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rng&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;roll_dice&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;20&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;target&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;// the dungeon&amp;#39;s occasional cruel joke&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;2d3 - 4&lt;/code&gt; term is a wobble of plus or minus two around the floor&amp;#8217;s depth, and every monster within two points of the resulting target is eligible, weighted by its own commonness scaled by how close its danger sits to the target. Creatures fade in above their home depth and fade out below it instead of appearing and disappearing at a boundary, and one encounter in twenty gets an extra nudge of plus three toward the deeper end of the&amp;nbsp;bestiary.&lt;/p&gt;
&lt;p&gt;Note that the wobble and the eligibility spread stack, so out-of-depth encounters do not need the nudge at all: on floor 2 the target can already reach 4, which makes a danger 5 mimic eligible on its own. The nudge just widens the tail. A mimic on floor 2 is unfair, survivably so, and produces a story. That is the whole design goal of the game in one&amp;nbsp;function.&lt;/p&gt;
&lt;p&gt;Below the deepest creature in the bestiary, the window comes up empty, and the deepest monsters hold the fort rather than leaving an eerily peaceful abyss. A gap in the &lt;em&gt;middle&lt;/em&gt; of the table stays a quiet spawn, though, because falling back there would unleash the worst things far too&amp;nbsp;early.&lt;/p&gt;
&lt;h2 id="status-effects"&gt;Status&amp;nbsp;effects&lt;/h2&gt;
&lt;p&gt;A status is a&amp;nbsp;small &lt;code&gt;Copy&lt;/code&gt; component with a countdown, ticked once per&amp;nbsp;round. &lt;code&gt;Poisoned&lt;/code&gt; rolls fresh damage each round and can kill, and the tombstone will name the&amp;nbsp;poison. &lt;code&gt;Confused&lt;/code&gt; randomises movement instead of honouring&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;Two rules made confusion actually interesting rather than&amp;nbsp;annoying:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A stumble always costs the turn&lt;/strong&gt;, including one that lurches into a wall. Otherwise confusion is nearly free in a corridor, where three of four directions are wall, and you could simply re-press until the roll went your way, so it would never tick&amp;nbsp;down.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A stumble that lands on somebody is a swing at them&lt;/strong&gt;, whoever they are. That keeps confusion from being a &amp;#8220;monster switched off&amp;#8221; button, and it means a confused monster will brawl with its neighbours. Monster versus monster fights are only narrated when the player can see&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;The afflicted entity carries the status&amp;nbsp;(&lt;code&gt;Poisoned&lt;/code&gt;), and whatever inflicted it carries a matching ability component&amp;nbsp;(&lt;code&gt;InflictsPoison&lt;/code&gt;), which is the same split as an&amp;nbsp;item&amp;#8217;s &lt;code&gt;ProvidesHealing&lt;/code&gt; versus the hit points it actually restores. Both come from optional &lt;span class="caps"&gt;RON&lt;/span&gt; fields, so a venomous bite is&amp;nbsp;data.&lt;/p&gt;
&lt;h2 id="save-games-without-serialising-the-ecs"&gt;Save games without serialising the &lt;span class="caps"&gt;ECS&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Legion worlds are full of runtime-only things: entity ids, component storage layouts, archetypes. Serialising all of that is painful and mostly meaningless across&amp;nbsp;runs.&lt;/p&gt;
&lt;p&gt;But the game is data-driven, and that turns out to solve the problem. Every monster and item is spawned from a template. So the only facts a save needs are which template, where, and what has changed since spawning, plus the map, what has been explored, and the player&amp;#8217;s own&amp;nbsp;sheet.&lt;/p&gt;
&lt;p&gt;Loading is then just replaying spawns: look each template up, override the changed bits. The file is small, human-readable &lt;span class="caps"&gt;RON&lt;/span&gt;, immune to entity-id churn, and honestly fun to peek&amp;nbsp;at.&lt;/p&gt;
&lt;p&gt;The trade-off, stated plainly, is that anything not captured is forgotten. There is a test&amp;nbsp;called &lt;code&gt;saving_captures_what_loading_needs&lt;/code&gt; that acts as a tripwire. It is also the source of the most instructive bug in the project so far: the tripwire only catches what it &lt;em&gt;exercises&lt;/em&gt;. When I added status effects, save and load silently dropped them, and the test kept passing, because it never set up a poisoned hero in the first place. A test that asserts on the facts it already knows about is blind to a new kind of mutable state until somebody adds it to the&amp;nbsp;setup.&lt;/p&gt;
&lt;p&gt;Permadeath is enforced at the file level. The save is deleted the instant the player dies, before the tombstone is even drawn, so quitting the process at the death screen resurrects&amp;nbsp;nobody.&lt;/p&gt;
&lt;h2 id="testing-something-this-random"&gt;Testing something this&amp;nbsp;random&lt;/h2&gt;
&lt;p&gt;There are 62 unit tests,&amp;nbsp;and &lt;code&gt;cargo test&lt;/code&gt; runs them in well under a second because none of them need a&amp;nbsp;terminal.&lt;/p&gt;
&lt;p&gt;The categories that earned their&amp;nbsp;keep:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Pure rules.&lt;/strong&gt; Dice parsing, to-hit edges, attribute modifiers, healing that tops up but never overheals. Cheap and&amp;nbsp;exhaustive.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Invariants over random output.&lt;/strong&gt; Generators cannot be asserted against a fixed shape, so the tests seed the &lt;span class="caps"&gt;RNG&lt;/span&gt;, generate several levels, and check properties: every walkable tile is reachable, stairs exist, region 0 holds the player start, exactly one corridor enters the&amp;nbsp;vault.&lt;/p&gt;
&lt;p&gt;That style of test has a blind spot, and the cavern generator found it for me. Its retry loop generated up to ten candidates, kept the first one whose main cave covered a quarter of the map, and pruned each candidate&amp;#8217;s sealed pockets as part of judging it. But when all ten failed, the loop generated one more map and returned it without ever pruning it. The connectivity guarantee therefore had a hole in exactly the path that runs when generation has been unlucky. The existing test never noticed, because on a full-size 78 by 40 level the retry budget is never exhausted. Reproducing it took a cramped 20 by 10 map, where it shows up in roughly one run in two hundred. The fix restructured the loop so every candidate, including the last-resort one, is pruned exactly once before it can be returned, and the regression test pins a seed that used to come back with an unreachable pocket in&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Compile-time content validation.&lt;/strong&gt; Because prefabs are embedded&amp;nbsp;with &lt;code&gt;include_str!&lt;/code&gt;, the test that parses them &lt;em&gt;is&lt;/em&gt; the file validation. Break a drawing and the build&amp;#8217;s test run fails, long before a player sees&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Regressions of subtle rules.&lt;/strong&gt; A confused lurch into a wall still costs the turn. An unconfused wall bump does not. God mode declines death. Dying deletes the save&amp;nbsp;file.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Rendering logic without rendering.&lt;/strong&gt; The header line is a pure function that formats the &lt;span class="caps"&gt;HUD&lt;/span&gt; into a string, so a test can assert that status tags survive an 80-column terminal, where ratatui truncates an over-wide title on the right. Afflictions sit directly after the hit points precisely because that is where they will not be cut off exactly when the player most needs to read&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;One more small thing I enjoy: the cheat menu, opened by&amp;nbsp;typing &lt;code&gt;iddqd&lt;/code&gt; in the dungeon, exists only in debug builds&amp;nbsp;via &lt;code&gt;cfg!(debug_assertions)&lt;/code&gt;. Unlike a C preprocessor block, the disabled code still has to compile in release mode, so it cannot quietly rot the way an&amp;nbsp;untested &lt;code&gt;#ifdef&lt;/code&gt; branch can. Release builds ship honest, and cheats never persist into a save&amp;nbsp;file.&lt;/p&gt;
&lt;h2 id="what-is-next"&gt;What is&amp;nbsp;next&lt;/h2&gt;
&lt;p&gt;The backlog is long and deliberately additive: item identification with blessed, uncursed and cursed states, ranged combat, a hunger clock, shops, bosses, more map algorithms (&lt;span class="caps"&gt;BSP&lt;/span&gt;, drunkard&amp;#8217;s walk, mazes), and a builder chain so a single floor can mix strategies rather than picking one. There is a full setting document behind all this, six dungeon strata with their own ecologies and factions, and six possible endings, none of which is implemented&amp;nbsp;yet.&lt;/p&gt;
&lt;p&gt;Whether any of that ships publicly is an open question. The groundwork, though, is the part I actually wanted to write about: an &lt;span class="caps"&gt;ECS&lt;/span&gt; that makes features additive, two trait implementations that hand me field of view and pathfinding on my own data type, four map generators behind one interface, content that lives in text files, and a save format that sidesteps the hardest serialisation problem by storing facts instead of&amp;nbsp;objects.&lt;/p&gt;
&lt;p&gt;Rust turned out to be a very good fit for this. Not because of performance, which is irrelevant when the frame budget is &amp;#8220;whenever the player presses a key&amp;#8221;, but because an enum you must match exhaustively, a borrow checker that pushed me into a clean read, decide, write structure for the &lt;span class="caps"&gt;AI&lt;/span&gt;, and traits as the interface to other people&amp;#8217;s algorithms are all exactly the tools a slowly growing hobby codebase&amp;nbsp;needs.&lt;/p&gt;</content><category term="Linux"/><category term="rust"/><category term="roguelike"/><category term="gamedev"/><category term="ratatui"/><category term="ecs"/><category term="legion"/><category term="procedural-generation"/><category term="field-of-view"/><category term="pathfinding"/><category term="serde"/><category term="ron"/><category term="terminal"/></entry><entry><title>Chezmoi and Git: Dotfiles Without Pretending Every Machine Is the Same</title><link href="https://hofstede.it/chezmoi-and-git-dotfiles-without-pretending-every-machine-is-the-same/" rel="alternate"/><published>2026-08-12T00:00:00+02:00</published><updated>2026-08-12T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-08-12:/chezmoi-and-git-dotfiles-without-pretending-every-machine-is-the-same/</id><summary type="html">&lt;p&gt;A Git repository is a good place for configuration files, but it does not know where they belong, which permissions they need, or why a work laptop and a home server should differ. Chezmoi adds that missing layer. This is how I use both tools to manage fish, Git, Neovim, GnuPG, OpenSSH and Atuin, including the recovery command for the inevitable day I edit a live file&amp;nbsp;directly.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="A GenAI Slop Image of a home-directory source repository distributing related configurations to different computers. I'm bad at pictures" src="https://hofstede.it/images/2026-08-12-chezmoi-git-dotfiles.webp" title="One source state, rendered for several machines. AI Slop image. I'm bad at pictures"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-three-states-that-make-chezmoi-click"&gt;The Three States That Make Chezmoi&amp;nbsp;Click&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#starting-small"&gt;Starting&amp;nbsp;Small&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#my-normal-edit-review-and-commit-loop"&gt;My Normal Edit, Review and Commit&amp;nbsp;Loop&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#one-repository-different-machines"&gt;One Repository, Different&amp;nbsp;Machines&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#oopsie-i-edited-the-real-config"&gt;Oopsie: I Edited the Real&amp;nbsp;Config&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-second-machine-without-surprises"&gt;A Second Machine Without&amp;nbsp;Surprises&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#secrets-encryption-and-private-repositories"&gt;Secrets, Encryption and Private&amp;nbsp;Repositories&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#sharp-edges-worth-remembering"&gt;Sharp Edges Worth Remembering&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#do-not-make-config-exact-by-accident"&gt;Do Not Make ~/.config Exact by&amp;nbsp;Accident&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#source-diff-and-destination-diff-are-different"&gt;Source Diff and Destination Diff Are&amp;nbsp;Different&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#updating-is-not-publishing"&gt;Updating Is Not&amp;nbsp;Publishing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#test-the-rendered-file-not-only-the-template"&gt;Test the Rendered File, Not Only the&amp;nbsp;Template&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-command-card-i-actually-need"&gt;The Command Card I Actually&amp;nbsp;Need&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#further-reading"&gt;Further&amp;nbsp;Reading&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;My workstation configuration is not one file. It is fish functions and abbreviations, a Git configuration, Neovim Lua, GnuPG and &lt;span class="caps"&gt;SSH&lt;/span&gt; settings, and an Atuin configuration. It has grown over years, which is another way of saying that recreating it from memory would produce something incomplete and subtly&amp;nbsp;annoying.&lt;/p&gt;
&lt;p&gt;Putting all of it in Git solves the history and distribution problem. It does not solve the deployment problem. Git does not know&amp;nbsp;that &lt;code&gt;dot_gitconfig&lt;/code&gt; belongs&amp;nbsp;at &lt;code&gt;~/.gitconfig&lt;/code&gt;,&amp;nbsp;that &lt;code&gt;~/.ssh&lt;/code&gt; should not be world-readable, or that my work machine needs a different email address from a personal laptop. A repository full of symlinks can cover the first problem, and a sufficiently heroic shell script can cover the others, but at that point I have written a small and probably bad configuration&amp;nbsp;manager.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.chezmoi.io/"&gt;Chezmoi&lt;/a&gt; is the small configuration manager I use instead. Git remains the versioned source of truth. Chezmoi understands how that source should become real files in my home directory, including permissions, templates, encryption, scripts and machine-specific&amp;nbsp;differences.&lt;/p&gt;
&lt;p&gt;The distinction matters: &lt;strong&gt;Git stores the source; chezmoi turns it into the right configuration for this&amp;nbsp;machine.&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id="the-three-states-that-make-chezmoi-click"&gt;The Three States That Make Chezmoi&amp;nbsp;Click&lt;/h2&gt;
&lt;p&gt;Chezmoi terminology can sound more complicated than the job, but its &lt;a href="https://www.chezmoi.io/reference/concepts/"&gt;three-state model&lt;/a&gt; explains nearly every&amp;nbsp;command:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;State&lt;/th&gt;
&lt;th&gt;What it means&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Source state&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;The files&amp;nbsp;in &lt;code&gt;~/.local/share/chezmoi&lt;/code&gt;, normally a Git working tree. This is what I edit and commit.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Target state&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;What chezmoi calculates this machine &lt;em&gt;should&lt;/em&gt; have after templates, local data and attributes have been evaluated.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Destination state&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;What is actually in my home directory right now.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The target state is the useful extra layer. On one&amp;nbsp;machine, &lt;code&gt;dot_gitconfig.tmpl&lt;/code&gt; may render an address for work and a company signing key. On another, the same committed template can render my personal identity. Git sees one template. Each machine receives a&amp;nbsp;valid &lt;code&gt;~/.gitconfig&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The everyday commands follow directly from that&amp;nbsp;model:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;home directory  --chezmoi add--&amp;gt;  source state
source state    --chezmoi apply-&amp;gt; home directory
source state    --Git-----------&amp;gt; other machines
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;chezmoi diff&lt;/code&gt; compares the calculated target with the current destination. In practical terms, it answers the question I care about before applying anything: &lt;strong&gt;what would this change in my home&amp;nbsp;directory?&lt;/strong&gt;&lt;/p&gt;
&lt;h2 id="starting-small"&gt;Starting&amp;nbsp;Small&lt;/h2&gt;
&lt;p&gt;Chezmoi is available through many operating-system package managers and as a single binary; the &lt;a href="https://www.chezmoi.io/install/"&gt;installation page&lt;/a&gt; has the current choices. FreeBSD is covered too, either&amp;nbsp;as &lt;code&gt;sysutils/chezmoi&lt;/code&gt; in the ports tree or&amp;nbsp;with &lt;code&gt;pkg install chezmoi&lt;/code&gt;, which matters to me because the machines I manage this way are not all Linux. Once installed, this creates the source directory and initializes a Git repository inside&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;init
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I would not begin by feeding it the whole&amp;nbsp;of &lt;code&gt;~/.config&lt;/code&gt;. That directory contains caches, databases, tokens, device identifiers and application state alongside actual configuration. Start with files whose purpose is&amp;nbsp;clear:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish
chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;~/.config/nvim
chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;~/.gnupg/gpg.conf
chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;~/.gnupg/gpg-agent.conf
chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;~/.ssh/config
chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;~/.config/atuin/config.toml
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Adding a directory is recursive by default, so the Neovim command above brings in everything currently&amp;nbsp;below &lt;code&gt;~/.config/nvim&lt;/code&gt;. That deserves a review. Configuration modules and a plugin lock file belong in Git; downloaded plugins, swap files and generated state do&amp;nbsp;not.&lt;/p&gt;
&lt;p&gt;The same rule applies to the other&amp;nbsp;tools:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;For &lt;strong&gt;fish&lt;/strong&gt;, I manage configuration, functions and completions that I&amp;nbsp;wrote. &lt;code&gt;fish_variables&lt;/code&gt; holds universal variables that fish rewrites on its own, so I keep it out. Several of my fish functions are adaptations of the &lt;a href="https://hofstede.it/shell-tricks-that-actually-make-life-easier-and-save-your-sanity/"&gt;shell tricks I collected earlier&lt;/a&gt;.&lt;/li&gt;
&lt;li&gt;For &lt;strong&gt;Git&lt;/strong&gt;, I&amp;nbsp;manage &lt;code&gt;.gitconfig&lt;/code&gt; and selected include files, never a credential store containing&amp;nbsp;tokens.&lt;/li&gt;
&lt;li&gt;For &lt;strong&gt;Neovim&lt;/strong&gt;, I manage the Lua/Vim configuration and lock file, not plugins or caches downloaded from elsewhere. The &lt;a href="https://hofstede.it/neovim-crash-course-for-sysadmins-the-20-that-solve-80-of-the-pain/"&gt;crash course configuration&lt;/a&gt; is exactly the kind of thing worth carrying between&amp;nbsp;machines.&lt;/li&gt;
&lt;li&gt;For &lt;strong&gt;GnuPG&lt;/strong&gt;, I manage files such&amp;nbsp;as &lt;code&gt;gpg.conf&lt;/code&gt;, &lt;code&gt;gpg-agent.conf&lt;/code&gt; and &lt;code&gt;dirmngr.conf&lt;/code&gt;, not the private keyring, trust database, sockets or random&amp;nbsp;seed.&lt;/li&gt;
&lt;li&gt;For &lt;strong&gt;OpenSSH&lt;/strong&gt;, I manage client configuration and perhaps public material. Private keys, host keys, control sockets and the constantly&amp;nbsp;changing &lt;code&gt;known_hosts&lt;/code&gt; file are separate&amp;nbsp;concerns.&lt;/li&gt;
&lt;li&gt;For &lt;strong&gt;Atuin&lt;/strong&gt;, I&amp;nbsp;manage &lt;code&gt;config.toml&lt;/code&gt;, not the history database, session material or encryption&amp;nbsp;key.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This boundary is more important than whether the dotfiles repository is public or private. A private Git repository reduces exposure; it does not turn committed secrets into a good&amp;nbsp;idea.&lt;/p&gt;
&lt;p&gt;After adding the first files, I inspect what chezmoi&amp;nbsp;created:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;cd&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;--short
find&lt;span class="w"&gt; &lt;/span&gt;.&lt;span class="w"&gt; &lt;/span&gt;-path&lt;span class="w"&gt; &lt;/span&gt;./.git&lt;span class="w"&gt; &lt;/span&gt;-prune&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;-type&lt;span class="w"&gt; &lt;/span&gt;f&lt;span class="w"&gt; &lt;/span&gt;-print&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Names in the source state encode&amp;nbsp;attributes. &lt;code&gt;dot_gitconfig&lt;/code&gt; becomes &lt;code&gt;~/.gitconfig&lt;/code&gt;; &lt;code&gt;private_&lt;/code&gt; removes group and world&amp;nbsp;permissions; &lt;code&gt;executable_&lt;/code&gt; sets executable&amp;nbsp;bits; &lt;code&gt;.tmpl&lt;/code&gt; marks a template. The complete list is in the &lt;a href="https://www.chezmoi.io/reference/source-state-attributes/"&gt;source-state attributes reference&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;I let chezmoi change those attributes instead of hand-renaming&amp;nbsp;files:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;chattr&lt;span class="w"&gt; &lt;/span&gt;+private&lt;span class="w"&gt; &lt;/span&gt;~/.ssh
chezmoi&lt;span class="w"&gt; &lt;/span&gt;chattr&lt;span class="w"&gt; &lt;/span&gt;+private&lt;span class="w"&gt; &lt;/span&gt;~/.ssh/config
chezmoi&lt;span class="w"&gt; &lt;/span&gt;chattr&lt;span class="w"&gt; &lt;/span&gt;+private&lt;span class="w"&gt; &lt;/span&gt;~/.gnupg
chezmoi&lt;span class="w"&gt; &lt;/span&gt;chattr&lt;span class="w"&gt; &lt;/span&gt;+private&lt;span class="w"&gt; &lt;/span&gt;~/.gnupg/gpg.conf
chezmoi&lt;span class="w"&gt; &lt;/span&gt;chattr&lt;span class="w"&gt; &lt;/span&gt;+private&lt;span class="w"&gt; &lt;/span&gt;~/.gnupg/gpg-agent.conf
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The corresponding source paths will look roughly&amp;nbsp;like &lt;code&gt;private_dot_ssh/private_config&lt;/code&gt; and &lt;code&gt;private_dot_gnupg/private_gpg.conf&lt;/code&gt;. Git does not preserve all Unix permission bits; chezmoi&amp;#8217;s attributes make the intended permissions portable and&amp;nbsp;explicit.&lt;/p&gt;
&lt;h2 id="my-normal-edit-review-and-commit-loop"&gt;My Normal Edit, Review and Commit&amp;nbsp;Loop&lt;/h2&gt;
&lt;p&gt;For a managed file, this is the compact&amp;nbsp;path:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;edit&lt;span class="w"&gt; &lt;/span&gt;--apply&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;chezmoi edit&lt;/code&gt; opens the source&amp;nbsp;version. &lt;code&gt;--apply&lt;/code&gt; updates the real file after the editor exits. I use the longer form when the change is larger or affects more than one&amp;nbsp;file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;edit&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
chezmoi&lt;span class="w"&gt; &lt;/span&gt;edit&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish

chezmoi&lt;span class="w"&gt; &lt;/span&gt;status
chezmoi&lt;span class="w"&gt; &lt;/span&gt;diff
chezmoi&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;--dry-run&lt;span class="w"&gt; &lt;/span&gt;--verbose
chezmoi&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;--verbose
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;status&lt;/code&gt; is the short&amp;nbsp;summary. &lt;code&gt;diff&lt;/code&gt; shows content changes. The verbose dry run is the last sanity check for permissions, directories, scripts and removals. Applying only one file is also&amp;nbsp;possible:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then I use ordinary Git in the source directory. I prefer this explicit review over automatic commits because a dotfiles repository is executable infrastructure for my user&amp;nbsp;account:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;cd&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;--short
git&lt;span class="w"&gt; &lt;/span&gt;diff
git&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;-A
git&lt;span class="w"&gt; &lt;/span&gt;commit&lt;span class="w"&gt; &lt;/span&gt;-S&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;update fish and neovim configuration&amp;quot;&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;push
&lt;span class="nb"&gt;exit&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Chezmoi can auto-commit and auto-push through&amp;nbsp;the &lt;code&gt;git.autoCommit&lt;/code&gt; and &lt;code&gt;git.autoPush&lt;/code&gt; settings, both of which are off by default. The manual pause catches machine-local paths, accidental secrets and generated junk before they become every machine&amp;#8217;s&amp;nbsp;problem.&lt;/p&gt;
&lt;h2 id="one-repository-different-machines"&gt;One Repository, Different&amp;nbsp;Machines&lt;/h2&gt;
&lt;p&gt;The local chezmoi configuration normally lives&amp;nbsp;at &lt;code&gt;~/.config/chezmoi/chezmoi.toml&lt;/code&gt;. It is specific to the current machine and can provide values to committed templates. For&amp;nbsp;example:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[data]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;email&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;me@example.net&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;gitSigningKey&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0123456789ABCDEF&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;work&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kc"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I can turn an already&amp;nbsp;managed &lt;code&gt;.gitconfig&lt;/code&gt; into a template&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;chattr&lt;span class="w"&gt; &lt;/span&gt;+template&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
chezmoi&lt;span class="w"&gt; &lt;/span&gt;edit&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The source file is&amp;nbsp;now &lt;code&gt;dot_gitconfig.tmpl&lt;/code&gt;. A small example looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[user]
    name = Christian Hofstede
    email = {{ .email | quote }}
    signingkey = {{ .gitSigningKey | quote }}

[commit]
    gpgsign = true

{{- if .work }}
[include]
    path = ~/.gitconfig-work
{{- end }}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The signing key is a good example of why the template layer earns its place: the key &lt;span class="caps"&gt;ID&lt;/span&gt; differs per identity, while the &lt;a href="https://hofstede.it/card-wars-hiding-smartcard-readers-from-eager-rust-agents-with-ld_preload/"&gt;smartcard that actually holds it&lt;/a&gt; is a machine-local concern that never belongs in the repository at&amp;nbsp;all.&lt;/p&gt;
&lt;p&gt;The literal Git configuration in my home directory contains no template syntax. Chezmoi renders it using the local data. Before applying it, I can inspect exactly what the template&amp;nbsp;produces:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;data
chezmoi&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
chezmoi&lt;span class="w"&gt; &lt;/span&gt;diff&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Chezmoi also provides built-in facts&amp;nbsp;including &lt;code&gt;.chezmoi.os&lt;/code&gt;, &lt;code&gt;.chezmoi.arch&lt;/code&gt; and &lt;code&gt;.chezmoi.hostname&lt;/code&gt;. A fish template can contain a small operating-system&amp;nbsp;difference:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;set -gx EDITOR nvim

{{- if eq .chezmoi.os &amp;quot;darwin&amp;quot; }}
fish_add_path /opt/homebrew/bin
{{- else if eq .chezmoi.os &amp;quot;linux&amp;quot; }}
fish_add_path ~/.local/bin
{{- end }}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I prefer semantic local data such&amp;nbsp;as &lt;code&gt;work = true&lt;/code&gt; over scattering hostnames throughout templates. Hostname checks are useful for a genuinely unique machine; roles survive the day a laptop is replaced and receives a new&amp;nbsp;name.&lt;/p&gt;
&lt;p&gt;For entire files or directories that should exist only on some&amp;nbsp;machines, &lt;code&gt;.chezmoiignore&lt;/code&gt; is cleaner than wrapping all their contents in conditionals. It is itself a template. This example deploys a work-only Git include only&amp;nbsp;when &lt;code&gt;work&lt;/code&gt; is&amp;nbsp;true:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;{{- if not .work }}
.gitconfig-work
{{- end }}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The official guide covers these patterns in more depth under &lt;a href="https://www.chezmoi.io/user-guide/manage-machine-to-machine-differences/"&gt;machine-to-machine differences&lt;/a&gt;.&amp;nbsp;A &lt;code&gt;.chezmoi.toml.tmpl&lt;/code&gt; in the repository can even prompt for machine-specific values&amp;nbsp;during &lt;code&gt;chezmoi init&lt;/code&gt;, which turns bringing up a new machine into a repeatable questionnaire rather than an editing scavenger&amp;nbsp;hunt.&lt;/p&gt;
&lt;h2 id="oopsie-i-edited-the-real-config"&gt;Oopsie: I Edited the Real&amp;nbsp;Config&lt;/h2&gt;
&lt;p&gt;Sooner or later I&amp;nbsp;open &lt;code&gt;~/.config/fish/config.fish&lt;/code&gt; directly, make the perfect fix, and only remember chezmoi afterwards. The change is not lost. The first step is to&amp;nbsp;look:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;diff&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There is a command designed specifically for this&amp;nbsp;situation:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;re-add&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish
chezmoi&lt;span class="w"&gt; &lt;/span&gt;source-path&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;source-path&lt;/code&gt; tells me exactly which file in the Git working tree was&amp;nbsp;updated.&lt;/p&gt;
&lt;p&gt;&lt;a href="https://www.chezmoi.io/reference/commands/re-add/"&gt;&lt;code&gt;chezmoi re-add&lt;/code&gt;&lt;/a&gt; copies the destination back into the source state while preserving&amp;nbsp;the &lt;code&gt;encrypted_&lt;/code&gt; attribute, and it deliberately refuses to overwrite templates. With no path it re-adds every modified managed file, which is convenient but broader than I normally&amp;nbsp;want:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;re-add
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A&amp;nbsp;plain &lt;code&gt;chezmoi add&lt;/code&gt; on an already-managed file does something similar, replacing its source state with the current destination, but it is the blunter instrument: it has no special handling for encrypted files, and&amp;nbsp;with &lt;code&gt;--force&lt;/code&gt; it will happily flatten a template. I reach&amp;nbsp;for &lt;code&gt;re-add&lt;/code&gt; and&amp;nbsp;keep &lt;code&gt;add&lt;/code&gt; for genuinely new&amp;nbsp;files.&lt;/p&gt;
&lt;p&gt;Either way, I then review the &lt;em&gt;Git&lt;/em&gt; diff,&amp;nbsp;because &lt;code&gt;chezmoi diff&lt;/code&gt; should now be empty for that&amp;nbsp;file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;cd&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;diff
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Templates need more judgment, which is exactly why both commands refuse to touch them. The&amp;nbsp;live &lt;code&gt;.gitconfig&lt;/code&gt; is the &lt;em&gt;rendered result&lt;/em&gt;, while the source contains conditionals and variables. Flattening the result back into the source would throw those away, and no amount&amp;nbsp;of &lt;code&gt;--force&lt;/code&gt; makes that a&amp;nbsp;recovery.&lt;/p&gt;
&lt;p&gt;Instead I use the destination diff as a guide and reproduce the intended change in the&amp;nbsp;template:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;diff&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
chezmoi&lt;span class="w"&gt; &lt;/span&gt;edit&lt;span class="w"&gt; &lt;/span&gt;--apply&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For a complicated conflict, this opens the configured three-way merge&amp;nbsp;tool, &lt;code&gt;vimdiff&lt;/code&gt; by&amp;nbsp;default:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;merge&lt;span class="w"&gt; &lt;/span&gt;~/.gitconfig
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The three inputs are the current destination, the source and the rendered target. That is precisely the context needed to preserve both the local fix and the template&amp;nbsp;logic.&lt;/p&gt;
&lt;p&gt;If the direct edit was a mistake and I want to discard it, the flow goes the other&amp;nbsp;direction:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;diff&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish
chezmoi&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Chezmoi notices when a destination has changed since it last wrote it and prompts before overwriting. I still read the diff first. Configuration managers are wonderfully consistent, including when they consistently apply the wrong&amp;nbsp;thing.&lt;/p&gt;
&lt;h2 id="a-second-machine-without-surprises"&gt;A Second Machine Without&amp;nbsp;Surprises&lt;/h2&gt;
&lt;p&gt;Once the source repository has a remote, a second machine can clone it&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;init&lt;span class="w"&gt; &lt;/span&gt;ssh://git@example.net/user/dotfiles.git
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I do not&amp;nbsp;add &lt;code&gt;--apply&lt;/code&gt; the first time. First I inspect the local data and rendered&amp;nbsp;result:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;data
chezmoi&lt;span class="w"&gt; &lt;/span&gt;doctor
chezmoi&lt;span class="w"&gt; &lt;/span&gt;diff
chezmoi&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;--dry-run&lt;span class="w"&gt; &lt;/span&gt;--verbose
chezmoi&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;--verbose
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After I trust the repository and its initialization&amp;nbsp;path, &lt;code&gt;chezmoi init --apply ...&lt;/code&gt; is a useful shortcut for future machines. It should still be treated like running configuration code: templates can retrieve secrets and chezmoi source repositories can contain&amp;nbsp;scripts.&lt;/p&gt;
&lt;p&gt;To bring an established machine up to date, the convenient command&amp;nbsp;is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;update&lt;span class="w"&gt; &lt;/span&gt;--verbose
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;By default, that pulls the Git repository with rebase and autostash, then applies it. When I want the pull and review as two separate steps, I use the safer, more talkative form from the &lt;a href="https://www.chezmoi.io/user-guide/daily-operations/"&gt;daily operations guide&lt;/a&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;git&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;--&lt;span class="w"&gt; &lt;/span&gt;--autostash&lt;span class="w"&gt; &lt;/span&gt;--rebase
chezmoi&lt;span class="w"&gt; &lt;/span&gt;diff
chezmoi&lt;span class="w"&gt; &lt;/span&gt;apply&lt;span class="w"&gt; &lt;/span&gt;--verbose
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That review matters most when I have not used a machine for a while. Six months of accumulated shell, editor and &lt;span class="caps"&gt;SSH&lt;/span&gt; changes deserve more than an optimistic Enter&amp;nbsp;key.&lt;/p&gt;
&lt;h2 id="secrets-encryption-and-private-repositories"&gt;Secrets, Encryption and Private&amp;nbsp;Repositories&lt;/h2&gt;
&lt;p&gt;Chezmoi can encrypt files with age, rage or &lt;span class="caps"&gt;GPG&lt;/span&gt;, and its templates can retrieve values from a long list of password managers. Those are useful capabilities, but I keep the default rule simple: if a file is really a database, keyring or machine identity rather than configuration, I do not add it merely because encryption is&amp;nbsp;available.&lt;/p&gt;
&lt;p&gt;When a configuration file genuinely must contain a secret, I prefer a password-manager template so the repository stores a reference rather than ciphertext that lives forever in Git history. Encrypted source files are appropriate when the whole file must travel and the decryption key lifecycle is already solved. The &lt;a href="https://www.chezmoi.io/user-guide/password-managers/"&gt;password-manager&lt;/a&gt; and &lt;a href="https://www.chezmoi.io/user-guide/encryption/"&gt;encryption&lt;/a&gt; guides document both&amp;nbsp;models.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;private_&lt;/code&gt; is not encryption. It controls permissions on the destination.&amp;nbsp;A &lt;code&gt;private_dot_ssh&lt;/code&gt; directory can correctly become&amp;nbsp;mode &lt;code&gt;0700&lt;/code&gt; while its contents remain plain text in Git. That is perfect for &lt;span class="caps"&gt;SSH&lt;/span&gt; client configuration and completely insufficient for a private key in a public&amp;nbsp;repository.&lt;/p&gt;
&lt;p&gt;Before every first commit of a newly added directory, I read the diff as though I were reviewing someone else&amp;#8217;s pull request. Filenames alone often reveal&amp;nbsp;trouble: &lt;code&gt;credentials&lt;/code&gt;, &lt;code&gt;token&lt;/code&gt;, &lt;code&gt;history.db&lt;/code&gt;, &lt;code&gt;key&lt;/code&gt;, &lt;code&gt;known_hosts&lt;/code&gt;, &lt;code&gt;trustdb.gpg&lt;/code&gt;, &lt;code&gt;random_seed&lt;/code&gt;, &lt;code&gt;socket&lt;/code&gt;, &lt;code&gt;cache&lt;/code&gt;. If I am not sure what an application file contains, it does not get committed&amp;nbsp;yet.&lt;/p&gt;
&lt;h2 id="sharp-edges-worth-remembering"&gt;Sharp Edges Worth&amp;nbsp;Remembering&lt;/h2&gt;
&lt;h3 id="do-not-make-config-exact-by-accident"&gt;Do Not&amp;nbsp;Make &lt;code&gt;~/.config&lt;/code&gt; Exact by&amp;nbsp;Accident&lt;/h3&gt;
&lt;p&gt;An &lt;code&gt;exact_&lt;/code&gt; directory tells chezmoi to remove destination entries that are not represented in the source. That can be valuable for a small directory that chezmoi owns completely. It is dangerous near broad shared&amp;nbsp;parents.&lt;/p&gt;
&lt;p&gt;In&amp;nbsp;particular, &lt;code&gt;chezmoi add --exact --recursive ~/.config/nvim&lt;/code&gt; sets the attribute on every directory it creates along the way.&amp;nbsp;If &lt;code&gt;~/.config&lt;/code&gt; was not represented in the source state before, it arrives&amp;nbsp;as &lt;code&gt;exact_dot_config&lt;/code&gt;, and the next apply considers every unrelated application directory&amp;nbsp;beside &lt;code&gt;nvim&lt;/code&gt; to be surplus. The &lt;a href="https://www.chezmoi.io/reference/commands/add/"&gt;&lt;code&gt;add&lt;/code&gt; reference&lt;/a&gt; calls this &amp;#8220;predictable but surprising&amp;#8221; and documents the fix: give the parent an ordinary source entry first, so the recursive add never has to invent&amp;nbsp;one.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;touch&lt;span class="w"&gt; &lt;/span&gt;~/.config/.keep
chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;~/.config/.keep
chezmoi&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;--recursive&lt;span class="w"&gt; &lt;/span&gt;--exact&lt;span class="w"&gt; &lt;/span&gt;~/.config/nvim
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now &lt;code&gt;dot_config&lt;/code&gt; exists without the exact attribute, and&amp;nbsp;only &lt;code&gt;dot_config/exact_nvim&lt;/code&gt; and its created descendants are strict. My simpler rule still applies:&amp;nbsp;avoid &lt;code&gt;--exact&lt;/code&gt; until I can explain the generated source tree and have inspected a verbose dry&amp;nbsp;run.&lt;/p&gt;
&lt;h3 id="source-diff-and-destination-diff-are-different"&gt;Source Diff and Destination Diff Are&amp;nbsp;Different&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;git diff&lt;/code&gt; shows uncommitted changes in the source&amp;nbsp;repository. &lt;code&gt;chezmoi diff&lt;/code&gt; shows what applying the calculated target would change in the home directory. Before applying a normal source-first edit I expect both to show something. After re-adding an accidental destination edit, I&amp;nbsp;expect &lt;code&gt;git diff&lt;/code&gt; to show it&amp;nbsp;and &lt;code&gt;chezmoi diff&lt;/code&gt; to be&amp;nbsp;empty.&lt;/p&gt;
&lt;p&gt;Remembering that distinction removes most of the apparent&amp;nbsp;magic.&lt;/p&gt;
&lt;h3 id="updating-is-not-publishing"&gt;Updating Is Not&amp;nbsp;Publishing&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;chezmoi apply&lt;/code&gt; changes the current machine. It does not commit or&amp;nbsp;push. &lt;code&gt;git push&lt;/code&gt; publishes the source change but does not update the other&amp;nbsp;machines. &lt;code&gt;chezmoi update&lt;/code&gt; on those machines pulls and applies it. The separation is a feature: every boundary has a diff and a place to&amp;nbsp;stop.&lt;/p&gt;
&lt;h3 id="test-the-rendered-file-not-only-the-template"&gt;Test the Rendered File, Not Only the&amp;nbsp;Template&lt;/h3&gt;
&lt;p&gt;A template can be syntactically valid and still generate invalid application&amp;nbsp;configuration. &lt;code&gt;chezmoi cat&lt;/code&gt; shows the rendered file without writing it. After applying, use the application&amp;#8217;s own checker when it has&amp;nbsp;one:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi&lt;span class="w"&gt; &lt;/span&gt;cat&lt;span class="w"&gt; &lt;/span&gt;~/.ssh/config
ssh&lt;span class="w"&gt; &lt;/span&gt;-G&lt;span class="w"&gt; &lt;/span&gt;example-host&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;/dev/null

git&lt;span class="w"&gt; &lt;/span&gt;config&lt;span class="w"&gt; &lt;/span&gt;--global&lt;span class="w"&gt; &lt;/span&gt;--list&lt;span class="w"&gt; &lt;/span&gt;--show-origin
fish&lt;span class="w"&gt; &lt;/span&gt;--no-execute&lt;span class="w"&gt; &lt;/span&gt;~/.config/fish/config.fish
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The exact checks depend on the application, but the principle is the same as any configuration deployment: rendering successfully is not the same as working&amp;nbsp;correctly.&lt;/p&gt;
&lt;h2 id="the-command-card-i-actually-need"&gt;The Command Card I Actually&amp;nbsp;Need&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chezmoi add FILE             start managing a new file
chezmoi re-add FILE          capture a modified managed file; keep encryption
chezmoi edit FILE            edit the source version
chezmoi edit --apply FILE    edit the source and apply it immediately
chezmoi status               show a short destination-change summary
chezmoi diff [FILE]          show what apply would change
chezmoi cat FILE             print the rendered target without applying
chezmoi apply [FILE]         make the destination match the target
chezmoi merge FILE           reconcile destination, source and target
chezmoi source-path FILE     print the corresponding source path
chezmoi data                 show data available to templates
chezmoi doctor               check the setup for common problems
chezmoi cd                   enter the Git working tree
chezmoi update               pull and apply on an established machine
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is most of my daily interaction with chezmoi. The tool has scripts, externals, password-manager integrations and enough template functions to build something extremely elaborate. I try not to. A configuration repository should reduce surprise, not become the most surprising program in my home&amp;nbsp;directory.&lt;/p&gt;
&lt;p&gt;The payoff is pleasantly boring. A new machine gets the same fish functions, Git defaults, Neovim setup, &lt;span class="caps"&gt;SSH&lt;/span&gt; aliases, GnuPG preferences and Atuin behavior as the others. The differences are explicit data or reviewed template branches, not forgotten manual edits. Git tells me how the configuration changed over time; chezmoi tells me what that history means on the machine in front of&amp;nbsp;me.&lt;/p&gt;
&lt;p&gt;And when I inevitably edit the live file&amp;nbsp;anyway, &lt;code&gt;chezmoi re-add&lt;/code&gt; is there to forgive&amp;nbsp;me.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.chezmoi.io/user-guide/setup/"&gt;chezmoi user&amp;nbsp;guide&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.chezmoi.io/reference/concepts/"&gt;chezmoi concepts: the three&amp;nbsp;states&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.chezmoi.io/reference/source-state-attributes/"&gt;chezmoi source state attributes&amp;nbsp;reference&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.chezmoi.io/reference/commands/"&gt;chezmoi command&amp;nbsp;reference&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.chezmoi.io/user-guide/manage-machine-to-machine-differences/"&gt;Managing machine-to-machine&amp;nbsp;differences&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.chezmoi.io/user-guide/daily-operations/"&gt;Daily&amp;nbsp;operations&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.chezmoi.io/user-guide/password-managers/"&gt;Password manager&amp;nbsp;integration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.chezmoi.io/user-guide/encryption/"&gt;Encryption with age, rage and &lt;span class="caps"&gt;GPG&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freshports.org/sysutils/chezmoi/"&gt;FreshPorts:&amp;nbsp;sysutils/chezmoi&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/neovim-crash-course-for-sysadmins-the-20-that-solve-80-of-the-pain/"&gt;Neovim Crash Course for&amp;nbsp;Sysadmins&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/shell-tricks-that-actually-make-life-easier-and-save-your-sanity/"&gt;Shell Tricks That Actually Make Life&amp;nbsp;Easier&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="chezmoi"/><category term="dotfiles"/><category term="git"/><category term="fish"/><category term="neovim"/><category term="gnupg"/><category term="openssh"/><category term="atuin"/><category term="configuration-management"/><category term="freebsd"/></entry><entry><title>Sane Development Environments for Ansible Automation Code</title><link href="https://hofstede.it/sane-development-environments-for-ansible-automation-code/" rel="alternate"/><published>2026-08-03T00:00:00+02:00</published><updated>2026-08-03T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-08-03:/sane-development-environments-for-ansible-automation-code/</id><summary type="html">&lt;p&gt;Ansible content deserves something better than a laptop full of loosely related pip packages. My preferred setup puts the editor on &lt;span class="caps"&gt;RHEL&lt;/span&gt;, the Red Hat-packaged Ansible toolchain in a versioned Dev Container, and Podman underneath both development and Execution Environment workflows. The result is reproducible without pretending that upstream community bits and supported downstream product content are the same&amp;nbsp;thing.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="VS Code running an Ansible workspace in the Red Hat Ansible Development Tools container" src="https://hofstede.it/images/2026-08-03-sane-ansible-development-environments.webp" title="VS Code running an Ansible workspace in a Dev Container"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-stack-i-actually-use"&gt;The Stack I Actually&amp;nbsp;Use&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#dev-container-and-execution-environment-are-different-jobs"&gt;Dev Container and Execution Environment Are Different&amp;nbsp;Jobs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#host-preparation"&gt;Host&amp;nbsp;Preparation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-devcontainerjson"&gt;The devcontainer.json&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#those-privileges-are-not-decorative"&gt;Those Privileges Are Not&amp;nbsp;Decorative&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-is-actually-in-the-image"&gt;What Is Actually in the&amp;nbsp;Image?&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-this-is-better-than-a-carefully-tended-venv"&gt;Why This Is Better Than a Carefully Tended&amp;nbsp;Venv&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#upstream-alternative-community-ansible-dev-tools"&gt;Upstream Alternative: Community Ansible Dev&amp;nbsp;Tools&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-editor-is-a-separate-choice"&gt;The Editor Is a Separate&amp;nbsp;Choice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-practical-team-policy"&gt;A Practical Team&amp;nbsp;Policy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#further-reading"&gt;Further&amp;nbsp;Reading&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;Ansible has a deceptively low barrier to entry. Install a Python package, write a &lt;span class="caps"&gt;YAML&lt;/span&gt; file, and you have automation. Six months later the same workstation has three Python virtual environments, two copies&amp;nbsp;of &lt;code&gt;ansible-core&lt;/code&gt;, collections in both the project&amp;nbsp;and &lt;code&gt;~/.ansible&lt;/code&gt;, and&amp;nbsp;an &lt;code&gt;ansible-lint&lt;/code&gt; version that disagrees with &lt;span class="caps"&gt;CI&lt;/span&gt;. It still works, but nobody can say exactly &lt;em&gt;why&lt;/em&gt; it&amp;nbsp;works.&lt;/p&gt;
&lt;p&gt;That is not a sane development&amp;nbsp;environment.&lt;/p&gt;
&lt;p&gt;For work intended for Red Hat Ansible Automation Platform, my preferred answer is a containerized workspace built from the same downstream, Red Hat-packaged toolchain that the organization has chosen to consume. The editor remains a normal desktop application. The Ansible tools, Python environment and supporting utilities live in a versioned &lt;span class="caps"&gt;OCI&lt;/span&gt; image. &lt;span class="caps"&gt;VS&lt;/span&gt; Code attaches the project to that container and Podman handles the&amp;nbsp;lifecycle.&lt;/p&gt;
&lt;p&gt;This is an opinionated setup, and I need to establish two boundaries before getting into&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;First, I work for Red Hat, but this article is my personal workflow and opinion. Consult the &lt;a href="https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/"&gt;Red Hat Ansible Automation Platform documentation&lt;/a&gt; and your subscription terms for the authoritative product&amp;nbsp;position.&lt;/p&gt;
&lt;p&gt;Second, &lt;strong&gt;upstream and downstream are not synonyms&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Upstream&lt;/strong&gt; means community projects such&amp;nbsp;as &lt;code&gt;ansible-core&lt;/code&gt;, Ansible Development Tools and their releases on PyPI or &lt;span class="caps"&gt;GHCR&lt;/span&gt;. They are excellent ways to develop, learn, contribute and follow current project&amp;nbsp;work.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Downstream&lt;/strong&gt; means the tested and packaged content delivered as part of Red Hat Ansible Automation Platform (&lt;span class="caps"&gt;AAP&lt;/span&gt;), with its product lifecycle, errata and support&amp;nbsp;boundaries.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Enterprise customers who require Red Hat support should stay within the versions, repositories, images and documented combinations delivered for their &lt;span class="caps"&gt;AAP&lt;/span&gt; subscription. Replacing a downstream &lt;span class="caps"&gt;RPM&lt;/span&gt; or container with a newer upstream PyPI or &lt;span class="caps"&gt;GHCR&lt;/span&gt; build may be technically valid, but it changes the support conversation. “It is the same project” is not the same as “it is the supported product&amp;nbsp;artifact.”&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Side note:&lt;/strong&gt; If you are new to Ansible automation, &lt;a href="https://www.redhat.com/en/services/training/au294-red-hat-linux-automation-with-ansible"&gt;Red Hat &lt;span class="caps"&gt;AU294&lt;/span&gt;: Red Hat Enterprise Linux Automation with Ansible&lt;/a&gt; is an excellent course for getting started. I can highly recommend&amp;nbsp;it.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;h2 id="the-stack-i-actually-use"&gt;The Stack I Actually&amp;nbsp;Use&lt;/h2&gt;
&lt;p&gt;As of 3 August 2026, my workstation stack&amp;nbsp;is:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Host &lt;span class="caps"&gt;OS&lt;/span&gt;:&lt;/strong&gt; Red Hat Enterprise Linux 10.2 (Coughlan), using the default&amp;nbsp;desktop.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Editor:&lt;/strong&gt; Microsoft Visual Studio Code, installed from Microsoft&amp;#8217;s &lt;span class="caps"&gt;RPM&lt;/span&gt;&amp;nbsp;repository.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Editor integration:&lt;/strong&gt; Dev Containers and Python from Microsoft, plus Ansible from Red Hat. Python and Ansible are activated in the remote container workspace; the Ansible extension is also available locally to scaffold the Dev Container&amp;nbsp;configuration.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Container runtime:&lt;/strong&gt; Podman from &lt;span class="caps"&gt;RHEL&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Development&amp;nbsp;image:&lt;/strong&gt; &lt;code&gt;registry.redhat.io/ansible-automation-platform-27/ansible-dev-tools-rhel9:26.7.1-1783617973&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The last item is the important one. The image contains the Ansible content development toolchain as Red Hat-packaged RPMs,&amp;nbsp;including &lt;code&gt;ansible-core&lt;/code&gt;, &lt;code&gt;ansible-builder&lt;/code&gt;, &lt;code&gt;ansible-creator&lt;/code&gt;, &lt;code&gt;ansible-lint&lt;/code&gt;, &lt;code&gt;ansible-navigator&lt;/code&gt;, &lt;code&gt;ansible-runner&lt;/code&gt; and &lt;code&gt;ansible-sign&lt;/code&gt;. Pulling it requires authentication&amp;nbsp;to &lt;code&gt;registry.redhat.io&lt;/code&gt; and appropriate &lt;span class="caps"&gt;AAP&lt;/span&gt; subscription&amp;nbsp;entitlements.&lt;/p&gt;
&lt;p&gt;That full tag is a snapshot of the image I tested for this article, not a promise that it will remain the recommended build forever. Before standardizing it across a team, select the tag for your &lt;span class="caps"&gt;AAP&lt;/span&gt; release from the &lt;a href="https://catalog.redhat.com/software/containers/search"&gt;Red Hat Ecosystem Catalog&lt;/a&gt; or the current &lt;span class="caps"&gt;AAP&lt;/span&gt; documentation, test it, and pin it deliberately. A&amp;nbsp;floating &lt;code&gt;latest&lt;/code&gt; tag is convenient for discovery; an exact build tag is better for a tested configuration, and a manifest-list digest is the strongest team contract when immutable reproducibility is&amp;nbsp;required.&lt;/p&gt;
&lt;p&gt;There is one subtle support distinction in this stack. &lt;span class="caps"&gt;RHEL&lt;/span&gt;, Podman and the downstream Ansible content are Red Hat deliverables. Visual Studio Code, the Python extension and the Dev Containers extension are Microsoft deliverables installed from Microsoft&amp;#8217;s channels. The Ansible extension is published by Red Hat. I therefore call this a &lt;strong&gt;downstream-first Ansible environment&lt;/strong&gt;, not a magically single-vendor-supported desktop stack. The important part is that the Ansible binaries that produce, lint, build and run content come from the downstream product&amp;nbsp;ecosystem.&lt;/p&gt;
&lt;h2 id="dev-container-and-execution-environment-are-different-jobs"&gt;Dev Container and Execution Environment Are Different&amp;nbsp;Jobs&lt;/h2&gt;
&lt;p&gt;Container terminology becomes muddy quickly, so it is worth stopping&amp;nbsp;here.&lt;/p&gt;
&lt;p&gt;A &lt;strong&gt;Dev Container&lt;/strong&gt; is the interactive workspace in which I edit and test content. &lt;span class="caps"&gt;VS&lt;/span&gt; Code installs its remote server and selected extensions in it, mounts the repository, opens an integrated terminal and manages the container as part of the workspace&amp;nbsp;lifecycle.&lt;/p&gt;
&lt;p&gt;An &lt;strong&gt;Ansible Execution Environment&lt;/strong&gt; (&lt;span class="caps"&gt;EE&lt;/span&gt;) is an &lt;span class="caps"&gt;OCI&lt;/span&gt; image used as an Ansible control node when automation runs. It&amp;nbsp;contains &lt;code&gt;ansible-core&lt;/code&gt;, &lt;code&gt;ansible-runner&lt;/code&gt;, collections and their Python and system&amp;nbsp;dependencies. &lt;code&gt;ansible-builder&lt;/code&gt; builds&amp;nbsp;it; &lt;code&gt;ansible-navigator&lt;/code&gt; and &lt;span class="caps"&gt;AAP&lt;/span&gt; can run&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;ADT&lt;/span&gt; image provides the development tools and can run Ansible content during development, but I do not treat it as the project&amp;#8217;s production runtime. For &lt;span class="caps"&gt;AAP&lt;/span&gt;, the project-specific &lt;span class="caps"&gt;EE&lt;/span&gt; should be built from the base image documented for the selected &lt;span class="caps"&gt;AAP&lt;/span&gt;&amp;nbsp;and &lt;code&gt;ansible-core&lt;/code&gt; release. For most custom EEs in &lt;span class="caps"&gt;AAP&lt;/span&gt; 2.7, Red Hat recommends the&amp;nbsp;appropriate &lt;code&gt;ee-minimal&lt;/code&gt; image. &lt;code&gt;ee-supported&lt;/code&gt; is the alternative when its broader curated set of content and dependencies fits the use case. The &lt;a href="https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/2.7/administer-define__create__and_build_execution_environments"&gt;&lt;span class="caps"&gt;AAP&lt;/span&gt; 2.7 execution environment image guide&lt;/a&gt; is the authority for selecting the image and&amp;nbsp;tag.&lt;/p&gt;
&lt;p&gt;I therefore develop &lt;em&gt;inside&lt;/em&gt; the Dev Container and test my content &lt;em&gt;with&lt;/em&gt; the project-specific &lt;span class="caps"&gt;EE&lt;/span&gt; that will eventually run it. The first standardizes the developer tooling; the second standardizes the automation runtime. The&amp;nbsp;documented &lt;code&gt;ee-minimal&lt;/code&gt; or other appropriate &lt;span class="caps"&gt;EE&lt;/span&gt; base is the build foundation for that&amp;nbsp;runtime.&lt;/p&gt;
&lt;p&gt;That separation matters. A role may lint perfectly in the &lt;span class="caps"&gt;ADT&lt;/span&gt; workspace and still need an &lt;span class="caps"&gt;EE&lt;/span&gt;&amp;nbsp;containing &lt;code&gt;kubernetes.core&lt;/code&gt;, a vendor &lt;span class="caps"&gt;SDK&lt;/span&gt; or an operating-system package that the generic development image does not&amp;nbsp;ship.&lt;/p&gt;
&lt;h2 id="host-preparation"&gt;Host&amp;nbsp;Preparation&lt;/h2&gt;
&lt;p&gt;Podman is part of the supported &lt;span class="caps"&gt;RHEL&lt;/span&gt; container-tools stack. Install it from &lt;span class="caps"&gt;RHEL&lt;/span&gt; rather than adding an unrelated container&amp;nbsp;repository:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;container-tools
podman&lt;span class="w"&gt; &lt;/span&gt;--version
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Red Hat documents Podman as a container-platform option for this &lt;span class="caps"&gt;AAP&lt;/span&gt; development workflow. The Microsoft Dev Containers extension reaches it through its Docker-compatible &lt;span class="caps"&gt;CLI&lt;/span&gt; integration: Microsoft describes Podman 5 and later as mostly Docker-&lt;span class="caps"&gt;CLI&lt;/span&gt; compatible, while noting that alternative compatible CLIs are not formally supported by the extension itself. Those are different support statements, and both&amp;nbsp;matter.&lt;/p&gt;
&lt;p&gt;On this workstation I select Podman explicitly in the &lt;span class="caps"&gt;VS&lt;/span&gt; Code user&amp;nbsp;settings:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;dev.containers.dockerPath&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;podman&amp;quot;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the setting documented by Microsoft for Podman. It tells Dev Containers which &lt;span class="caps"&gt;CLI&lt;/span&gt; to invoke; the&amp;nbsp;Podman-specific &lt;code&gt;devcontainer.json&lt;/code&gt; still defines how this particular workspace is&amp;nbsp;launched.&lt;/p&gt;
&lt;p&gt;I use rootless Podman for the workspace. The downstream image is authenticated content, so log in before launching &lt;span class="caps"&gt;VS&lt;/span&gt;&amp;nbsp;Code:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;login&lt;span class="w"&gt; &lt;/span&gt;registry.redhat.io
podman&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;registry.redhat.io/ansible-automation-platform-27/ansible-dev-tools-rhel9:26.7.1-1783617973
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The order is not superstition. &lt;span class="caps"&gt;VS&lt;/span&gt; Code needs access to the existing Podman registry credentials when it opens the container. If the container later needs to pull another authenticated &lt;span class="caps"&gt;EE&lt;/span&gt;, authenticate from its integrated terminal as well; the inner Podman has its own authentication context unless you explicitly share&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;Microsoft publishes stable &lt;span class="caps"&gt;VS&lt;/span&gt; Code RPMs for &lt;span class="caps"&gt;RHEL&lt;/span&gt;-family systems from its own repository. I&amp;nbsp;install &lt;code&gt;code&lt;/code&gt; there and then add these extensions from the Visual Studio&amp;nbsp;Marketplace:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ms-python.python
ms-vscode-remote.remote-containers
redhat.ansible
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;a href="https://code.visualstudio.com/docs/setup/linux"&gt;Microsoft Linux installation documentation&lt;/a&gt; contains the current repository setup. I prefer following it instead of copying repository bootstrap commands into an article that will outlive&amp;nbsp;them.&lt;/p&gt;
&lt;h2 id="the-devcontainerjson"&gt;The &lt;code&gt;devcontainer.json&lt;/code&gt;&lt;/h2&gt;
&lt;p&gt;The Red Hat Ansible extension can generate the Dev Container configuration: open the Ansible activity view, choose &lt;strong&gt;Devcontainer&lt;/strong&gt; under Ansible Development Tools, select the downstream image, and then reopen the workspace in the Podman configuration. The &lt;a href="https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/2.7/install-assembly_devtools_install"&gt;&lt;span class="caps"&gt;AAP&lt;/span&gt; 2.7 Ansible Development Tools installation guide&lt;/a&gt; documents the supported installation paths and current&amp;nbsp;requirements.&lt;/p&gt;
&lt;p&gt;Here is the configuration behind the screenshot, cleaned up so the image reference is on one line and with the Python extension declared&amp;nbsp;explicitly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;name&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ansible-dev-container-podman&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;image&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;registry.redhat.io/ansible-automation-platform-27/ansible-dev-tools-rhel9:26.7.1-1783617973&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;containerUser&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;root&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;runArgs&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--cap-add=CAP_MKNOD&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--cap-add=NET_ADMIN&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--cap-add=SYS_ADMIN&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--cap-add=SYS_RESOURCE&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--device&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/dev/fuse&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--security-opt&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;seccomp=unconfined&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--security-opt&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;label=disable&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--security-opt&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;apparmor=unconfined&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--security-opt&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;unmask=/sys/fs/cgroup&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--userns=host&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--hostname=ansible-dev-container&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;customizations&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;vscode&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;extensions&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ms-python.python&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;redhat.ansible&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;redhat.vscode-redhat-account&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The Dev Containers extension is the local user-interface extension that orchestrates the connection, so it is not in the&amp;nbsp;container-side &lt;code&gt;customizations.vscode.extensions&lt;/code&gt; list. Python and Ansible can be installed in the local &lt;span class="caps"&gt;VS&lt;/span&gt; Code profile, but their workspace-facing components need to run where Python, Ansible and the project files live. Declaring them&amp;nbsp;in &lt;code&gt;customizations&lt;/code&gt; installs and activates them in the remote container workspace. The Ansible extension must also be available locally if you want to use its activity view to scaffold the Dev Container in the first place. The Red Hat Authentication extension is useful for Red Hat service sign-in but does not&amp;nbsp;replace &lt;code&gt;podman login&lt;/code&gt; for the registry&amp;nbsp;pull.&lt;/p&gt;
&lt;p&gt;Save the file&amp;nbsp;as &lt;code&gt;.devcontainer/podman/devcontainer.json&lt;/code&gt;. In &lt;span class="caps"&gt;VS&lt;/span&gt; Code, run &lt;strong&gt;Dev Containers: Reopen in Container&lt;/strong&gt; and select the Podman definition. &lt;span class="caps"&gt;VS&lt;/span&gt; Code creates the container, attaches the workspace and installs the declared remote extensions. &lt;span class="caps"&gt;VS&lt;/span&gt; Code manages the generated container as part of the workspace lifecycle and starts or recreates it when&amp;nbsp;required.&lt;/p&gt;
&lt;h3 id="those-privileges-are-not-decorative"&gt;Those Privileges Are Not&amp;nbsp;Decorative&lt;/h3&gt;
&lt;p&gt;The configuration&amp;nbsp;grants &lt;code&gt;SYS_ADMIN&lt;/code&gt;,&amp;nbsp;exposes &lt;code&gt;/dev/fuse&lt;/code&gt;, disables seccomp filtering and SELinux label separation for the container, and runs its user as root. This generated configuration is deliberately permissive to support nested Podman and Execution Environment builds. Not every project necessarily requires every capability, device, namespace option or security exception in that&amp;nbsp;baseline.&lt;/p&gt;
&lt;p&gt;It also means the Dev Container should not be treated as a strong security sandbox for hostile code. Rootless Podman still limits the container to the privileges of the invoking host user, but these capabilities, devices and security exceptions deliberately weaken several layers of isolation. Use it for repositories you trust. Do not open unreviewed code and assume the container protects the host. If a project does not need nested container builds, reduce the capabilities and security exceptions, then verify that its actual workflow still functions. Standardization should include the threat model, not just the tool&amp;nbsp;versions.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;apparmor=unconfined&lt;/code&gt; option is relevant on AppArmor hosts. &lt;span class="caps"&gt;RHEL&lt;/span&gt; uses SELinux,&amp;nbsp;where &lt;code&gt;label=disable&lt;/code&gt; is the material setting. Keeping both in a cross-platform generated file is practical, but they do not describe the same security&amp;nbsp;mechanism.&lt;/p&gt;
&lt;h2 id="what-is-actually-in-the-image"&gt;What Is Actually in the&amp;nbsp;Image?&lt;/h2&gt;
&lt;p&gt;Once &lt;span class="caps"&gt;VS&lt;/span&gt; Code has opened the workspace, its integrated terminal is already inside the container. There is normally no reason to chase the generated container name&amp;nbsp;with &lt;code&gt;podman exec&lt;/code&gt;, but the host can confirm that it is&amp;nbsp;running:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;ps&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;table {{.ID}}\t{{.Image}}\t{{.Status}}\t{{.Names}}&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For the image used here, shell completion shows the expected&amp;nbsp;commands:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible-builder       ansible-doc           ansible-navigator     ansible-sign
ansible-config        ansible-galaxy        ansible-playbook      ansible-vault
ansible-connection    ansible-inventory     ansible-pull
ansible-console       ansible-lint          ansible-runner
ansible-creator
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;RPM&lt;/span&gt; query is even more useful because it proves that these are productized packages rather than a mystery venv assembled&amp;nbsp;by &lt;code&gt;pip&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible-creator-26.6.1-2.el9ap.noarch
ansible-sign-0.1.6-1.el9ap.noarch
ansible-builder-3.1.1-3.1.el9ap.noarch
ansible-dev-environment-26.6.1-1.el9ap.noarch
ansible-core-2.16.19-1.el9ap.noarch
ansible-lint-26.6.0-1.el9ap.noarch
ansible-navigator-26.6.0-2.el9ap.noarch
ansible-dev-tools-26.7.1-1.el9ap.noarch
ansible-dev-tools+server-26.7.1-1.el9ap.noarch
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The versions are intentionally not all identical. The &lt;span class="caps"&gt;ADT&lt;/span&gt; image is an integrated deliverable composed of multiple independently versioned components. The image tag identifies the tested assembly; it does not imply that every &lt;span class="caps"&gt;RPM&lt;/span&gt; inside has the same version&amp;nbsp;number.&lt;/p&gt;
&lt;p&gt;For a quick health check I&amp;nbsp;use:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible&lt;span class="w"&gt; &lt;/span&gt;--version
ansible-lint&lt;span class="w"&gt; &lt;/span&gt;--version
ansible-navigator&lt;span class="w"&gt; &lt;/span&gt;--version
ansible-builder&lt;span class="w"&gt; &lt;/span&gt;--version
rpm&lt;span class="w"&gt; &lt;/span&gt;-qa&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;ansible*&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sort
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then I lint and run a trivial playbook before doing real&amp;nbsp;work:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Verify the development workspace&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;localhost&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;gather_facts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;tasks&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Report a working Ansible environment&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.debug&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;msg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Hello from the Ansible Dev Tools container&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible-lint&lt;span class="w"&gt; &lt;/span&gt;test.yml
ansible-navigator&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;test.yml&lt;span class="w"&gt; &lt;/span&gt;--mode&lt;span class="w"&gt; &lt;/span&gt;stdout
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The screenshot at the top shows exactly that boundary working: &lt;span class="caps"&gt;VS&lt;/span&gt; Code is attached to the Dev Container, the Ansible language tooling understands the playbook,&amp;nbsp;and &lt;code&gt;ansible-navigator&lt;/code&gt; runs it through&amp;nbsp;Podman.&lt;/p&gt;
&lt;h2 id="why-this-is-better-than-a-carefully-tended-venv"&gt;Why This Is Better Than a Carefully Tended&amp;nbsp;Venv&lt;/h2&gt;
&lt;p&gt;A Python virtual environment is useful, but it only owns Python packages. An Ansible development workflow also depends on collections, RPMs, container tooling, compiler libraries, &lt;span class="caps"&gt;CA&lt;/span&gt; trust, shell utilities and editor-side language services. A &lt;span class="caps"&gt;README&lt;/span&gt; full of installation commands describes an environment; a pinned &lt;span class="caps"&gt;OCI&lt;/span&gt; image &lt;em&gt;is&lt;/em&gt; an&amp;nbsp;environment.&lt;/p&gt;
&lt;p&gt;The downstream &lt;span class="caps"&gt;ADT&lt;/span&gt; image gives a team several useful&amp;nbsp;properties:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;One tested tool assembly.&lt;/strong&gt; Developers do not independently choose incompatible versions&amp;nbsp;of &lt;code&gt;ansible-core&lt;/code&gt;, &lt;code&gt;ansible-lint&lt;/code&gt;, &lt;code&gt;ansible-builder&lt;/code&gt; and &lt;code&gt;ansible-navigator&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A clean host.&lt;/strong&gt; &lt;span class="caps"&gt;RHEL&lt;/span&gt; needs Podman and the editor, not a growing pile of project-specific Python&amp;nbsp;packages.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Cheap onboarding.&lt;/strong&gt; Clone, authenticate, reopen in the container and start&amp;nbsp;working.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Visible upgrades.&lt;/strong&gt; Changing the image tag is a reviewable repository diff instead of silent workstation&amp;nbsp;drift.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Alignment with the product ecosystem.&lt;/strong&gt; The tools that validate and package enterprise automation are the downstream artifacts associated with &lt;span class="caps"&gt;AAP&lt;/span&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The strongest benefit is not that containers are fashionable. It is that the environment becomes a versioned team&amp;nbsp;decision.&lt;/p&gt;
&lt;p&gt;There are still limits. The Dev Container does not make project dependencies disappear, it does not replace a purpose-built &lt;span class="caps"&gt;EE&lt;/span&gt;, and it does not guarantee that content is correct. Pin collections, lint in &lt;span class="caps"&gt;CI&lt;/span&gt;, test against the intended managed nodes, and run the same project &lt;span class="caps"&gt;EE&lt;/span&gt; in development and automation controller. A standardized mistake is still a mistake; it is merely easier to&amp;nbsp;reproduce.&lt;/p&gt;
&lt;h2 id="upstream-alternative-community-ansible-dev-tools"&gt;Upstream Alternative: Community Ansible Dev&amp;nbsp;Tools&lt;/h2&gt;
&lt;p&gt;Not every project needs, or has access to, &lt;span class="caps"&gt;AAP&lt;/span&gt; content. The direct upstream equivalent&amp;nbsp;is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ghcr.io/ansible/community-ansible-dev-tools:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The Ansible Project documents this image specifically for &lt;span class="caps"&gt;VS&lt;/span&gt; Code Dev Containers and for Podman-based development. It is Fedora Minimal based and contains the upstream &lt;span class="caps"&gt;ADT&lt;/span&gt; packages. Replace&amp;nbsp;the &lt;code&gt;image&lt;/code&gt; value in the&amp;nbsp;same &lt;code&gt;devcontainer.json&lt;/code&gt;, or let the Ansible extension generate an upstream&amp;nbsp;configuration.&lt;/p&gt;
&lt;p&gt;For experiments and community&amp;nbsp;work, &lt;code&gt;latest&lt;/code&gt; is convenient. For a shared or long-lived project, pin a release tag or, where exact reproducibility matters, an image digest. The &lt;a href="https://docs.ansible.com/projects/dev-tools/container/"&gt;upstream &lt;span class="caps"&gt;ADT&lt;/span&gt; image documentation&lt;/a&gt; explains the published tag semantics; in&amp;nbsp;particular, &lt;code&gt;latest&lt;/code&gt;, &lt;code&gt;main&lt;/code&gt; and &lt;code&gt;devel&lt;/code&gt; carry different stability&amp;nbsp;expectations.&lt;/p&gt;
&lt;p&gt;This is a first-class &lt;strong&gt;upstream project&lt;/strong&gt; workflow. It is not the downstream &lt;span class="caps"&gt;AAP&lt;/span&gt; image and should not be described as product-supported simply because many of the package names are familiar. Conversely, “not the downstream product artifact” does not mean “bad” or “unsafe.” It means that its release and support contract is the community project&amp;#8217;s&amp;nbsp;contract.&lt;/p&gt;
&lt;p&gt;The same rule applies&amp;nbsp;to &lt;code&gt;pip install ansible-dev-tools&lt;/code&gt;. It is the documented upstream Python installation path and an excellent option for a venv-based community environment. It is not a substitute for &lt;span class="caps"&gt;AAP&lt;/span&gt;-packaged RPMs when downstream supportability is the&amp;nbsp;requirement.&lt;/p&gt;
&lt;h2 id="the-editor-is-a-separate-choice"&gt;The Editor Is a Separate&amp;nbsp;Choice&lt;/h2&gt;
&lt;p&gt;The provenance decision does not dictate the editor. &lt;span class="caps"&gt;VS&lt;/span&gt; Code is not the only editor capable of a good Ansible experience. I maintain &lt;a href="https://codeberg.org/Larvitz/nvim-ansible"&gt;nvim-ansible&lt;/a&gt;, a Neovim configuration with completion, linting, diagnostics, syntax highlighting and Ansible-aware&amp;nbsp;documentation.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Neovim with Ansible completion and module documentation" src="https://hofstede.it/images/2026-08-03-nvim-ansible.webp" title="Neovim with Ansible completion and module documentation"&gt;&lt;/p&gt;
&lt;p&gt;This community configuration can sit on top of upstream Ansible tools from a venv or container, and it can equally well talk to a downstream toolchain if the language-server and executable paths are configured accordingly. The editor choice and the artifact provenance are separate&amp;nbsp;decisions.&lt;/p&gt;
&lt;p&gt;I prefer &lt;span class="caps"&gt;VS&lt;/span&gt; Code plus Dev Containers for a team because the onboarding path is exceptionally short and the Red Hat Ansible extension can scaffold the workspace. I prefer Neovim when I want a terminal-native environment I understand down to the last&amp;nbsp;mapping.&lt;/p&gt;
&lt;h2 id="a-practical-team-policy"&gt;A Practical Team&amp;nbsp;Policy&lt;/h2&gt;
&lt;p&gt;If I were standardizing this beyond my own workstation, I would put the following in the&amp;nbsp;repository:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;A&amp;nbsp;reviewed &lt;code&gt;.devcontainer/&lt;/code&gt; definition with an exact &lt;span class="caps"&gt;ADT&lt;/span&gt; build tag or, where immutable reproducibility is required, a manifest-list&amp;nbsp;digest.&lt;/li&gt;
&lt;li&gt;A documented upstream or downstream provenance decision. Do not mix the two&amp;nbsp;accidentally.&lt;/li&gt;
&lt;li&gt;A&amp;nbsp;project-specific &lt;code&gt;execution-environment.yml&lt;/code&gt;, based on the appropriate documented &lt;span class="caps"&gt;EE&lt;/span&gt; base, plus pinned collection and Python&amp;nbsp;requirements.&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;CI&lt;/span&gt; jobs that&amp;nbsp;run &lt;code&gt;ansible-lint&lt;/code&gt; and tests using the agreed image&amp;nbsp;versions.&lt;/li&gt;
&lt;li&gt;A scheduled upgrade pull request rather than silent use of floating&amp;nbsp;tags.&lt;/li&gt;
&lt;li&gt;A short security note explaining why the nested-Podman capabilities are&amp;nbsp;present.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;For an &lt;span class="caps"&gt;AAP&lt;/span&gt; customer, I would also map the &lt;span class="caps"&gt;ADT&lt;/span&gt; and &lt;span class="caps"&gt;EE&lt;/span&gt; versions to the organization&amp;#8217;s supported &lt;span class="caps"&gt;AAP&lt;/span&gt; release and lifecycle before rollout. “Newest” is not automatically “correct,” especially when automation controller, certified collections and managed-node Python support all impose compatibility&amp;nbsp;boundaries.&lt;/p&gt;
&lt;p&gt;That is my definition of sane: boring setup, explicit provenance, reviewable upgrades and no ambiguity about whether a component came from the Ansible Project or the Red Hat&amp;nbsp;product.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/"&gt;Developing automation content with Red Hat Ansible Automation&amp;nbsp;Platform&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/2.7/install-assembly_devtools_install"&gt;Red Hat Ansible Development Tools installation guide for &lt;span class="caps"&gt;AAP&lt;/span&gt;&amp;nbsp;2.7&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_ansible_automation_platform/2.7/administer-define__create__and_build_execution_environments"&gt;&lt;span class="caps"&gt;AAP&lt;/span&gt; 2.7 execution environment image&amp;nbsp;and &lt;code&gt;ansible-core&lt;/code&gt; selection&amp;nbsp;guide&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.ansible.com/projects/vscode-ansible/"&gt;Ansible &lt;span class="caps"&gt;VS&lt;/span&gt; Code extension: Dev&amp;nbsp;Containers&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.ansible.com/projects/dev-tools/container/"&gt;Upstream Ansible Development Tools container&amp;nbsp;image&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.ansible.com/projects/dev-tools/"&gt;Ansible Development Tools package&amp;nbsp;overview&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/10/html/building_running_and_managing_containers/"&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 container tools&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://code.visualstudio.com/docs/setup/linux"&gt;Microsoft&amp;#8217;s &lt;span class="caps"&gt;VS&lt;/span&gt; Code installation instructions for&amp;nbsp;Linux&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://code.visualstudio.com/remote/advancedcontainers/docker-options"&gt;Microsoft&amp;#8217;s Dev Containers documentation for Podman and alternative&amp;nbsp;CLIs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/reproducible-ansible-with-execution-environments/"&gt;Reproducible Ansible with Execution&amp;nbsp;Environments&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="ansible"/><category term="ansible-automation-platform"/><category term="ansible-dev-tools"/><category term="vscode"/><category term="dev-containers"/><category term="podman"/><category term="rhel"/><category term="automation"/><category term="development"/></entry><entry><title>Android Development on Fedora 44 Without Installing an Android Development Stack</title><link href="https://hofstede.it/android-development-on-fedora-44-without-installing-an-android-development-stack/" rel="alternate"/><published>2026-07-24T00:00:00+02:00</published><updated>2026-07-24T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-07-24:/android-development-on-fedora-44-without-installing-an-android-development-stack/</id><summary type="html">&lt;p&gt;I wanted to build a native Android companion for Kollect without filling Fedora 44 with a &lt;span class="caps"&gt;JDK&lt;/span&gt;, Gradle, Android SDKs, build tools, emulator images, and assorted mobile-development debris. My complete Android toolchain now lives in an Ubuntu 24.04 Distrobox instead. It builds the &lt;span class="caps"&gt;APK&lt;/span&gt;, talks to my phone over wireless &lt;span class="caps"&gt;ADB&lt;/span&gt;, runs the Android emulator, and even lets Kollect enrich links for free with Gemini Nano through Android&amp;nbsp;AICore.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;I recently wrote about &lt;a href="https://hofstede.it/kollect-the-native-kde-link-collector-i-built-for-myself/"&gt;Kollect, the native &lt;span class="caps"&gt;KDE&lt;/span&gt; link collector I built for myself&lt;/a&gt;. At the time it was primarily a Plasma application: give it a &lt;span class="caps"&gt;URL&lt;/span&gt;, let a language model turn an untidy web page into useful metadata, and store the result in plain files or on my small sync&amp;nbsp;server.&lt;/p&gt;
&lt;p&gt;The predictable next step was an Android companion. Links often reach me on my phone first, and Android already has exactly the capture mechanism I want: the share sheet. Share a page to Kollect, let the application fetch and enrich it, and the same entry appears in the collection on my&amp;nbsp;desktop.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Kollect for Android showing metadata generated locally by AICore and Gemini Nano on a Pixel 10 Pro" src="https://hofstede.it/images/2026-07-24-kollect-android-aicore.webp" title="A link enriched on the Pixel 10 Pro in about 15 seconds. The audit fields show the on-device AICore provider and nano-v3 model."&gt;&lt;/p&gt;
&lt;p&gt;There was just one problem. I did not want to turn my Fedora 44 workstation into an Android development&amp;nbsp;workstation.&lt;/p&gt;
&lt;p&gt;Android development brings a &lt;span class="caps"&gt;JDK&lt;/span&gt;, Gradle distributions, Android &lt;span class="caps"&gt;SDK&lt;/span&gt; platforms, build tools, platform tools, emulator binaries, system images, caches, and usually a large &lt;span class="caps"&gt;IDE&lt;/span&gt;. None of those things are objectionable on their own. I simply did not want that whole ecosystem mixed into the operating system I use every day, especially for one small&amp;nbsp;application.&lt;/p&gt;
&lt;p&gt;So I put all of it in a&amp;nbsp;Distrobox.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-boundary-i-wanted"&gt;The boundary I&amp;nbsp;wanted&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#creating-the-android-box"&gt;Creating the Android&amp;nbsp;box&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#installing-only-the-command-line-stack"&gt;Installing only the command-line&amp;nbsp;stack&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#wireless-adb-from-inside-the-container"&gt;Wireless &lt;span class="caps"&gt;ADB&lt;/span&gt; from inside the&amp;nbsp;container&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-emulator-is-in-the-box-too"&gt;The emulator is in the box&amp;nbsp;too&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-the-android-application-adds"&gt;What the Android application&amp;nbsp;adds&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#metadata-enrichment-with-gemini-nano-and-aicore"&gt;Metadata enrichment with Gemini Nano and&amp;nbsp;AICore&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#secrets-and-failure-records"&gt;Secrets and failure&amp;nbsp;records&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-this-setup-does-and-does-not-buy-me"&gt;What this setup does and does not buy&amp;nbsp;me&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="the-boundary-i-wanted"&gt;The boundary I&amp;nbsp;wanted&lt;/h2&gt;
&lt;p&gt;The host has Fedora 44, Podman, Distrobox, my editor, and the Kollect checkout. That is the complete host-side requirement. Java does not need to be installed there. Neither do&amp;nbsp;Gradle, &lt;code&gt;adb&lt;/code&gt;, the Android &lt;span class="caps"&gt;SDK&lt;/span&gt;, the emulator, or Android&amp;nbsp;Studio.&lt;/p&gt;
&lt;p&gt;The box contains the complete mobile&amp;nbsp;toolchain:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Ubuntu&amp;nbsp;24.04&lt;/li&gt;
&lt;li&gt;OpenJDK&amp;nbsp;17&lt;/li&gt;
&lt;li&gt;the Android command-line tools and &lt;span class="caps"&gt;SDK&lt;/span&gt;&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;API&lt;/span&gt; 37, Build Tools 36.0.0, and platform&amp;nbsp;tools&lt;/li&gt;
&lt;li&gt;Gradle 9.5.1, downloaded and pinned by the project&amp;#8217;s Gradle&amp;nbsp;wrapper&lt;/li&gt;
&lt;li&gt;Android emulator binaries and an &lt;span class="caps"&gt;API&lt;/span&gt; 36 virtual&amp;nbsp;device&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I use Ubuntu for this box for a wonderfully mundane reason: Fedora 44 no longer provides&amp;nbsp;the &lt;code&gt;java-17-openjdk-devel&lt;/code&gt; package name on which my first setup attempt relied. Ubuntu 24.04&amp;nbsp;has &lt;code&gt;openjdk-17-jdk&lt;/code&gt;, so the path of least resistance was to use an Ubuntu userspace on my Fedora host. Choosing whichever distribution makes a project boring is one of Distrobox&amp;#8217;s best&amp;nbsp;features.&lt;/p&gt;
&lt;p&gt;This is the same general pattern I described in &lt;a href="https://hofstede.it/distrobox-different-distributions-in-a-box-powered-by-podman/"&gt;Distrobox: Different Distributions in a Box, Powered by Podman&lt;/a&gt;, but Android development is an unusually good example because the toolchain is both large and mostly&amp;nbsp;self-contained.&lt;/p&gt;
&lt;h2 id="creating-the-android-box"&gt;Creating the Android&amp;nbsp;box&lt;/h2&gt;
&lt;p&gt;My development box is&amp;nbsp;called &lt;code&gt;kollect-android&lt;/code&gt; and uses the stock Ubuntu 24.04 image. I gave it a separate home directory as&amp;nbsp;well:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;distrobox&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;kollect-android&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--image&lt;span class="w"&gt; &lt;/span&gt;docker.io/library/ubuntu:24.04&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--home&lt;span class="w"&gt; &lt;/span&gt;/home/chofstede/.local/share/distrobox/kollect-android-home&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--additional-packages&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;openjdk-17-jdk curl unzip&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The custom home matters more than the distribution choice. A normal Distrobox shares the normal host home, which is convenient but would put Gradle caches, Android configuration, debug keys, &lt;span class="caps"&gt;SDK&lt;/span&gt; files, and whatever else the tools invent among my regular dotfiles. This box instead sees its own clean home while Distrobox still exposes the host filesystem&amp;nbsp;below &lt;code&gt;/run/host&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;That makes the split very easy to&amp;nbsp;understand:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Fedora host
├── /home/chofstede/tmp/Kollect/              source checkout
└── ~/.local/share/distrobox/
    └── kollect-android-home/
        ├── android-sdk/                       SDK and platform tools
        ├── .gradle/                           Gradle downloads and caches
        └── .config/.android/debug.keystore    debug signing identity
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The source remains a normal file on the host. Inside the box, the same checkout is available&amp;nbsp;at:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;/run/host/home/chofstede/tmp/Kollect
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There is no copy, synchronisation step, or container volume to remember. My editor works on the host path and the compiler works on the&amp;nbsp;corresponding &lt;code&gt;/run/host&lt;/code&gt; path. Both are looking at the same&amp;nbsp;files.&lt;/p&gt;
&lt;p&gt;One important caveat: this is organisation, not a security boundary. Distrobox deliberately integrates the container with the desktop and mounts host filesystems into it. I use the box to keep packages and caches tidy, not to execute hostile software&amp;nbsp;safely.&lt;/p&gt;
&lt;h2 id="installing-only-the-command-line-stack"&gt;Installing only the command-line&amp;nbsp;stack&lt;/h2&gt;
&lt;p&gt;Enter the box&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;distrobox&lt;span class="w"&gt; &lt;/span&gt;enter&lt;span class="w"&gt; &lt;/span&gt;kollect-android
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;My interactive shell is Fish. After unpacking Google&amp;#8217;s current Android command-line tools&amp;nbsp;into &lt;code&gt;$HOME/android-sdk/cmdline-tools/latest&lt;/code&gt;, I make the &lt;span class="caps"&gt;SDK&lt;/span&gt; and its tools persistent in the&amp;nbsp;box:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;set&lt;/span&gt; -Ux ANDROID_HOME &lt;span class="nv"&gt;$HOME&lt;/span&gt;/android-sdk
fish_add_path &lt;span class="nv"&gt;$ANDROID_HOME&lt;/span&gt;/platform-tools
fish_add_path &lt;span class="nv"&gt;$ANDROID_HOME&lt;/span&gt;/cmdline-tools/latest/bin
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The current Android &lt;span class="caps"&gt;CLI&lt;/span&gt; uses&amp;nbsp;the &lt;code&gt;android&lt;/code&gt; command rather than the&amp;nbsp;old &lt;code&gt;sdkmanager&lt;/code&gt; examples found in years of search results. Kollect currently compiles against Android 17/&lt;span class="caps"&gt;API&lt;/span&gt; 37, targets &lt;span class="caps"&gt;API&lt;/span&gt; 37, and requires at least Android 16/&lt;span class="caps"&gt;API&lt;/span&gt; 36. The matching &lt;span class="caps"&gt;SDK&lt;/span&gt; components are installed like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;android --sdk&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$ANDROID_HOME&lt;/span&gt; sdk install &lt;span class="se"&gt;\&lt;/span&gt;
  platforms/android-37.0 &lt;span class="se"&gt;\&lt;/span&gt;
  build-tools/36.0.0 &lt;span class="se"&gt;\&lt;/span&gt;
  platform-tools
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The authoritative versions remain in the project, not in my&amp;nbsp;memory: &lt;code&gt;compileSdk&lt;/code&gt;, &lt;code&gt;targetSdk&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;minSdk&lt;/code&gt; live&amp;nbsp;in &lt;code&gt;app/build.gradle.kts&lt;/code&gt;; Android Gradle Plugin 9.2.1 is pinned in the top-level build file; and the wrapper pins Gradle 9.5.1. The box only supplies &lt;span class="caps"&gt;JDK&lt;/span&gt; 17 and the requested Android &lt;span class="caps"&gt;SDK&lt;/span&gt;&amp;nbsp;components.&lt;/p&gt;
&lt;p&gt;I can use Android Studio with this setup, but I do not need it to build the project. Kollect is a native Kotlin application with Jetpack Compose, and the checked-in Gradle wrapper is&amp;nbsp;enough:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;cd&lt;/span&gt; /run/host/home/chofstede/tmp/Kollect/android
./gradlew assembleDebug
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The resulting &lt;span class="caps"&gt;APK&lt;/span&gt; is a normal host-visible&amp;nbsp;file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;app/build/outputs/apk/debug/app-debug.apk
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I can inspect the version embedded in it before&amp;nbsp;installation:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;apkanalyzer manifest version-code app/build/outputs/apk/debug/app-debug.apk
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Nothing in that workflow knows or cares that Fedora itself has no Java&amp;nbsp;installation.&lt;/p&gt;
&lt;h2 id="wireless-adb-from-inside-the-container"&gt;Wireless &lt;span class="caps"&gt;ADB&lt;/span&gt; from inside the&amp;nbsp;container&lt;/h2&gt;
&lt;p&gt;Building an &lt;span class="caps"&gt;APK&lt;/span&gt; in a container is the easy part. The more interesting question is whether the tools in the box can actually reach a physical Android&amp;nbsp;device.&lt;/p&gt;
&lt;p&gt;They can. I prefer Android&amp;#8217;s wireless debugging because it avoids &lt;span class="caps"&gt;USB&lt;/span&gt; device permissions and&amp;nbsp;host-side &lt;code&gt;udev&lt;/code&gt; rules entirely. After enabling Developer options and Wireless debugging on the phone, I pair and connect from inside the&amp;nbsp;Distrobox:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;adb pair PHONE_IP:PAIRING_PORT
adb connect PHONE_IP:DEBUG_PORT
adb devices -l
adb install -r app/build/outputs/apk/debug/app-debug.apk
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The pairing port and debugging port shown by Android are normally different. That is an easy detail to miss when pairing succeeds but the subsequent connection does&amp;nbsp;not.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;-r&lt;/code&gt; updates the existing application while preserving its data. Debug builds are signed with a machine-specific key, and this box keeps that key&amp;nbsp;under &lt;code&gt;$HOME/.config/.android/debug.keystore&lt;/code&gt;. Preserving the custom Distrobox home therefore also preserves the identity that Android expects when I install the next&amp;nbsp;build.&lt;/p&gt;
&lt;p&gt;If I build with a different debug key, Android correctly refuses to update the installed package. Uninstalling first would solve the signature mismatch, but it would also clear local credentials, settings, cached entries, and any unsynchronised work. Keeping the box home backed up is a much nicer&amp;nbsp;solution.&lt;/p&gt;
&lt;h2 id="the-emulator-is-in-the-box-too"&gt;The emulator is in the box&amp;nbsp;too&lt;/h2&gt;
&lt;p&gt;Distrobox passes enough of the desktop session through that the Android emulator can run as a normal graphical application from the container. I have an &lt;span class="caps"&gt;API&lt;/span&gt; 36 virtual device&amp;nbsp;named &lt;code&gt;aps_test&lt;/code&gt;, which I start&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;$ANDROID_HOME&lt;/span&gt;/emulator/emulator &lt;span class="se"&gt;\&lt;/span&gt;
  -avd aps_test &lt;span class="se"&gt;\&lt;/span&gt;
  -noaudio &lt;span class="se"&gt;\&lt;/span&gt;
  -no-boot-anim &lt;span class="se"&gt;\&lt;/span&gt;
  -no-snapshot-load
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On this machine the details are slightly counter-intuitive. I must not&amp;nbsp;add &lt;code&gt;-no-window&lt;/code&gt;: the current emulator and graphics combination crashes while initialising a headless display. I also do not&amp;nbsp;force &lt;code&gt;QT_QPA_PLATFORM=wayland&lt;/code&gt;, because Google&amp;#8217;s emulator bundle does not contain the native Qt Wayland platform plugin. Its &lt;span class="caps"&gt;GUI&lt;/span&gt; uses Qt/&lt;span class="caps"&gt;XCB&lt;/span&gt; through XWayland and works perfectly well that&amp;nbsp;way.&lt;/p&gt;
&lt;p&gt;Once Android has booted, the same build and install loop works against the virtual&amp;nbsp;device:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;./gradlew assembleDebug
&lt;span class="nv"&gt;$ANDROID_HOME&lt;/span&gt;/platform-tools/adb install -r &lt;span class="se"&gt;\&lt;/span&gt;
  app/build/outputs/apk/debug/app-debug.apk
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the point where this stopped feeling like a workaround. The compiler, &lt;span class="caps"&gt;SDK&lt;/span&gt;, device bridge, and graphical emulator all live in the container, yet the source tree, windows, network, and physical phone behave exactly as I need them&amp;nbsp;to.&lt;/p&gt;
&lt;h2 id="what-the-android-application-adds"&gt;What the Android application&amp;nbsp;adds&lt;/h2&gt;
&lt;p&gt;Kollect for Android is not a port of the Plasma interface. It is a small native Compose companion built for the phone&amp;nbsp;workflow.&lt;/p&gt;
&lt;p&gt;It receives URLs from Android&amp;#8217;s share sheet, stores them in an on-device SQLite queue, and uploads them to the existing Kollect server. WorkManager retries uploads when connectivity returns. The server remains the source of truth; SQLite is a cache and durable queue rather than a second local-collection&amp;nbsp;mode.&lt;/p&gt;
&lt;p&gt;The application can then fetch the page, remove markup and other noise, keep at most 14,000 characters of readable text, and ask a model for the same structured result as the desktop client: headline, publication date, tags, topic, author, and a short summary. OpenRouter remains available and works on any supported phone with network access and an &lt;span class="caps"&gt;API&lt;/span&gt;&amp;nbsp;key.&lt;/p&gt;
&lt;p&gt;But my Pixel 10 Pro offered a much more interesting second&amp;nbsp;path.&lt;/p&gt;
&lt;h2 id="metadata-enrichment-with-gemini-nano-and-aicore"&gt;Metadata enrichment with Gemini Nano and&amp;nbsp;AICore&lt;/h2&gt;
&lt;p&gt;The Pixel 10 Pro comes with Google&amp;#8217;s on-device Gemini Nano model, exposed to applications through Android AICore and the &lt;span class="caps"&gt;ML&lt;/span&gt; Kit Prompt &lt;span class="caps"&gt;API&lt;/span&gt;. Kollect&amp;nbsp;uses &lt;code&gt;com.google.mlkit:genai-prompt&lt;/code&gt; to send its existing metadata prompt to that local&amp;nbsp;model.&lt;/p&gt;
&lt;p&gt;There is no cloud model request in this mode. Kollect still downloads the web page, because it needs content to summarise, and it still uploads the completed entry to my own sync server. The actual language-model inference, however, stays on the phone. No page text or OpenRouter &lt;span class="caps"&gt;API&lt;/span&gt; key has to go to a third-party model provider, and there is no per-token&amp;nbsp;charge.&lt;/p&gt;
&lt;p&gt;The flow is deliberately visible rather than&amp;nbsp;magical:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Kollect asks AICore whether Gemini Nano is&amp;nbsp;available.&lt;/li&gt;
&lt;li&gt;If the model is downloadable but not installed, it requests the download and reports&amp;nbsp;progress.&lt;/li&gt;
&lt;li&gt;It fetches and cleans the web page, then asks the local model for one &lt;span class="caps"&gt;JSON&lt;/span&gt;&amp;nbsp;object.&lt;/li&gt;
&lt;li&gt;It validates that response, updates the entry, and records an audit trail with the provider, resolved model, completion time, status, and&amp;nbsp;duration.&lt;/li&gt;
&lt;li&gt;It uploads the enriched entry to the Kollect&amp;nbsp;server.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The screenshot at the top is a real result. A page published by Hannes Brecher was enriched in about 15.1 seconds. Kollect extracted five sensible tags, and the audit section&amp;nbsp;records &lt;code&gt;On-device (AICore)&lt;/code&gt; with the&amp;nbsp;resolved &lt;code&gt;nano-v3&lt;/code&gt; model. For this narrow structured-extraction task, the small local model works remarkably&amp;nbsp;well.&lt;/p&gt;
&lt;p&gt;There is one Android restriction which shapes the implementation: AICore inference must happen while Kollect is the top foreground application. WorkManager can upload queued entries and OpenRouter can enrich them in the background, but Android rejects the on-device generative-&lt;span class="caps"&gt;AI&lt;/span&gt; call from a background worker. In AICore mode, sharing a link brings Kollect to the foreground while enrichment runs. That is a platform constraint I am happy to accept for a roughly 15-second, private, effectively free model&amp;nbsp;call.&lt;/p&gt;
&lt;p&gt;Device support also matters. The Pixel 10 Pro supports this Prompt &lt;span class="caps"&gt;API&lt;/span&gt; path. My Fairphone 6 currently reports Gemini Nano as unavailable and is not on Google&amp;#8217;s supported-device list, so OpenRouter remains the practical fallback there. &amp;#8220;On-device &lt;span class="caps"&gt;AI&lt;/span&gt;&amp;#8221; is not yet a generic Android capability simply because a device runs Android&amp;nbsp;16.&lt;/p&gt;
&lt;h2 id="secrets-and-failure-records"&gt;Secrets and failure&amp;nbsp;records&lt;/h2&gt;
&lt;p&gt;The Android version protects both the Kollect server token and the optional OpenRouter key with &lt;span class="caps"&gt;AES&lt;/span&gt;-&lt;span class="caps"&gt;GCM&lt;/span&gt; keys held by Android Keystore. The on-device mode needs no model-provider credential at&amp;nbsp;all.&lt;/p&gt;
&lt;p&gt;I also made &lt;span class="caps"&gt;AI&lt;/span&gt; generation auditable rather than silently overwriting metadata. Every entry can&amp;nbsp;carry:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;the &lt;span class="caps"&gt;UTC&lt;/span&gt; generation&amp;nbsp;timestamp&lt;/li&gt;
&lt;li&gt;success or failure status and an error&amp;nbsp;message&lt;/li&gt;
&lt;li&gt;elapsed&amp;nbsp;time&lt;/li&gt;
&lt;li&gt;provider&amp;nbsp;(&lt;code&gt;openrouter&lt;/code&gt; or &lt;code&gt;aicore&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;the actual model&amp;nbsp;name&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Those are the fields visible in the screenshot. They are useful while developing, but they also answer a future question that otherwise becomes surprisingly difficult: &lt;em&gt;where did this summary come&amp;nbsp;from?&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Failures do not throw the link away. Kollect keeps the entry, records the error, and lets me retry. On-device generation has a five-minute timeout, model availability and download states produce explicit messages, and malformed model output remains an enrichment error rather than corrupting the&amp;nbsp;collection.&lt;/p&gt;
&lt;h2 id="what-this-setup-does-and-does-not-buy-me"&gt;What this setup does and does not buy&amp;nbsp;me&lt;/h2&gt;
&lt;p&gt;The largest benefit is not disk space. The Android &lt;span class="caps"&gt;SDK&lt;/span&gt;, Gradle caches, and emulator images consume essentially the same bytes wherever I put them. The benefit is a clean ownership&amp;nbsp;boundary.&lt;/p&gt;
&lt;p&gt;Everything Android-shaped belongs&amp;nbsp;to &lt;code&gt;kollect-android&lt;/code&gt;. If I stop mobile development, I can remove the Distrobox and its custom home without wondering which packages, environment variables, dotfiles, &lt;span class="caps"&gt;SDK&lt;/span&gt; revisions, and signing keys were added to Fedora over the previous year. If the toolchain gets into a strange state, recreating the environment is a contained operation rather than surgery on the&amp;nbsp;host.&lt;/p&gt;
&lt;p&gt;There are trade-offs. A stateful Distrobox is not a hermetic &lt;span class="caps"&gt;CI&lt;/span&gt; image, and this particular setup is documented rather than fully declarative. Hardware acceleration and &lt;span class="caps"&gt;GUI&lt;/span&gt; forwarding depend on host integration. The container shares the Fedora kernel, and, again, it must not be mistaken for a hostile-code&amp;nbsp;sandbox.&lt;/p&gt;
&lt;p&gt;For interactive development, those are good trade-offs. I get a normal shell, a normal Gradle workflow, wireless access to my phone, and a real emulator window. Fedora stays a workstation instead of becoming the accumulated history of every toolchain I have ever&amp;nbsp;tried.&lt;/p&gt;
&lt;p&gt;And the application produced by that container now uses the phone in exactly the same spirit: the capable hardware already in my pocket performs a small, well-defined job locally instead of sending it to a cloud service by default. The development stack is contained, the model is on-device, and the result remains ordinary data on a server I&amp;nbsp;control.&lt;/p&gt;
&lt;p&gt;That combination feels right for Kollect: a small application, a deliberately narrow &lt;span class="caps"&gt;AI&lt;/span&gt; feature, and no more infrastructure than the problem actually&amp;nbsp;needs.&lt;/p&gt;</content><category term="Linux"/><category term="android"/><category term="fedora"/><category term="distrobox"/><category term="podman"/><category term="kotlin"/><category term="gradle"/><category term="adb"/><category term="android emulator"/><category term="aicore"/><category term="gemini nano"/><category term="kollect"/></entry><entry><title>Kollect: The Native KDE Link Collector I Built for Myself</title><link href="https://hofstede.it/kollect-the-native-kde-link-collector-i-built-for-myself/" rel="alternate"/><published>2026-07-14T00:00:00+02:00</published><updated>2026-07-14T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-07-14:/kollect-the-native-kde-link-collector-i-built-for-myself/</id><summary type="html">&lt;p&gt;Kollect is a small native Plasma 6 application I wrote to get interesting links out of browser tabs and into a durable collection. It uses Kirigami and C++ on the desktop, plain &lt;span class="caps"&gt;TOML&lt;/span&gt; files by default, optional multi-user server storage, OpenRouter-powered metadata, KWallet for secrets, and a proper Fedora &lt;span class="caps"&gt;RPM&lt;/span&gt;. The source is not public, but the design is worth&amp;nbsp;documenting.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Kollect on KDE Plasma, showing links grouped into ISO week 2026w29 and the metadata detail view for a collected article" src="https://hofstede.it/images/2026-07-14-kollect-kde-link-collector.webp" title="Kollect is deliberately a desktop application: weekly link groups on the left, the selected article and its metadata on the right, and no browser pretending to be a window manager."&gt;&lt;/p&gt;
&lt;p&gt;I have a very predictable failure mode when reading the web. I open an interesting article, decide I will come back to it later, and leave the tab open. Then I do the same thing another twenty times. The tabs stop being a reading list and become archaeological&amp;nbsp;layers.&lt;/p&gt;
&lt;p&gt;Browser bookmarks never fixed this for me. They are good at preserving a &lt;span class="caps"&gt;URL&lt;/span&gt;, but a &lt;span class="caps"&gt;URL&lt;/span&gt; and a title are not enough to answer the question I actually have two weeks later: &lt;em&gt;why did I save this?&lt;/em&gt; Read-it-later services answer more of that question, but most of them turn a small personal collection into another account, another cloud database, and often another web&amp;nbsp;application.&lt;/p&gt;
&lt;p&gt;So I built &lt;strong&gt;Kollect&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Kollect is a small native &lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma 6 link collector. I paste a &lt;span class="caps"&gt;URL&lt;/span&gt;, and it immediately files the link into the current &lt;span class="caps"&gt;ISO&lt;/span&gt; week. In the background it fetches the page and asks an OpenRouter model to extract a useful title, publication date, author, tags, a primary topic, and a short summary. The result is a collection I can skim later instead of a graveyard of titles I no longer&amp;nbsp;recognise.&lt;/p&gt;
&lt;p&gt;This is not a product announcement. I have not published the application or its source, and there is no download link hiding at the end of this article. It is a personal 0.1 application built around my own workflow. But the design came together cleanly enough that it is worth documenting, especially because it combines several things I care about: a genuinely native Linux desktop, boring local files, optional self-hosting, secrets in KWallet, and distribution-native&amp;nbsp;packaging.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-using-it-looks-like"&gt;What using it looks&amp;nbsp;like&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-native-plasma-application-on-purpose"&gt;A native Plasma application, on&amp;nbsp;purpose&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#local-first-means-ordinary-files"&gt;Local-first means ordinary&amp;nbsp;files&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#enrichment-without-turning-the-app-into-a-chatbot"&gt;Enrichment without turning the app into a&amp;nbsp;chatbot&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#secrets-belong-in-kwallet"&gt;Secrets belong in&amp;nbsp;KWallet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#one-interface-two-storage-backends"&gt;One interface, two storage&amp;nbsp;backends&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-optional-server"&gt;The optional&amp;nbsp;server&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#packaged-as-a-fedora-application-not-a-build-directory"&gt;Packaged as a Fedora application, not a build&amp;nbsp;directory&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#things-i-deliberately-did-not-build"&gt;Things I deliberately did not&amp;nbsp;build&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-part-i-like-most"&gt;The part I like&amp;nbsp;most&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="what-using-it-looks-like"&gt;What using it looks&amp;nbsp;like&lt;/h2&gt;
&lt;p&gt;The main interaction is intentionally uneventful. Paste or type a link into the field and press Enter. That is&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;There are a few other paths because the best capture tool is the one that does not interrupt what I am&amp;nbsp;doing:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;Ctrl+Shift+V&lt;/code&gt; adds whatever &lt;span class="caps"&gt;URL&lt;/span&gt; is currently on the&amp;nbsp;clipboard.&lt;/li&gt;
&lt;li&gt;A &lt;span class="caps"&gt;URL&lt;/span&gt; can be dragged and dropped anywhere onto the&amp;nbsp;window.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;kollect https://example.net/article&lt;/code&gt; adds a &lt;span class="caps"&gt;URL&lt;/span&gt; passed on the command&amp;nbsp;line.&lt;/li&gt;
&lt;li&gt;The desktop file registers Kollect as an &lt;span class="caps"&gt;HTTP&lt;/span&gt; and &lt;span class="caps"&gt;HTTPS&lt;/span&gt; &lt;span class="caps"&gt;URL&lt;/span&gt; handler, so it can receive links from other desktop&amp;nbsp;applications.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The entry appears immediately with a pending indicator. Metadata fetching is asynchronous, so a slow website or &lt;span class="caps"&gt;API&lt;/span&gt; call does not freeze the interface. When enrichment completes, the raw &lt;span class="caps"&gt;URL&lt;/span&gt; turns into a recognisable headline with a topic and tags. If fetching fails, the entry remains in the collection, changes to an error state, and can be retried from the context&amp;nbsp;menu.&lt;/p&gt;
&lt;p&gt;Links are grouped into folders such&amp;nbsp;as &lt;code&gt;2026w29&lt;/code&gt;. The current week is expanded by default, older weeks are collapsed, and both weeks and entries are sorted newest first. Clicking an entry opens the detail page shown in the screenshot: the original &lt;span class="caps"&gt;URL&lt;/span&gt;, author, publication date, topic, tags, collection time, and summary. From there I can open the page, copy its &lt;span class="caps"&gt;URL&lt;/span&gt;, re-fetch metadata, or delete the&amp;nbsp;entry.&lt;/p&gt;
&lt;p&gt;That last part matters. Kollect does not try to become a browser. It remembers and organises links, then hands them back to the browser when I actually want to read&amp;nbsp;them.&lt;/p&gt;
&lt;h2 id="a-native-plasma-application-on-purpose"&gt;A native Plasma application, on&amp;nbsp;purpose&lt;/h2&gt;
&lt;p&gt;Kollect is not an Electron application, a local web server with a tray icon, or a Python &lt;span class="caps"&gt;GUI&lt;/span&gt; wearing a vaguely native stylesheet. The desktop client is C++20 with Qt 6, &lt;span class="caps"&gt;KDE&lt;/span&gt; Frameworks 6, &lt;span class="caps"&gt;QML&lt;/span&gt;, and&amp;nbsp;Kirigami.&lt;/p&gt;
&lt;p&gt;The executable owns the application logic and exposes&amp;nbsp;one &lt;code&gt;KollectApp&lt;/code&gt; object to &lt;span class="caps"&gt;QML&lt;/span&gt;. The interface is split into three small &lt;span class="caps"&gt;QML&lt;/span&gt;&amp;nbsp;files:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;Main.qml&lt;/code&gt; contains the application window, global drawer, capture field, weekly list, drag-and-drop area, and delete&amp;nbsp;confirmation.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;DetailPage.qml&lt;/code&gt; renders one selected entry and its&amp;nbsp;actions.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;SettingsDialog.qml&lt;/code&gt; configures OpenRouter and the optional sync&amp;nbsp;server.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Kirigami provides the application structure, actions, responsive pages, form layout, inline messages, passive notifications, spacing, and icon integration. Qt Quick Controls&amp;nbsp;uses &lt;code&gt;org.kde.desktop&lt;/code&gt;, so widgets follow the active Plasma style rather than drawing a separate visual universe inside the window. The application uses the system icon theme and standard &lt;span class="caps"&gt;KDE&lt;/span&gt; shortcuts such as the Preferences, Close, and Quit&amp;nbsp;keys.&lt;/p&gt;
&lt;p&gt;This buys more than appearance. The global drawer behaves like a &lt;span class="caps"&gt;KDE&lt;/span&gt; global drawer. Actions can move between the toolbar and overflow areas as space changes. Labels are selectable where that is useful. Notifications use the expected Plasma presentation. The result feels at home on my desktop because it is built out of the same primitives as the rest of the&amp;nbsp;desktop.&lt;/p&gt;
&lt;p&gt;The C++ side follows the usual Qt model/view&amp;nbsp;boundary. &lt;code&gt;LinkModel&lt;/code&gt; is&amp;nbsp;a &lt;code&gt;QAbstractListModel&lt;/code&gt; which presents both week headers and entries to the &lt;span class="caps"&gt;QML&lt;/span&gt; &lt;code&gt;ListView&lt;/code&gt;. Roles expose the title, &lt;span class="caps"&gt;URL&lt;/span&gt;, topic, tags, timestamp, state, and summary. A store&amp;nbsp;emits &lt;code&gt;entryAdded&lt;/code&gt;, &lt;code&gt;entryUpdated&lt;/code&gt;, &lt;code&gt;entryRemoved&lt;/code&gt;,&amp;nbsp;or &lt;code&gt;reloaded&lt;/code&gt;, the model rebuilds or updates the relevant row, and &lt;span class="caps"&gt;QML&lt;/span&gt; reacts. The view does not need to know whether that store is a directory on disk or an &lt;span class="caps"&gt;HTTP&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;That separation became the key to the entire&amp;nbsp;design.&lt;/p&gt;
&lt;h2 id="local-first-means-ordinary-files"&gt;Local-first means ordinary&amp;nbsp;files&lt;/h2&gt;
&lt;p&gt;The default backend&amp;nbsp;is &lt;code&gt;LocalLinkStore&lt;/code&gt;, and it stores the collection&amp;nbsp;here:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~/.local/share/kollect/
├── 2026w28/
│   └── example-org-an-interesting-article.toml
└── 2026w29/
    ├── vermaden-wordpress-com-gitlab-on-freebsd.toml
    └── lwn-net-articles.toml
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There is one directory per &lt;span class="caps"&gt;ISO&lt;/span&gt; week and one &lt;span class="caps"&gt;TOML&lt;/span&gt; document per link. A real entry looks roughly like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://vermaden.wordpress.com/2026/07/09/gitlab-on-freebsd/&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;added_at&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2026-07-14T16:38:21&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;done&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;headline&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;GitLab on FreeBSD&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;date&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2026-07-09&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;tags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;freebsd&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;gitlab&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;server&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;git&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;postgresql&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;nginx&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="n"&gt;topic&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;FreeBSD&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;author&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;vermaden&amp;quot;&lt;/span&gt;
&lt;span class="n"&gt;summary&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;A step-by-step guide to installing and configuring a self-hosted GitLab server on FreeBSD. The setup uses a dedicated bhyve VM and a PostgreSQL, Redis, Nginx, and GitLab-CE backend stack.&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is deliberately not a single opaque database. I can inspect the files&amp;nbsp;with &lt;code&gt;less&lt;/code&gt;, search them&amp;nbsp;with &lt;code&gt;rg&lt;/code&gt;, copy one entry, synchronise the directory with any file tool, put it in a private Git repository, or write a conversion script ten years from now without first reconstructing an abandoned application&amp;#8217;s database&amp;nbsp;schema.&lt;/p&gt;
&lt;p&gt;The file name is derived from the host and the last meaningful &lt;span class="caps"&gt;URL&lt;/span&gt; path segment, normalised to a lowercase slug and capped at 80 characters. If two links in a week produce the same slug, Kollect&amp;nbsp;adds &lt;code&gt;-2&lt;/code&gt;, &lt;code&gt;-3&lt;/code&gt;, and so on. Duplicate URLs are rejected across the whole collection before a file is&amp;nbsp;created.&lt;/p&gt;
&lt;p&gt;Writes go&amp;nbsp;through &lt;code&gt;QSaveFile&lt;/code&gt;. That gives the local backend an important property for almost no complexity: a new document is written and committed atomically instead of truncating the existing file in place. A crash during a metadata update should not leave half a &lt;span class="caps"&gt;TOML&lt;/span&gt; document behind. Deleting the final entry in a week also removes the now-empty week&amp;nbsp;directory.&lt;/p&gt;
&lt;p&gt;The state machine is tiny and visible in the&amp;nbsp;data:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;pending  -&amp;gt;  done
   |
   +----&amp;gt; error  -&amp;gt;  pending (manual re-fetch)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;URL&lt;/span&gt; is persisted&amp;nbsp;in &lt;code&gt;pending&lt;/code&gt; state before any network request begins. Metadata is helpful, but it is not allowed to decide whether the link itself&amp;nbsp;survives.&lt;/p&gt;
&lt;h2 id="enrichment-without-turning-the-app-into-a-chatbot"&gt;Enrichment without turning the app into a&amp;nbsp;chatbot&lt;/h2&gt;
&lt;p&gt;The &lt;span class="caps"&gt;AI&lt;/span&gt; part of Kollect is intentionally narrow. There is no chat panel and no prompt box. A model is used as a structured metadata extractor because web pages are inconsistent and because a useful two-to-four sentence reminder is hard to derive&amp;nbsp;from &lt;code&gt;&amp;lt;meta&amp;gt;&lt;/code&gt; tags&amp;nbsp;alone.&lt;/p&gt;
&lt;p&gt;For each queued entry, Kollect performs two network&amp;nbsp;operations:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;It fetches the page with Qt&amp;#8217;s network stack, follows redirects only when they do not downgrade security (Qt&amp;#8217;s NoLessSafeRedirectPolicy), and reads at most 2 MiB. A 30-second timeout covers both this request and the OpenRouter call that&amp;nbsp;follows.&lt;/li&gt;
&lt;li&gt;It removes scripts, styles, &lt;span class="caps"&gt;SVG&lt;/span&gt;, templates, comments, and &lt;span class="caps"&gt;HTML&lt;/span&gt; markup, decodes entities&amp;nbsp;with &lt;code&gt;QTextDocument&lt;/code&gt;, normalises whitespace, and keeps at most 14,000 characters of readable&amp;nbsp;text.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;That text and the &lt;span class="caps"&gt;URL&lt;/span&gt; go to OpenRouter&amp;#8217;s chat-completions endpoint. The request demands one &lt;span class="caps"&gt;JSON&lt;/span&gt; object with exactly the fields Kollect&amp;nbsp;understands: &lt;code&gt;headline&lt;/code&gt;, &lt;code&gt;date&lt;/code&gt;, &lt;code&gt;tags&lt;/code&gt;, &lt;code&gt;topic&lt;/code&gt;, &lt;code&gt;author&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;summary&lt;/code&gt;. In production I use the Chinese &lt;a href="https://openrouter.ai/minimax/minimax-m3"&gt;MiniMax M3&lt;/a&gt; model under the OpenRouter&amp;nbsp;identifier &lt;code&gt;minimax/minimax-m3&lt;/code&gt;. It is more than capable enough for website summaries and, as of July 14, 2026, costs $0.30 per million input tokens and $1.20 per million output tokens. The model is a normal setting and can be replaced without rebuilding the application. Kollect also requests &lt;span class="caps"&gt;JSON&lt;/span&gt; response formatting and defensively extracts the&amp;nbsp;outermost &lt;code&gt;{ ... }&lt;/code&gt; span because models occasionally wrap &lt;span class="caps"&gt;JSON&lt;/span&gt; in a Markdown fence despite being told not&amp;nbsp;to.&lt;/p&gt;
&lt;p&gt;Jobs run through a simple in-process queue, one at a time. That avoids a burst of page downloads and &lt;span class="caps"&gt;API&lt;/span&gt; calls when I paste several links in quick succession. It also makes error handling predictable: finish or fail one entry, then schedule the next job on the Qt event&amp;nbsp;loop.&lt;/p&gt;
&lt;p&gt;There is an obvious privacy boundary here which should not be hand-waved away. &lt;strong&gt;Local storage does not mean offline processing.&lt;/strong&gt; Enrichment sends the link and extracted page text to OpenRouter and whichever model provider handles the selected model. Anyone using this design should make that decision consciously and should not feed it private pages or sensitive intranet URLs. Without an &lt;span class="caps"&gt;API&lt;/span&gt; key, the link is still collected, but enrichment ends in an error&amp;nbsp;state.&lt;/p&gt;
&lt;h2 id="secrets-belong-in-kwallet"&gt;Secrets belong in&amp;nbsp;KWallet&lt;/h2&gt;
&lt;p&gt;Kollect needs two credentials: the OpenRouter &lt;span class="caps"&gt;API&lt;/span&gt; key and, when server mode is enabled, a bearer token. On Plasma, both belong in&amp;nbsp;KWallet.&lt;/p&gt;
&lt;p&gt;The client talks directly&amp;nbsp;to &lt;code&gt;org.kde.kwalletd6&lt;/code&gt; over D-Bus, opens the user&amp;#8217;s network wallet asynchronously since that call may display an unlock dialog, and stores passwords under a&amp;nbsp;dedicated &lt;code&gt;Kollect&lt;/code&gt; folder. Loading is additionally deferred until the Qt event loop is running, so no secret can be reported before the rest of the application has connected to the corresponding signals. Metadata jobs and remote requests wait until the corresponding secret has been resolved instead of racing application&amp;nbsp;startup.&lt;/p&gt;
&lt;p&gt;Non-secret settings live&amp;nbsp;in &lt;code&gt;~/.config/kollectrc&lt;/code&gt; through KConfig: model name, server &lt;span class="caps"&gt;URL&lt;/span&gt;, username, and whether the remote backend is enabled. An older &lt;span class="caps"&gt;API&lt;/span&gt; key found in that config is migrated into KWallet the first time a wallet becomes available and then removed from the config&amp;nbsp;file.&lt;/p&gt;
&lt;p&gt;There are pragmatic fallbacks. If KWallet is unavailable, secrets can be stored in the config file, and the OpenRouter key can come&amp;nbsp;from &lt;code&gt;OPENROUTER_API_KEY&lt;/code&gt;. That is useful in unusual sessions and during development, but KWallet is the intended path on a normal Plasma&amp;nbsp;desktop.&lt;/p&gt;
&lt;h2 id="one-interface-two-storage-backends"&gt;One interface, two storage&amp;nbsp;backends&lt;/h2&gt;
&lt;p&gt;I originally needed Kollect on one machine, so local &lt;span class="caps"&gt;TOML&lt;/span&gt; was the right starting point. Then came the predictable next requirement: the same collection on more than one&amp;nbsp;workstation.&lt;/p&gt;
&lt;p&gt;I did not want to destroy the local design by making the server mandatory.&amp;nbsp;Instead, &lt;code&gt;LinkStore&lt;/code&gt; defines the operations the rest of the application needs, while two implementations provide&amp;nbsp;persistence:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;QML  &amp;lt;-&amp;gt;  LinkModel  &amp;lt;-&amp;gt;  KollectApp
                \             /
                 \           /
                  LinkStore
                 /         \
        LocalLinkStore   RemoteLinkStore
           TOML files       JSON/HTTP
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;At&amp;nbsp;startup, &lt;code&gt;KollectApp&lt;/code&gt; reads the configuration and constructs exactly one backend. Everything above it stays the same. Adding an entry, updating metadata, deleting it, detecting duplicates, sorting weeks, and feeding the view all use the shared&amp;nbsp;abstraction.&lt;/p&gt;
&lt;p&gt;The remote backend is deliberately small. Once a token is available, it loads the user&amp;#8217;s complete collection&amp;nbsp;with &lt;code&gt;GET /entries&lt;/code&gt;. An entry update is a&amp;nbsp;full-document &lt;code&gt;PUT&lt;/code&gt; to &lt;code&gt;/entries/{week}/{entry_id}&lt;/code&gt;, and deletion uses the&amp;nbsp;corresponding &lt;code&gt;DELETE&lt;/code&gt;. The &lt;span class="caps"&gt;JSON&lt;/span&gt; representation carries the same fields as the &lt;span class="caps"&gt;TOML&lt;/span&gt; representation, so switching storage does not change what an entry&amp;nbsp;means.&lt;/p&gt;
&lt;p&gt;Server mode also has a one-time &lt;strong&gt;Upload Local Collection&lt;/strong&gt; action. It reads the existing &lt;span class="caps"&gt;TOML&lt;/span&gt; backend, skips URLs or IDs already present remotely, and pushes the rest. Local files are left&amp;nbsp;untouched.&lt;/p&gt;
&lt;p&gt;It is important to call this what it is: &lt;strong&gt;an alternative remote store, not bidirectional offline synchronisation&lt;/strong&gt;. In server mode the server is the source of truth and the client needs a network connection. There is no conflict-resolution algorithm, offline write journal, or background merge between &lt;span class="caps"&gt;TOML&lt;/span&gt; and SQLite. Building those features would be possible, but they would turn a clean storage switch into a distributed-systems project. For my use, an honest online backend is the better&amp;nbsp;tradeoff.&lt;/p&gt;
&lt;h2 id="the-optional-server"&gt;The optional&amp;nbsp;server&lt;/h2&gt;
&lt;p&gt;The server is a separate, compact Python application built with FastAPI, SQLAlchemy 2, Pydantic, and SQLite. Its complete public surface is five&amp;nbsp;operations:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Method&lt;/th&gt;
&lt;th&gt;Path&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;POST&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/auth/login&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Exchange a username and password for a bearer token&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;GET&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/entries&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Return the authenticated user&amp;#8217;s collection&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;PUT&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/entries/{week}/{entry_id}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Create or replace a complete entry&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;DELETE&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/entries/{week}/{entry_id}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Delete one entry&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;GET&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;/healthz&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Liveness check without authentication&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;User passwords are hashed with Argon2. Login creates a random 32-byte &lt;span class="caps"&gt;URL&lt;/span&gt;-safe token, returns it once, and stores only its &lt;span class="caps"&gt;SHA&lt;/span&gt;-256 hash in the database. A leaked SQLite file therefore does not directly reveal reusable bearer tokens. Every entry is keyed by user, week, and entry &lt;span class="caps"&gt;ID&lt;/span&gt;, and a uniqueness constraint prevents the same user from collecting one &lt;span class="caps"&gt;URL&lt;/span&gt; under multiple IDs. Different users are isolated and may independently save the same &lt;span class="caps"&gt;URL&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;Administration is intentionally &lt;span class="caps"&gt;CLI&lt;/span&gt;-only. The server package provides commands to create users, change passwords, and list accounts. I do not need an admin web interface for a private service with a handful of users, and not building one means there is no admin web interface to&amp;nbsp;secure.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;API&lt;/span&gt; tests cover the parts that would hurt if they broke: failed login, missing and invalid tokens, the full &lt;span class="caps"&gt;CRUD&lt;/span&gt; round trip, duplicate &lt;span class="caps"&gt;URL&lt;/span&gt; conflicts, and isolation between two&amp;nbsp;users.&lt;/p&gt;
&lt;p&gt;Deployment is equally unsurprising. A&amp;nbsp;small &lt;code&gt;python:3.12-slim&lt;/code&gt; image installs the server package, runs as &lt;span class="caps"&gt;UID&lt;/span&gt; 1000, and keeps SQLite&amp;nbsp;under &lt;code&gt;/data&lt;/code&gt;. A rootless Podman Quadlet binds it&amp;nbsp;to &lt;code&gt;127.0.0.1:8000&lt;/code&gt;, stores data in a named volume, restarts on failure, and can start with the user&amp;#8217;s systemd session. If I expose it beyond localhost, it goes behind a &lt;span class="caps"&gt;TLS&lt;/span&gt;-terminating reverse proxy because the bearer token is an &lt;span class="caps"&gt;HTTP&lt;/span&gt; header, not&amp;nbsp;magic.&lt;/p&gt;
&lt;h2 id="packaged-as-a-fedora-application-not-a-build-directory"&gt;Packaged as a Fedora application, not a build&amp;nbsp;directory&lt;/h2&gt;
&lt;p&gt;I run Fedora on my Plasma desktops, so Kollect is packaged as an &lt;span class="caps"&gt;RPM&lt;/span&gt;. This sounds like a finishing detail, but it changes how an application feels in daily use. A program launched out&amp;nbsp;of &lt;code&gt;~/src/project/build/&lt;/code&gt; is still a project. A program installed through &lt;span class="caps"&gt;RPM&lt;/span&gt;, visible in the application launcher, tracked by the package database, and removable cleanly is part of the&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;The package uses Fedora&amp;#8217;s CMake macros and declares the actual build&amp;nbsp;stack:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;CMake, &lt;span class="caps"&gt;GCC&lt;/span&gt; C++, and Extra CMake&amp;nbsp;Modules&lt;/li&gt;
&lt;li&gt;Qt 6 base and declarative development&amp;nbsp;packages&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;KDE&lt;/span&gt; Frameworks 6 KConfig, KCoreAddons, and KI18n development&amp;nbsp;packages&lt;/li&gt;
&lt;li&gt;Kirigami&amp;nbsp;and &lt;code&gt;qqc2-desktop-style&lt;/code&gt; as explicit &lt;span class="caps"&gt;QML&lt;/span&gt; runtime&amp;nbsp;requirements&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That last point is easy to miss. &lt;span class="caps"&gt;RPM&lt;/span&gt; dependency generation sees linked &lt;span class="caps"&gt;ELF&lt;/span&gt; libraries rather well, but &lt;span class="caps"&gt;QML&lt;/span&gt; imports loaded at runtime are not always inferred. Declaring Kirigami and the desktop Qt Quick style explicitly prevents the classic result where the package installs successfully and the application immediately fails because a &lt;span class="caps"&gt;QML&lt;/span&gt; module is&amp;nbsp;absent.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;RPM&lt;/span&gt; installs&amp;nbsp;the &lt;code&gt;kollect&lt;/code&gt; executable&amp;nbsp;and &lt;code&gt;org.kde.kollect.desktop&lt;/code&gt;, then validates the desktop file&amp;nbsp;during &lt;code&gt;%check&lt;/code&gt;. The build helper derives the version from the spec, creates a clean source archive from the current&amp;nbsp;Git &lt;code&gt;HEAD&lt;/code&gt;, and hands that tarball&amp;nbsp;to &lt;code&gt;rpmbuild&lt;/code&gt;. The package is therefore reproducible from committed source rather than whatever untracked experiment happens to be lying in the&amp;nbsp;worktree.&lt;/p&gt;
&lt;p&gt;For development, the underlying build remains&amp;nbsp;conventional:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;cmake&lt;span class="w"&gt; &lt;/span&gt;-B&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-DCMAKE_BUILD_TYPE&lt;span class="o"&gt;=&lt;/span&gt;RelWithDebInfo
cmake&lt;span class="w"&gt; &lt;/span&gt;--build&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;nproc&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
./build/kollect
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For daily use, I install the &lt;span class="caps"&gt;RPM&lt;/span&gt;. &lt;span class="caps"&gt;KDE&lt;/span&gt; gets a proper desktop&amp;nbsp;entry, &lt;code&gt;%u&lt;/code&gt; &lt;span class="caps"&gt;URL&lt;/span&gt; handling, the standard bookmarks icon from the current icon theme, and a binary under the normal system&amp;nbsp;prefix.&lt;/p&gt;
&lt;h2 id="things-i-deliberately-did-not-build"&gt;Things I deliberately did not&amp;nbsp;build&lt;/h2&gt;
&lt;p&gt;Small applications stay pleasant when they are allowed to have boundaries. Kollect 0.1 does not have browser extensions, full-text article archiving, reading progress, a mobile client, collaborative lists, server-side metadata workers, or magical offline conflict&amp;nbsp;resolution.&lt;/p&gt;
&lt;p&gt;It also does not currently extract metadata locally. The model call is useful, but it introduces cost, network dependence, and the privacy boundary described above. A future local model or deterministic extractor could fit behind the&amp;nbsp;existing &lt;code&gt;MetadataFetcher&lt;/code&gt; interface, but I would only add it when it improves the actual workflow rather than the feature&amp;nbsp;list.&lt;/p&gt;
&lt;p&gt;The server is optional because a server should remain optional for a link collector. The &lt;span class="caps"&gt;TOML&lt;/span&gt; backend is not a demo mode that exists until a user signs up. It is the simplest complete version of the application and the one with the best long-term ownership story. Remote storage solves a real multi-machine problem, but it is allowed to remain an&amp;nbsp;add-on.&lt;/p&gt;
&lt;h2 id="the-part-i-like-most"&gt;The part I like&amp;nbsp;most&lt;/h2&gt;
&lt;p&gt;Kollect is not technically enormous. That is exactly why I like&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;It takes one recurring irritation, too many tabs and too little context, and gives it a narrow desktop-shaped solution. The interface is native to Plasma. The default data format is readable without Kollect. Writes are atomic. Secrets use the desktop wallet. The &lt;span class="caps"&gt;AI&lt;/span&gt; model has one constrained job. The remote service can be self-hosted, but the application does not require it. Fedora gets a real package rather than&amp;nbsp;a &lt;code&gt;curl | bash&lt;/code&gt; installer.&lt;/p&gt;
&lt;p&gt;Most importantly, I know where the data is and what every moving part does. If I stop maintaining the client tomorrow, the local collection remains a directory of &lt;span class="caps"&gt;TOML&lt;/span&gt; files. If I stop running the server, SQLite remains a documented database full of ordinary rows. There is no export button I need to trust and no service whose business model has to survive longer than my&amp;nbsp;bookmarks.&lt;/p&gt;
&lt;p&gt;That is the kind of personal software I want more of: small, native, inspectable, and built to solve the problem in front of me without trying to own everything around&amp;nbsp;it.&lt;/p&gt;</content><category term="Linux"/><category term="kde"/><category term="plasma"/><category term="kirigami"/><category term="qt"/><category term="cpp"/><category term="fedora"/><category term="rpm"/><category term="fastapi"/><category term="sqlite"/><category term="openrouter"/><category term="self-hosting"/></entry><entry><title>bcachefs on RHEL 10.2: The Kernel That Said No</title><link href="https://hofstede.it/bcachefs-on-rhel-102-the-kernel-that-said-no/" rel="alternate"/><published>2026-06-01T00:00:00+02:00</published><updated>2026-06-01T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-06-01:/bcachefs-on-rhel-102-the-kernel-that-said-no/</id><summary type="html">&lt;p&gt;A few months ago I bullied &lt;span class="caps"&gt;RHEL&lt;/span&gt; onto a &lt;span class="caps"&gt;ZFS&lt;/span&gt; root and it worked. This time I tried to get bcachefs running on &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10, lost a whole evening to compile errors, and the kernel won. Here is the autopsy, plus what bcachefs actually is and why an out-of-tree filesystem and a conservative enterprise kernel are a terrible&amp;nbsp;couple.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;A while back I &lt;a href="https://hofstede.it/rhel-on-zfs-root-an-unholy-experiment/"&gt;put &lt;span class="caps"&gt;RHEL&lt;/span&gt; on a &lt;span class="caps"&gt;ZFS&lt;/span&gt; root&lt;/a&gt;, declared it cursed, and the homelab &lt;span class="caps"&gt;VM&lt;/span&gt; lived to tell the tale. That one &lt;em&gt;worked&lt;/em&gt;, which is the dangerous kind of outcome, because it teaches you the wrong lesson. The wrong lesson is &amp;#8220;the kernel does not care what filesystem holds root, so anything is possible if you disable enough&amp;nbsp;checks.&amp;#8221;&lt;/p&gt;
&lt;p&gt;This is the sequel where the kernel reminds me that it absolutely does care, thank you very much, and that &amp;#8220;anything is possible&amp;#8221; has an asterisk the size of a merge&amp;nbsp;window.&lt;/p&gt;
&lt;p&gt;This time the target was &lt;a href="https://bcachefs.org/"&gt;bcachefs&lt;/a&gt;. The plan was simple: get bcachefs running on a fresh &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 box, write a smug little post about subvolumes and snapshots, go to bed. What I got instead was a four hour speedrun through every layer of the out-of-tree kernel module experience, a defeat so total I had to switch distributions, and this post, which is less &amp;#8220;how to&amp;#8221; and more&amp;nbsp;&amp;#8220;post-mortem.&amp;#8221;&lt;/p&gt;
&lt;p&gt;Let me be clear from the top, in the grand tradition of these articles: &lt;strong&gt;this was a bad idea.&lt;/strong&gt; Not &amp;#8220;edgy homelab fun&amp;#8221; bad like the &lt;span class="caps"&gt;ZFS&lt;/span&gt; thing, which at least booted. This was &amp;#8220;fundamentally trying to bolt a filesystem that tracks the bleeding edge of mainline onto a distribution whose entire value proposition is that it never moves&amp;#8221; bad. I knew that going in. I did it anyway. I tried &lt;span class="caps"&gt;REALLY&lt;/span&gt; hard. The kernel said&amp;nbsp;no.&lt;/p&gt;
&lt;h2 id="what-bcachefs-is-and-why-youd-want-it"&gt;What bcachefs Is, and Why You&amp;#8217;d Want&amp;nbsp;It&lt;/h2&gt;
&lt;p&gt;Before the carnage, the pitch, because bcachefs genuinely is&amp;nbsp;interesting.&lt;/p&gt;
&lt;p&gt;bcachefs grew out&amp;nbsp;of &lt;code&gt;bcache&lt;/code&gt;, the block-layer &lt;span class="caps"&gt;SSD&lt;/span&gt; cache that has been in the Linux kernel for over a decade. Kent Overstreet took the same B-tree machinery and grew it upward into a full copy-on-write filesystem. The selling point is that it folds the features people reach for &lt;span class="caps"&gt;ZFS&lt;/span&gt; and Btrfs to get into a single, more uniform&amp;nbsp;design:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Subvolumes and snapshots&lt;/strong&gt; that are cheap and first-class, not bolted&amp;nbsp;on&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Transparent compression&lt;/strong&gt; (&lt;span class="caps"&gt;LZ4&lt;/span&gt;, gzip, zstd) per file or per&amp;nbsp;filesystem&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Copy-on-write&lt;/strong&gt; for atomic updates and consistent&amp;nbsp;snapshots&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Multi-device placement, replication, and caching/tiering-oriented designs&lt;/strong&gt; for combining fast and slow storage in one&amp;nbsp;filesystem&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Full data and metadata checksumming&lt;/strong&gt;, with optional&amp;nbsp;encryption&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The part worth emphasizing is &lt;em&gt;how&lt;/em&gt; it tries to deliver these. bcachefs combines subvolumes, snapshots, compression, and copy-on-write in a design that is meant to stay fast under heavy load. Sustained performance under write pressure is a sensitive comparison point: snapshot-heavy, write-heavy Btrfs workloads have historically produced enough fragmentation and write-amplification complaints to make it one. bcachefs was built from its caching roots to keep the allocator and the B-trees healthy when the I/O does not stop. Whether it fully delivers on that promise is a topic for benchmarks and not for a homelab &lt;span class="caps"&gt;VM&lt;/span&gt; that never got far enough to write a single byte, but that is the ambition, and it is a good&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;So why is it such a nightmare to install on &lt;span class="caps"&gt;RHEL&lt;/span&gt;? For that you need the&amp;nbsp;politics.&lt;/p&gt;
&lt;h2 id="the-current-situation-bcachefs-lives-outside-the-tree"&gt;The Current Situation: bcachefs Lives Outside the&amp;nbsp;Tree&lt;/h2&gt;
&lt;p&gt;bcachefs was merged into the mainline Linux kernel in 6.7, in early 2024. For a while you could build a sufficiently recent kernel&amp;nbsp;with &lt;code&gt;CONFIG_BCACHEFS_FS&lt;/code&gt; and go. That particular convenience did not&amp;nbsp;last.&lt;/p&gt;
&lt;p&gt;What preceded the end was a long, increasingly public series of disagreements between Kent Overstreet and Linus Torvalds, mostly about merge-window discipline. Linus runs the kernel on a strict rhythm: features land during the two-week merge window, and after that you send bug fixes, not new code. bcachefs is young and moving fast, and Kent repeatedly wanted to push substantial changes during the stabilization phase, arguing that a young filesystem needs its fixes in users&amp;#8217; hands now. Linus disagreed, loudly, more than&amp;nbsp;once.&lt;/p&gt;
&lt;p&gt;So bcachefs was marked externally maintained in Linux 6.17. In Linux 6.18, its in-tree code was removed entirely: ongoing development had moved to an external &lt;span class="caps"&gt;DKMS&lt;/span&gt; module, and keeping a stale copy inside the kernel would only create version&amp;nbsp;confusion.&lt;/p&gt;
&lt;p&gt;That last fact is the whole problem. &amp;#8220;Build against whatever kernel you happen to be running&amp;#8221; sounds wonderfully portable until you remember that Linux does not promise a stable internal &lt;span class="caps"&gt;API&lt;/span&gt; for arbitrary out-of-tree filesystems. Helpers get renamed, signatures change, structures get reorganized, and code inside the kernel tree is updated alongside those changes in the same commit so nobody notices. External code gets to discover them at compile time, usually at the least convenient possible&amp;nbsp;hour.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 is built on a 6.12 kernel line. bcachefs &lt;span class="caps"&gt;DKMS&lt;/span&gt; development is chasing much newer upstream internals. You can see where this is&amp;nbsp;going.&lt;/p&gt;
&lt;h2 id="the-patient"&gt;The&amp;nbsp;Patient&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@testvm2 ~]# uname -a
Linux testvm2 6.12.0-211.18.1.el10_2.x86_64 #1 SMP PREEMPT_DYNAMIC ... x86_64 GNU/Linux

[root@testvm2 ~]# cat /etc/os-release
PRETTY_NAME=&amp;quot;Red Hat Enterprise Linux 10.2 (Coughlan)&amp;quot;
RELEASE_TYPE=stable
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A clean, fully subscribed &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;10.2. &lt;code&gt;RELEASE_TYPE=stable&lt;/code&gt;. The system is, at this moment, working perfectly. Remember&amp;nbsp;that.&lt;/p&gt;
&lt;p&gt;There is&amp;nbsp;no &lt;code&gt;bcachefs-tools&lt;/code&gt; in the &lt;span class="caps"&gt;RHEL&lt;/span&gt; or &lt;span class="caps"&gt;EPEL&lt;/span&gt; repositories, which is the first hint from the universe. There is a &lt;a href="https://copr.fedorainfracloud.org/coprs/ngompa/bcachefs/"&gt;Copr repo&lt;/a&gt;&amp;nbsp;(&lt;code&gt;ngompa/bcachefs&lt;/code&gt;) that packages it nicely, but Copr builds per Fedora release. It&amp;nbsp;has &lt;code&gt;fedora-41&lt;/code&gt;, &lt;code&gt;fedora-44&lt;/code&gt;, &lt;code&gt;rawhide&lt;/code&gt;. It does not&amp;nbsp;have &lt;code&gt;epel-10&lt;/code&gt;, because nobody sane builds bcachefs for an enterprise&amp;nbsp;kernel.&lt;/p&gt;
&lt;p&gt;I am not sane on Friday evenings, so I pointed &lt;span class="caps"&gt;RHEL&lt;/span&gt; at the Fedora 41 packages&amp;nbsp;anyway:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;copr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;ngompa/bcachefs&lt;span class="w"&gt; &lt;/span&gt;fedora-41-x86_64
sed&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;s/$releasever/41/g&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/etc/yum.repos.d/*ngompa*bcachefs*.repo
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;First dependency&amp;nbsp;wall: &lt;code&gt;nothing provides libsodium.so.26()(64bit)&lt;/code&gt;. That is a Fedora soname that &lt;span class="caps"&gt;RHEL&lt;/span&gt; does not ship, so on went &lt;span class="caps"&gt;EPEL&lt;/span&gt; to satisfy it. Already I am stitching together three distributions&amp;#8217; worth of packages, which should have been the moment I stopped. It was&amp;nbsp;not.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;bcachefs-tools
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This pulls&amp;nbsp;in &lt;code&gt;bcachefs-tools-1.32.1-1.fc41&lt;/code&gt; and &lt;code&gt;dkms-bcachefs-1.32.1-1.fc41&lt;/code&gt;. Fedora 41 packages, on a &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 kernel, built by &lt;span class="caps"&gt;DKMS&lt;/span&gt; on the fly. What could go&amp;nbsp;wrong.&lt;/p&gt;
&lt;h2 id="why-it-failed"&gt;Why It&amp;nbsp;Failed&lt;/h2&gt;
&lt;p&gt;Everything. Specifically, three things, each one a complete reason on its&amp;nbsp;own.&lt;/p&gt;
&lt;h3 id="1-api-drift-or-whack-a-mole-with-the-kernel"&gt;1. &lt;span class="caps"&gt;API&lt;/span&gt; Drift, or: Whack-a-Mole With the&amp;nbsp;Kernel&lt;/h3&gt;
&lt;p&gt;The &lt;span class="caps"&gt;DKMS&lt;/span&gt; post-install build did not even try at first. The package ships&amp;nbsp;a &lt;code&gt;BUILD_EXCLUSIVE&lt;/code&gt; directive&amp;nbsp;in &lt;code&gt;dkms.conf&lt;/code&gt;, which is the maintainer&amp;#8217;s polite way of saying &amp;#8220;do not build me on a kernel like this one.&amp;#8221; The package &lt;em&gt;knows&lt;/em&gt;. I commented it out, because of course I&amp;nbsp;did:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;s/.*BUILD_EXCLUSIVE.*/#&amp;amp;/g&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/usr/src/bcachefs-1.32.1/dkms.conf
dkms&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;bcachefs&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.32.1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And then I met the kernel&amp;#8217;s internal &lt;span class="caps"&gt;API&lt;/span&gt;, one missing symbol at a&amp;nbsp;time:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;util.h:643:9: error: implicit declaration of function &amp;#39;TYPEOF_UNQUAL&amp;#39;
util.h:692:16: error: implicit declaration of function &amp;#39;cmp_int&amp;#39;; did you mean &amp;#39;smp_init&amp;#39;?
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;TYPEOF_UNQUAL&lt;/code&gt; and &lt;code&gt;cmp_int&lt;/code&gt; are helpers that exist in newer kernels and not in &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s 6.12. Fine, I thought, I will just define them&amp;nbsp;myself:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sed&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;...#ifndef TYPEOF_UNQUAL\n#define TYPEOF_UNQUAL(x) __typeof_unqual__(x)\n#endif...&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/usr/src/bcachefs-1.32.1/src/fs/bcachefs/util/util.h
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Build again. New error, deeper&amp;nbsp;in:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;journal_overlay.c:737: error: implicit declaration of function &amp;#39;sort_r_nonatomic&amp;#39;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Patch that. Build&amp;nbsp;again:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;node_scan.c:398: error: implicit declaration of function &amp;#39;sort_nonatomic&amp;#39;;
                 did you mean &amp;#39;sort_r_nonatomic&amp;#39;?
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is the &lt;span class="caps"&gt;API&lt;/span&gt; drift treadmill. Every fix reveals the next missing symbol, because the module was written against a kernel that has these helpers and &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s was branched before they existed. I was not porting bcachefs to &lt;span class="caps"&gt;RHEL&lt;/span&gt;. I was reimplementing six months of mainline kernel evolution by hand,&amp;nbsp;in &lt;code&gt;sed&lt;/code&gt;, at&amp;nbsp;8pm.&lt;/p&gt;
&lt;p&gt;Then I hit the wall that ended this branch of the&amp;nbsp;attempt:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;checksum.c: error: passing argument 1 of &amp;#39;chacha_init&amp;#39; from incompatible pointer type
  chacha.h: note: expected &amp;#39;u32 *&amp;#39; but argument is of type &amp;#39;struct chacha_state *&amp;#39;
checksum.c: error: storage size of &amp;#39;chacha_state&amp;#39; isn&amp;#39;t known
checksum.c: error: implicit declaration of function &amp;#39;chacha_zeroize_state&amp;#39;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is not a missing helper. The kernel&amp;#8217;s ChaCha crypto library was &lt;em&gt;redesigned&lt;/em&gt;. The new &lt;span class="caps"&gt;API&lt;/span&gt; takes&amp;nbsp;a &lt;code&gt;struct chacha_state&lt;/code&gt;; &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s 6.12 still has the old one that takes a&amp;nbsp;bare &lt;code&gt;u32 *&lt;/code&gt;. bcachefs 1.32 expects the new shape. You&amp;nbsp;cannot &lt;code&gt;#define&lt;/code&gt; your way around a struct that does not exist. This is a genuine &lt;span class="caps"&gt;ABI&lt;/span&gt; mismatch between two points in kernel history, and no amount of header forgery fixes it. The frozen enterprise kernel and the bleeding-edge module had drifted past the point of&amp;nbsp;reconciliation.&lt;/p&gt;
&lt;h3 id="2-dependency-hell"&gt;2. Dependency&amp;nbsp;Hell&lt;/h3&gt;
&lt;p&gt;Step back and count what I was already juggling: a Fedora 41 userspace tool, a Fedora 41 &lt;span class="caps"&gt;DKMS&lt;/span&gt; module, an &lt;span class="caps"&gt;EPEL&lt;/span&gt; &lt;code&gt;libsodium&lt;/code&gt;, all running on a &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 kernel, with the package&amp;#8217;s own &amp;#8220;do not build here&amp;#8221; guard rail sawn off. None of these components agreed on what kernel they were targeting.&amp;nbsp;The &lt;code&gt;$releasever&lt;/code&gt; rewrite, the soname mismatch,&amp;nbsp;the &lt;code&gt;BUILD_EXCLUSIVE&lt;/code&gt; override: each one is a small lie I told the package manager so it would keep going. Dependency resolution is the one part of an enterprise distro that is supposed to protect you from exactly this, and I had spent the evening systematically defeating&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="3-kernel-version-mismatch-the-hail-mary"&gt;3. Kernel Version Mismatch (the Hail&amp;nbsp;Mary)&lt;/h3&gt;
&lt;p&gt;If the problem is &amp;#8220;&lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s kernel is too old for this module,&amp;#8221; the obvious fix is a newer kernel. So I went to ELRepo and&amp;nbsp;pulled &lt;code&gt;kernel-ml&lt;/code&gt;, a mainline build, version&amp;nbsp;7.0.10:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;--enablerepo&lt;span class="o"&gt;=&lt;/span&gt;elrepo-kernel&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;kernel-ml&lt;span class="w"&gt; &lt;/span&gt;kernel-ml-devel&lt;span class="w"&gt; &lt;/span&gt;kernel-ml-headers
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Which immediately produced its own genre of pain. ELRepo&amp;#8217;s mirrorlist timed out (switch&amp;nbsp;to &lt;code&gt;baseurl&lt;/code&gt;).&amp;nbsp;Then &lt;code&gt;kernel-ml-headers&lt;/code&gt; conflicts with the&amp;nbsp;stock &lt;code&gt;kernel-headers&lt;/code&gt; on hundreds of&amp;nbsp;files:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;file /usr/include/linux/fs.h from install of kernel-ml-headers-7.0.10-1.el10.elrepo
  conflicts with file from package kernel-headers-6.12.0-211.18.1.el10_2
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&amp;#8230;repeated for what felt like every header in the tree. Removing the stock kernel headers and development packages on a running enterprise box to shoehorn in mainline ones is precisely the kind of thing the&amp;nbsp;protected &lt;code&gt;main&lt;/code&gt; branch of life is supposed to prevent. I did it anyway, booted into 7.0.10, and rebuilt bcachefs full of&amp;nbsp;hope.&lt;/p&gt;
&lt;p&gt;The hope lasted exactly one compile unit longer than&amp;nbsp;before:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;printbuf.h:250: error: implicit declaration of function &amp;#39;hex_asc_hi&amp;#39;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Add &lt;code&gt;#include &amp;lt;linux/hex.h&amp;gt;&lt;/code&gt;. Build again. It got &lt;em&gt;much&lt;/em&gt; further this time, dozens of object files, and&amp;nbsp;then:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;compress.c:626: error: implicit declaration of function &amp;#39;mempool_init_kvmalloc_pool&amp;#39;;
                did you mean &amp;#39;mempool_init_kmalloc_pool&amp;#39;?
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Even on a 7.x kernel, the packaged 1.32.1 source still referenced helpers this particular build did not expose (with a&amp;nbsp;quiet &lt;code&gt;pahole version differs&lt;/code&gt; warning alongside it, the debug-info tooling waving a small flag that this build environment was increasingly improvised). &amp;#8220;Newer&amp;#8221; was not the same as &amp;#8220;matching.&amp;#8221; The &lt;span class="caps"&gt;DKMS&lt;/span&gt; source expected a particular range of upstream kernel internals, and ELRepo&amp;#8217;s mainline rebuild was close but still outside that range. Close does not&amp;nbsp;compile.&lt;/p&gt;
&lt;p&gt;That was it. That was the&amp;nbsp;evening.&lt;/p&gt;
&lt;h2 id="the-surrender-and-the-two-minute-victory"&gt;The Surrender, and the Two-Minute&amp;nbsp;Victory&lt;/h2&gt;
&lt;p&gt;I gave up on &lt;span class="caps"&gt;RHEL&lt;/span&gt;. Officially. Honorably. I had patched headers, forged macros, swapped kernels, and erased system packages, and the score was Kernel 1, Christian&amp;nbsp;0.&lt;/p&gt;
&lt;p&gt;So I spun up a Fedora 44 &lt;span class="caps"&gt;VM&lt;/span&gt;, which is the distribution this software is actually built and tested for, and did the boring&amp;nbsp;version:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;copr&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;ngompa/bcachefs
dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;bcachefs-tools
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The Copr repo has&amp;nbsp;a &lt;code&gt;fedora-44&lt;/code&gt; branch, so there is&amp;nbsp;no &lt;code&gt;$releasever&lt;/code&gt; surgery. The package it installs&amp;nbsp;is &lt;code&gt;bcachefs-tools-1.38.3&lt;/code&gt; with a&amp;nbsp;matching &lt;code&gt;dkms-bcachefs-1.38.3&lt;/code&gt;, packaged for Fedora 44 and successfully built against the kernel my Fedora &lt;span class="caps"&gt;VM&lt;/span&gt; was actually running&amp;nbsp;(&lt;code&gt;7.0.10-201.fc44&lt;/code&gt;). Matched versions. No drift. The &lt;span class="caps"&gt;DKMS&lt;/span&gt; build completed cleanly,&amp;nbsp;no &lt;code&gt;sed&lt;/code&gt;, no header archaeology, first&amp;nbsp;try.&lt;/p&gt;
&lt;p&gt;There was exactly one speed bump, and it is a good educational&amp;nbsp;one:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@testvm3 ~]# modprobe bcachefs
modprobe: ERROR: could not insert &amp;#39;bcachefs&amp;#39;: Key was rejected by service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Reflexively, like every sysadmin who has been burned, I reached for the SELinux off&amp;nbsp;switch:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@testvm3 ~]# setenforce 0
[root@testvm3 ~]# modprobe bcachefs
modprobe: ERROR: could not insert &amp;#39;bcachefs&amp;#39;: Key was rejected by service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Still no. Because it is not&amp;nbsp;SELinux. &lt;code&gt;dmesg&lt;/code&gt; spells it&amp;nbsp;out:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;secureboot: Secure boot enabled
Kernel is locked down from EFI Secure Boot mode
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Key was rejected by service&amp;#8221; is Secure Boot refusing to load a module signed by a Machine Owner Key it has never been told to trust. The fix is to either enroll the &lt;span class="caps"&gt;DKMS&lt;/span&gt;-generated &lt;span class="caps"&gt;MOK&lt;/span&gt; or, in a throwaway &lt;span class="caps"&gt;VM&lt;/span&gt;, just turn Secure Boot off in the firmware and reboot.&amp;nbsp;(&lt;code&gt;setenforce 0&lt;/code&gt; did nothing, naturally, because reaching for SELinux when the problem is Secure Boot is the Linux equivalent of jiggling a light switch during a power&amp;nbsp;cut.)&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~ took 4m4s ❯ # Secureboot Disabling Noises ...
[root@testvm3 ~]# modprobe bcachefs
[root@testvm3 ~]#
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Silence. The beautiful silence of a module that loaded.&amp;nbsp;Then:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@testvm3 ~]# bcachefs format /dev/sdb1
  ...
  Version: need_discard_by_journal_seq (1.38)
  initializing new filesystem
  going read-write
  fs initialized, journal seq 5
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And, after a brief intermission where I tried to mount the root disk by mistake&amp;nbsp;(&lt;code&gt;/dev/sda1: invalid_sb_layout&lt;/code&gt;), fat-fingered the device node&amp;nbsp;(&lt;code&gt;/dev/sdab: No such file or directory&lt;/code&gt;), and finally typed it&amp;nbsp;correctly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@testvm3 ~]# mount -t bcachefs /dev/sdb1 /mnt
[root@testvm3 ~]#
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Mounted. A real bcachefs filesystem, on the distribution that builds it, in about two minutes of actual work. The four hours on &lt;span class="caps"&gt;RHEL&lt;/span&gt; produced nothing but a&amp;nbsp;long &lt;code&gt;make.log&lt;/code&gt; and&amp;nbsp;humility.&lt;/p&gt;
&lt;p&gt;&lt;img alt="bcachefs running on Fedora 44" src="https://hofstede.it/images/2026-06-01-bcachefs-rhel-10-dkms.png" title="bcachefs subvolumes on Fedora 44, where it built first try - the two-minute victory after a four-hour RHEL defeat"&gt;&lt;/p&gt;
&lt;h2 id="the-honest-assessment"&gt;The Honest&amp;nbsp;Assessment&lt;/h2&gt;
&lt;p&gt;The contrast is the entire lesson. On Fedora it took two commands and one Secure Boot detour, because I was using packages built for the distribution and kernel family they were intended to run on. On my &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10.2 attempt, the same idea turned into a porting exercise: the &lt;span class="caps"&gt;DKMS&lt;/span&gt; source expected upstream kernel interfaces that the &lt;span class="caps"&gt;RHEL&lt;/span&gt; kernel did not provide, and swapping in a newer kernel still did not produce a matching build&amp;nbsp;environment.&lt;/p&gt;
&lt;p&gt;That is not a knock on either project. &lt;span class="caps"&gt;RHEL&lt;/span&gt; provides a deliberately conservative, supportable kernel platform and preserves the compatibility guarantees needed by supported workloads. bcachefs, in its current out-of-tree form, is developing against much newer kernel internals. One wants controlled change; the other currently needs rapid change. My evening was what happens when you introduce them without a&amp;nbsp;chaperone.&lt;/p&gt;
&lt;p&gt;If you want supported advanced local storage on &lt;span class="caps"&gt;RHEL&lt;/span&gt;, &lt;a href="https://stratis-storage.github.io/"&gt;Stratis&lt;/a&gt; is the respectable answer for data volumes: pools, thin provisioning, snapshots, caching and encryption, with &lt;span class="caps"&gt;XFS&lt;/span&gt; underneath and documentation that does not begin with &amp;#8220;first, disable the guard rails.&amp;#8221; It is not currently a supported root-filesystem manager, so it would not reproduce my cursed &lt;span class="caps"&gt;ZFS&lt;/span&gt;-root experiment. That is probably a&amp;nbsp;feature.&lt;/p&gt;
&lt;p&gt;If you want bcachefs specifically, run it where its packaging and kernel expectations actually line up: a current Fedora system, Arch, or another sufficiently current environment. Let the filesystem and kernel grow together instead of arranging a blind date between a fast-moving &lt;span class="caps"&gt;DKMS&lt;/span&gt; module and an enterprise kernel whose job is to be&amp;nbsp;boring.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The &lt;span class="caps"&gt;ZFS&lt;/span&gt; experiment worked because I managed to provide a compatible filesystem module early enough for the kernel to mount root. The bcachefs experiment failed because I could not even build the code I wanted the kernel to load. Two filesystems, two completely different lessons, one homelab &lt;span class="caps"&gt;VM&lt;/span&gt; count that is now, mercifully,&amp;nbsp;even.&lt;/p&gt;
&lt;p&gt;bcachefs is good technology with an unfortunate current address: out in the cold, out of the tree, where every kernel that is not the newest one is a fresh fight. &lt;span class="caps"&gt;RHEL&lt;/span&gt; is good technology built specifically to never be the newest anything. Putting them together is like introducing your most punctual friend to your most chaotic one and expecting a calm&amp;nbsp;dinner.&lt;/p&gt;
&lt;p&gt;I tried &lt;span class="caps"&gt;REALLY&lt;/span&gt; hard. The kernel said no. Honestly, this time, the kernel was&amp;nbsp;right.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://bcachefs.org/"&gt;bcachefs.org&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://lwn.net/Kernel/Index/#bcachefs"&gt;&lt;span class="caps"&gt;LWN&lt;/span&gt;: bcachefs coverage (mainline inclusion through external maintenance and&amp;nbsp;removal)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://copr.fedorainfracloud.org/coprs/ngompa/bcachefs/"&gt;ngompa/bcachefs Copr&amp;nbsp;repository&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://stratis-storage.github.io/"&gt;Stratis Storage&amp;nbsp;Management&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10/html/managing_file_systems/managing-layered-local-storage-with-stratis_managing-file-systems"&gt;Red Hat: Managing layered local storage with&amp;nbsp;Stratis&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/dell/dkms"&gt;&lt;span class="caps"&gt;DKMS&lt;/span&gt; and Secure Boot module signing (&lt;span class="caps"&gt;MOK&lt;/span&gt;)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;The &lt;span class="caps"&gt;VM&lt;/span&gt; survived, the dignity did not, and somewhere&amp;nbsp;a &lt;code&gt;make.log&lt;/code&gt; is still scrolling. The things we do for a blog&amp;nbsp;post.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="rhel"/><category term="bcachefs"/><category term="dkms"/><category term="kernel"/><category term="fedora"/><category term="unsupported"/></entry><entry><title>Two Sites, One Cluster: My Hetzner Proxmox VE Setup - A low-cost two-node cluster with VXLAN, ZFS replication and an external quorum vote</title><link href="https://hofstede.it/hetzner-proxmox-cluster/" rel="alternate"/><published>2026-05-30T00:00:00+02:00</published><updated>2026-05-30T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-05-30:/hetzner-proxmox-cluster/</id><summary type="html">&lt;p&gt;How I run a single quorate Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; cluster across two cheap Hetzner dedicated servers in two different datacentres, glued together with Hetzner vSwitches, a &lt;span class="caps"&gt;VXLAN&lt;/span&gt; overlay, an external QDevice for quorum, and one OPNsense firewall per&amp;nbsp;node.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Proxmox VE 9 datacenter summary: the &amp;quot;noble&amp;quot; cluster with nodes krypton and xenon, both online and quorate" src="https://hofstede.it/images/2026-05-30-hetzner-proxmox-cluster.png" title="The noble cluster: two cheap Hetzner boxes, two cities, one quorum"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-big-picture"&gt;The Big Picture&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#what-this-is-and-what-it-is-not"&gt;What this is, and what it is&amp;nbsp;not&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-hardware"&gt;The&amp;nbsp;Hardware&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-vswitch-fabric"&gt;The vSwitch&amp;nbsp;Fabric&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#solving-the-two-node-quorum-problem"&gt;Solving the Two-Node Quorum&amp;nbsp;Problem&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-overlay-network-vxlan"&gt;The Overlay Network: &lt;span class="caps"&gt;VXLAN&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#firewalling-one-opnsense-per-node"&gt;Firewalling: One OPNsense per&amp;nbsp;Node&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#replication-and-migration"&gt;Replication and&amp;nbsp;Migration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#failure-modes"&gt;Failure&amp;nbsp;Modes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#lessons-learned"&gt;Lessons&amp;nbsp;Learned&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#conclusion"&gt;Conclusion&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;I have written a fair amount on this blog about FreeBSD, jails, and running my own &lt;span class="caps"&gt;AS&lt;/span&gt;. What I have never written about is the layer underneath a good chunk of it: a small Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; cluster built from two cheap Hetzner dedicated servers, sitting in two different datacentres, glued together over Hetzner&amp;#8217;s vSwitch fabric into a single quorate cluster. This article is the&amp;nbsp;tour.&lt;/p&gt;
&lt;p&gt;The guiding principle is the same one I apply everywhere: spend as little money as possible, understand every moving part, and avoid anything I cannot debug at 3 a.m. There is no enterprise &lt;span class="caps"&gt;SAN&lt;/span&gt; here, no fibre channel, no four-figure switch. Just two commodity boxes, a handful of VLANs, and Proxmox doing what it does&amp;nbsp;well.&lt;/p&gt;
&lt;h2 id="the-big-picture"&gt;The Big&amp;nbsp;Picture&lt;/h2&gt;
&lt;p&gt;The cluster is&amp;nbsp;called &lt;code&gt;noble&lt;/code&gt;, and the two nodes&amp;nbsp;are &lt;code&gt;krypton&lt;/code&gt; and &lt;code&gt;xenon&lt;/code&gt;. Yes, they are noble gases. I needed a naming scheme and that one is hard to run out&amp;nbsp;of.&lt;/p&gt;
&lt;p&gt;The two nodes are not in the same rack, the same row, or even the same&amp;nbsp;datacentre. &lt;code&gt;krypton&lt;/code&gt; lives in one Hetzner&amp;nbsp;location, &lt;code&gt;xenon&lt;/code&gt; in another city entirely. They are stitched together purely over the network: Hetzner&amp;#8217;s vSwitch lets you tag a &lt;span class="caps"&gt;VLAN&lt;/span&gt; onto a dedicated server&amp;#8217;s uplink and have it appear, at layer 2, on any other server you attach to the same vSwitch, regardless of where it physically sits. That single feature is what makes a geo-distributed two-node cluster on cheap hardware possible at all - as long as the latency between the sites stays low and stable, which is the caveat the rest of this article keeps coming back&amp;nbsp;to.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;            Internet (IPv4 + my own IPv6 / AS201379)
                 |                          |
        +--------+--------+        +--------+-----------+
        |    krypton      |        |     xenon          |
        |  (location A)   |        |  (location B)      |
        |                 |        |                    |
        |  OPNsense VM    |&amp;lt;==iBGP (v6)==&amp;gt;| OPNsense VM |
        |  Proxmox VE     |        |  Proxmox VE        |
        +--------+--------+        +--------+-----------+
                 |                          |
                 +======= Hetzner vSwitch ======+
                 |   4001  cluster / corosync   |
                 |   4002  replication          |
                 |   4003  VXLAN underlay       |
                 |   4000  Hetzner Cloud uplink |
                 |   + firewall sync            |
                 +==============================+
                                |
                        VXLAN id 1000
                  one stretched L2 for the VMs

         QDevice (QNetd) on a separate cheap VPS  --&amp;gt; 3rd vote
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Three votes, two nodes, one external arbiter. Everything the VMs see as &amp;#8220;the network&amp;#8221; is an overlay that floats on top of this, so a guest does not know or care which physical box it currently runs&amp;nbsp;on.&lt;/p&gt;
&lt;h3 id="what-this-is-and-what-it-is-not"&gt;What this is, and what it is&amp;nbsp;not&lt;/h3&gt;
&lt;p&gt;Before anyone copies this wholesale, be clear about what the design does and does not give&amp;nbsp;you:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;It is not synchronous storage.&lt;/strong&gt; Replication is asynchronous, so a failover loses whatever was written since the last sync&amp;nbsp;interval.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;It is not zero-data-loss, push-button &lt;span class="caps"&gt;HA&lt;/span&gt;.&lt;/strong&gt; Proxmox &lt;span class="caps"&gt;HA&lt;/span&gt; can restart guests on the survivor, but only the data that was already replicated comes with&amp;nbsp;them.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;It depends on low, stable inter-site latency.&lt;/strong&gt; Corosync wants something that behaves like a &lt;span class="caps"&gt;LAN&lt;/span&gt;. If the path between the sites degrades, the cluster suffers, full&amp;nbsp;stop.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;It is not redundant against every Hetzner failure.&lt;/strong&gt; Both nodes, both uplinks, and the vSwitch fabric are all Hetzner. A large enough Hetzner network event takes the whole thing down regardless of how clever the topology&amp;nbsp;is.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;With those caveats stated up front, here is how each piece&amp;nbsp;works.&lt;/p&gt;
&lt;h2 id="the-hardware"&gt;The&amp;nbsp;Hardware&lt;/h2&gt;
&lt;p&gt;Both nodes are ordinary Hetzner dedicated servers - the kind you get from the regular lineup or the server auction for tens of euros a month. Nothing exotic. Looking at the datacentre summary, the whole cluster adds up to 24 &lt;span class="caps"&gt;CPU&lt;/span&gt; threads, around 314 GiB of &lt;span class="caps"&gt;RAM&lt;/span&gt;, and roughly 1.9 TiB of usable storage, and at the moment it idles at about 2% &lt;span class="caps"&gt;CPU&lt;/span&gt; and a quarter of its memory while running thirteen VMs. That is a lot of headroom for the price of two mid-range&amp;nbsp;boxes.&lt;/p&gt;
&lt;p&gt;There is no shared storage. Each node has its own local &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool, and &lt;span class="caps"&gt;VM&lt;/span&gt; disks are kept in sync between the nodes with Proxmox&amp;#8217;s built-in &lt;span class="caps"&gt;ZFS&lt;/span&gt; replication. This is the deliberately un-fancy choice: shared storage is the part of a cluster most likely to take everything down with it, and I would rather have two independent pools and asynchronous replication than one expensive single point of&amp;nbsp;failure.&lt;/p&gt;
&lt;h2 id="the-vswitch-fabric"&gt;The vSwitch&amp;nbsp;Fabric&lt;/h2&gt;
&lt;p&gt;Hetzner&amp;#8217;s vSwitch is the backbone here. You enable a vSwitch in the Robot panel, assign it a &lt;span class="caps"&gt;VLAN&lt;/span&gt; &lt;span class="caps"&gt;ID&lt;/span&gt;, and then tag that &lt;span class="caps"&gt;VLAN&lt;/span&gt; on the server&amp;#8217;s physical interface. Anything you put on that &lt;span class="caps"&gt;VLAN&lt;/span&gt; reaches every other server attached to the same vSwitch. The one catch is the &lt;span class="caps"&gt;MTU&lt;/span&gt;: the vSwitch eats into the standard 1500-byte frame, so everything riding on it has to run at &lt;strong&gt;&lt;span class="caps"&gt;MTU&lt;/span&gt; 1400&lt;/strong&gt;.&lt;/p&gt;
&lt;p&gt;Rather than dump everything onto one flat &lt;span class="caps"&gt;VLAN&lt;/span&gt;, I split traffic by purpose. Each job gets its own &lt;span class="caps"&gt;VLAN&lt;/span&gt; and its own little&amp;nbsp;/24:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;&lt;span class="caps"&gt;VLAN&lt;/span&gt;&lt;/th&gt;
&lt;th&gt;Subnet&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;4000&lt;/td&gt;
&lt;td&gt;(bridged)&lt;/td&gt;
&lt;td&gt;Uplink to Hetzner Cloud&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4001&lt;/td&gt;
&lt;td&gt;172.16.1.0/24&lt;/td&gt;
&lt;td&gt;Proxmox cluster / corosync&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4002&lt;/td&gt;
&lt;td&gt;172.17.1.0/24&lt;/td&gt;
&lt;td&gt;Storage replication&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;4003&lt;/td&gt;
&lt;td&gt;172.18.1.0/24&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;VXLAN&lt;/span&gt; underlay&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;OPNsense&amp;#8217;s state synchronisation rides its own separate vSwitch segment as well, parallel to the four above. I am deliberately not giving its addressing here because it is the one piece you should size to your own firewall pair rather than copy from&amp;nbsp;mine.&lt;/p&gt;
&lt;p&gt;On &lt;code&gt;xenon&lt;/code&gt; (node &lt;span class="caps"&gt;ID&lt;/span&gt; 2), the host side looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;auto enp35s0.4001
iface enp35s0.4001 inet static
        address 172.16.1.1/24
        vlan-raw-device enp35s0
        mtu 1400
# Hetzner vSwitch (Proxmox Cluster)

auto enp35s0.4002
iface enp35s0.4002 inet static
        address 172.17.1.1/24
        vlan-raw-device enp35s0
        mtu 1400
# Hetzner vSwitch (Proxmox Replication)

auto enp35s0.4003
iface enp35s0.4003 inet static
        address 172.18.1.1/24
        vlan-raw-device enp35s0
        mtu 1400
# Hetzner vSwitch (Proxmox VXLAN VM Network)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;krypton&lt;/code&gt; (node &lt;span class="caps"&gt;ID&lt;/span&gt; 1) is the mirror image of this on&amp;nbsp;the &lt;code&gt;172.x.1.2&lt;/code&gt; addresses. The 4000 &lt;span class="caps"&gt;VLAN&lt;/span&gt; is handled slightly differently: it is bridged&amp;nbsp;into &lt;code&gt;vmbr4000&lt;/code&gt; rather than given an &lt;span class="caps"&gt;IP&lt;/span&gt;, because its job is to extend a layer-2 segment to machines over in Hetzner Cloud, not to terminate on the&amp;nbsp;host.&lt;/p&gt;
&lt;p&gt;Splitting the networks like this makes the design far easier to reason about: corosync, replication, the &lt;span class="caps"&gt;VXLAN&lt;/span&gt; underlay, and the Cloud uplink each get their own &lt;span class="caps"&gt;VLAN&lt;/span&gt;, their own addressing, their own firewall policy, and their own troubleshooting path. What it does &lt;em&gt;not&lt;/em&gt; do is magically create separate physical wires. All of these VLANs ride the same server uplink - Hetzner&amp;#8217;s vSwitch is a tagging-and-tunnelling feature on the one physical &lt;span class="caps"&gt;NIC&lt;/span&gt;, not extra cabling - so a big&amp;nbsp;enough &lt;code&gt;zfs send&lt;/code&gt; or live migration can still contend with corosync&amp;#8217;s heartbeat on that shared link unless you rate-limit or prioritise&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;That matters because Proxmox expects corosync to behave like a &lt;span class="caps"&gt;LAN&lt;/span&gt;: low latency, low jitter, reliable delivery. The textbook guidance is to keep the corosync round-trip well under a few milliseconds. So the honest version of this advice is not &amp;#8220;give corosync its own wire&amp;#8221; but &amp;#8220;give corosync its own &lt;em&gt;&lt;span class="caps"&gt;VLAN&lt;/span&gt;&lt;/em&gt; for clarity and filtering, then go measure the&amp;nbsp;path.&amp;#8221;&lt;/p&gt;
&lt;p&gt;Here is where I have to be honest rather than flattering, because my cluster does not meet Proxmox&amp;#8217;s documented latency target at all. The two nodes sit in Nuremberg and Helsinki, and the corosync path between them looks like&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@krypton ~]# ping -c5 172.16.1.1
64 bytes from 172.16.1.1: icmp_seq=1 ttl=64 time=25.2 ms
64 bytes from 172.16.1.1: icmp_seq=2 ttl=64 time=25.2 ms
64 bytes from 172.16.1.1: icmp_seq=3 ttl=64 time=25.2 ms
64 bytes from 172.16.1.1: icmp_seq=4 ttl=64 time=25.1 ms
64 bytes from 172.16.1.1: icmp_seq=5 ttl=64 time=25.1 ms

rtt min/avg/max/mdev = 25.069/25.150/25.234/0.065 ms
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is roughly 25 ms round-trip, well outside Proxmox&amp;#8217;s documented expectation of a sub-5 ms cluster network. I am publishing it because it is the real architecture I run, not because it is the conservative design Proxmox&amp;nbsp;recommends.&lt;/p&gt;
&lt;p&gt;In practice, this particular path has remained extremely stable, with very little observed jitter, and the cluster has behaved well under my workloads. That is an operational observation, not a guarantee: a different inter-site path, more congestion, packet loss, or a different workload could make the same design&amp;nbsp;unreliable.&lt;/p&gt;
&lt;p&gt;The lesson is therefore not that 25 ms is safe. The lesson is that, if you deliberately step outside the documented design envelope, you need to measure the actual path under replication and migration load, monitor corosync closely, and accept that you own the resulting failure&amp;nbsp;modes.&lt;/p&gt;
&lt;h2 id="solving-the-two-node-quorum-problem"&gt;Solving the Two-Node Quorum&amp;nbsp;Problem&lt;/h2&gt;
&lt;p&gt;Two-node clusters have a famous flaw: quorum. A cluster needs a strict majority of votes to stay live and avoid split-brain. With two nodes you have two votes, and as soon as one node drops, the survivor only has one vote out of two - not a majority - so it refuses to do anything. The very failure you built the cluster to survive takes the cluster&amp;nbsp;down.&lt;/p&gt;
&lt;p&gt;Proxmox&amp;#8217;s answer is a &lt;strong&gt;QDevice&lt;/strong&gt;: a lightweight external arbiter that holds a third vote. I run&amp;nbsp;a &lt;code&gt;corosync-qnetd&lt;/code&gt; daemon on a separate, cheap &lt;span class="caps"&gt;VPS&lt;/span&gt; that has nothing else to do with the cluster, and both nodes talk to it. Now there are three votes. If one node dies, the survivor plus the QDevice make two votes out of three, which is a majority, and the cluster keeps&amp;nbsp;serving.&lt;/p&gt;
&lt;p&gt;The result,&amp;nbsp;from &lt;code&gt;krypton&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@krypton ~]# pvecm status
Cluster information
-------------------
Name:             noble
Transport:        knet

Votequorum information
----------------------
Expected votes:   3
Highest expected: 3
Total votes:      3
Quorum:           2
Flags:            Quorate Qdevice

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
0x00000001          1    A,V,NMW 172.16.1.2 (local)
0x00000002          1    A,V,NMW 172.16.1.1
0x00000000          1            Qdevice
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The QDevice itself reports how it breaks a&amp;nbsp;tie:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@krypton ~]# corosync-qdevice-tool -s
Qdevice-net information
----------------------
Cluster name:           noble
QNetd host:             203.0.113.175:5403
Algorithm:              Fifty-Fifty split
Tie-breaker:            Node with lowest node ID
State:                  Connected
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;strong&gt;Fifty-Fifty split&lt;/strong&gt; algorithm is exactly what you want for two nodes across two sites. If the vSwitch link between the nodes breaks but both are still alive and both can still reach the QNetd &lt;span class="caps"&gt;VPS&lt;/span&gt;, the QDevice hands its vote to exactly one partition - here, the node with the lowest &lt;span class="caps"&gt;ID&lt;/span&gt; - so one side stays quorate and the other gracefully steps down. No split-brain, no two nodes both convinced they are the&amp;nbsp;survivor.&lt;/p&gt;
&lt;p&gt;Putting the QNetd box on a completely separate provider is intentional. Its whole value is being an independent witness, and an arbiter that shares a failure domain with the thing it is arbitrating is no arbiter at&amp;nbsp;all.&lt;/p&gt;
&lt;h2 id="the-overlay-network-vxlan"&gt;The Overlay Network: &lt;span class="caps"&gt;VXLAN&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Here is the interesting part. The nodes are in different datacentres, so there is no shared physical &lt;span class="caps"&gt;LAN&lt;/span&gt; I can simply bridge a &lt;span class="caps"&gt;VM&lt;/span&gt; onto. I want a &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;nbsp;on &lt;code&gt;krypton&lt;/code&gt; and a &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;nbsp;on &lt;code&gt;xenon&lt;/code&gt; to sit on the same layer-2 segment as if they were plugged into the same switch. That is what Proxmox &lt;span class="caps"&gt;SDN&lt;/span&gt; with &lt;span class="caps"&gt;VXLAN&lt;/span&gt; gives&amp;nbsp;me.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;VXLAN&lt;/span&gt; wraps ethernet frames inside &lt;span class="caps"&gt;UDP&lt;/span&gt; and ships them between the nodes. I run a single &lt;span class="caps"&gt;VXLAN&lt;/span&gt; tunnel (&lt;span class="caps"&gt;VNI&lt;/span&gt; 1000) over the dedicated 4003 underlay &lt;span class="caps"&gt;VLAN&lt;/span&gt;, and present it to the VMs as one &lt;span class="caps"&gt;VLAN&lt;/span&gt;-aware&amp;nbsp;bridge:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# /etc/network/interfaces.d/sdn  (on xenon)
auto vxlan_mainvx
iface vxlan_mainvx
        vxlan-id 1000
        vxlan_remoteip 172.18.1.2
        mtu 1332

auto mainvx
iface mainvx
        bridge_ports vxlan_mainvx
        bridge_stp off
        bridge_fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094
        mtu 1332
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;krypton&lt;/code&gt; is identical except&amp;nbsp;its &lt;code&gt;vxlan_remoteip&lt;/code&gt; points back&amp;nbsp;at &lt;code&gt;172.18.1.2&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s counterpart. Each node tunnels to the other&amp;#8217;s underlay address, and&amp;nbsp;the &lt;code&gt;mainvx&lt;/code&gt; bridge is where guest NICs&amp;nbsp;land.&lt;/p&gt;
&lt;p&gt;Then there is the &lt;span class="caps"&gt;MTU&lt;/span&gt; bookkeeping, which is the thing everyone gets wrong the first time. Hetzner recommends an &lt;span class="caps"&gt;MTU&lt;/span&gt; of 1400 on vSwitch &lt;span class="caps"&gt;VLAN&lt;/span&gt; interfaces. &lt;span class="caps"&gt;VXLAN&lt;/span&gt; adds 50 bytes of encapsulation overhead on top of that, so &lt;strong&gt;1350 is the calculated upper bound&lt;/strong&gt; for traffic inside the overlay. I deliberately set&amp;nbsp;both &lt;code&gt;mainvx&lt;/code&gt; and &lt;code&gt;vxlan_mainvx&lt;/code&gt; to &lt;strong&gt;1332&lt;/strong&gt;, leaving a little extra margin in my particular&amp;nbsp;setup.&lt;/p&gt;
&lt;p&gt;That 1332 is conservative, not magical - another deployment may run perfectly at the full 1350. The rule that actually matters is the one underneath the numbers: the guest-facing &lt;span class="caps"&gt;MTU&lt;/span&gt; has to fit inside the smallest &lt;span class="caps"&gt;MTU&lt;/span&gt; along the entire encapsulated path, or large packets fail in far more confusing ways than small pings ever will. That is the &amp;#8220;small pings work, large transfers hang&amp;#8221; class of bug, and it is always an &lt;span class="caps"&gt;MTU&lt;/span&gt; that does not add&amp;nbsp;up.&lt;/p&gt;
&lt;p&gt;Because the bridge is &lt;span class="caps"&gt;VLAN&lt;/span&gt;-aware&amp;nbsp;with &lt;code&gt;bridge-vids 2-4094&lt;/code&gt;, I can carve the stretched segment into as many tagged &lt;span class="caps"&gt;VM&lt;/span&gt; networks as I like, all riding the single&amp;nbsp;tunnel.&lt;/p&gt;
&lt;h2 id="firewalling-one-opnsense-per-node"&gt;Firewalling: One OPNsense per&amp;nbsp;Node&lt;/h2&gt;
&lt;p&gt;Each node runs its own OPNsense &lt;span class="caps"&gt;VM&lt;/span&gt; as the firewall and router for everything behind it.&amp;nbsp;On &lt;code&gt;krypton&lt;/code&gt;, the host&amp;#8217;s public bridge is explicitly shared with&amp;nbsp;OPNsense:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;auto vmbr0
iface vmbr0 inet static
        address 192.0.2.115/26
        gateway 192.0.2.65
        bridge-ports enp4s0
        bridge-stp off
        bridge-fd 0
#OPNSense WAN - Proxmox LAN
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;(The public addresses throughout this article are replaced with documentation-range examples&amp;nbsp;- &lt;code&gt;192.0.2.0/24&lt;/code&gt;, &lt;code&gt;198.51.100.0/24&lt;/code&gt;, &lt;code&gt;203.0.113.0/24&lt;/code&gt; and &lt;code&gt;2001:db8::/32&lt;/code&gt; - so substitute your&amp;nbsp;own.)&lt;/p&gt;
&lt;p&gt;The two firewalls are not islands. A dedicated vSwitch segment carries OPNsense&amp;#8217;s state synchronisation (pfSync) between them, and the pair runs an &lt;strong&gt;iBGP session&lt;/strong&gt; so that my own IPv6 space (the prefixes I announce from &lt;span class="caps"&gt;AS201379&lt;/span&gt;, which I have written about in &lt;a href="https://hofstede.it/running-your-own-as-bgp-on-freebsd-with-frr-gre-tunnels-and-policy-routing/"&gt;other posts&lt;/a&gt;) is distributed across both sites. IPv6 is, to me, the real network: every &lt;span class="caps"&gt;VM&lt;/span&gt; gets globally routable addresses out of my own space, and iBGP makes sure both nodes know how to reach all of it no matter which one a &lt;span class="caps"&gt;VM&lt;/span&gt; lives&amp;nbsp;on.&lt;/p&gt;
&lt;p&gt;It is worth being precise about what actually fails over here, because pfSync alone does not move traffic: it synchronises firewall state, while &lt;span class="caps"&gt;BGP&lt;/span&gt; convergence redirects routed traffic. Each site announces routes independently; when one OPNsense &lt;span class="caps"&gt;VM&lt;/span&gt; disappears, its &lt;span class="caps"&gt;BGP&lt;/span&gt; sessions drop, its routes are withdrawn, and traffic reconverges through the surviving&amp;nbsp;firewall.&lt;/p&gt;
&lt;p&gt;In principle, synchronised state gives established connections a chance to survive that routing change. In practice, survival still depends on the traffic returning through a firewall with matching state and on the surrounding &lt;span class="caps"&gt;NAT&lt;/span&gt; and routing behaviour. There is no &lt;span class="caps"&gt;CARP&lt;/span&gt; virtual &lt;span class="caps"&gt;IP&lt;/span&gt; in this design: failover is driven by routing, not by a shared layer-2&amp;nbsp;address.&lt;/p&gt;
&lt;p&gt;IPv4 gets the treatment it deserves at this point in history. I genuinely do not care about legacy &lt;span class="caps"&gt;IP&lt;/span&gt; for outbound any more, so&amp;nbsp;on &lt;code&gt;xenon&lt;/code&gt; I do not bother routing a public range to it. The host just&amp;nbsp;NATs:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;auto vmbr1
iface vmbr1 inet static
        address 172.19.1.2/24
        bridge-ports none
        post-up iptables -t nat -A POSTROUTING -s &amp;#39;172.19.1.0/24&amp;#39; -o vmbr0 -j MASQUERADE
        post-down iptables -t nat -D POSTROUTING -s &amp;#39;172.19.1.0/24&amp;#39; -o vmbr0 -j MASQUERADE
# NAT
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Inbound, &lt;span class="caps"&gt;SSH&lt;/span&gt; and the Proxmox web &lt;span class="caps"&gt;UI&lt;/span&gt; terminate on the host so I can still manage it when the OPNsense &lt;span class="caps"&gt;VM&lt;/span&gt; is down or misconfigured. All &lt;em&gt;other&lt;/em&gt; inbound &lt;span class="caps"&gt;TCP&lt;/span&gt; is DNATed straight to the OPNsense &lt;span class="caps"&gt;VM&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;post-up   iptables -t nat -A PREROUTING -i vmbr0 -p tcp -m multiport ! --dports 22,8006 -j DNAT --to-destination 172.19.1.1
post-down iptables -t nat -D PREROUTING -i vmbr0 -p tcp -m multiport ! --dports 22,8006 -j DNAT --to-destination 172.19.1.1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two things this rule does &lt;em&gt;not&lt;/em&gt; do, on purpose. It only&amp;nbsp;matches &lt;code&gt;-p tcp&lt;/code&gt;, so any &lt;span class="caps"&gt;UDP&lt;/span&gt; service I wanted behind OPNsense would need its own equivalent rule - this example is &lt;span class="caps"&gt;TCP&lt;/span&gt;-only. And I have left out the corresponding forwarding and filter policy, because the part worth showing is the boundary itself, not a full firewall&amp;nbsp;ruleset.&lt;/p&gt;
&lt;p&gt;That recovery path - management ports living on the host, outside the firewall &lt;span class="caps"&gt;VM&lt;/span&gt; - is convenient, but it is a security trade-off, not a best practice to copy blindly. Exposing the Proxmox web &lt;span class="caps"&gt;UI&lt;/span&gt; is not something you do openly: those management ports are restricted separately, with the Proxmox cluster/host firewall locking them down to a management allowlist rather than treating them as general internet-facing services. The design goal is that host management stays reachable independently of the firewall &lt;span class="caps"&gt;VM&lt;/span&gt;, &lt;em&gt;and&lt;/em&gt; stays locked&amp;nbsp;down.&lt;/p&gt;
&lt;h2 id="replication-and-migration"&gt;Replication and&amp;nbsp;Migration&lt;/h2&gt;
&lt;p&gt;With separate pools and the 4002 replication &lt;span class="caps"&gt;VLAN&lt;/span&gt; in place, Proxmox keeps each &lt;span class="caps"&gt;VM&lt;/span&gt;&amp;#8217;s disk replicated to the other node on a schedule. The payoff is twofold. Live migration between two boxes in different cities is genuinely fast, because only the delta since the last replication run has to move. And if a node falls over, a recent copy of every &lt;span class="caps"&gt;VM&lt;/span&gt; is already sitting on its&amp;nbsp;partner.&lt;/p&gt;
&lt;p&gt;The guests I care about are configured as &lt;strong&gt;Proxmox &lt;span class="caps"&gt;HA&lt;/span&gt; resources&lt;/strong&gt;, so that recovery is not a manual scramble: when a node is fenced, the &lt;span class="caps"&gt;HA&lt;/span&gt; manager restarts those VMs on the survivor automatically. This is worth stating clearly because replication on its own does &lt;em&gt;not&lt;/em&gt; restart anything - it only places a copy on the other node. &lt;span class="caps"&gt;HA&lt;/span&gt; is the part that acts on that copy, and it only acts on the resources you have actually added to the &lt;span class="caps"&gt;HA&lt;/span&gt;&amp;nbsp;manager.&lt;/p&gt;
&lt;p&gt;Here is the catch that no amount of &lt;span class="caps"&gt;HA&lt;/span&gt; configuration removes: replication is &lt;strong&gt;asynchronous&lt;/strong&gt;. When &lt;span class="caps"&gt;HA&lt;/span&gt; restarts a &lt;span class="caps"&gt;VM&lt;/span&gt; on the survivor, it boots from the last replicated snapshot, so anything written between that snapshot and the failure is gone. For my workloads - blogs, a few services, a Factorio server - a replication interval&amp;#8217;s worth of potential loss is completely acceptable, and the operational simplicity I get in return is worth far more than synchronous shared storage would be. Know your own tolerance before you copy&amp;nbsp;mine.&lt;/p&gt;
&lt;h2 id="failure-modes"&gt;Failure&amp;nbsp;Modes&lt;/h2&gt;
&lt;p&gt;The point of all this is what happens when something breaks. Here is how the cluster behaves for the failures I actually planned&amp;nbsp;for:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Failure&lt;/th&gt;
&lt;th&gt;What happens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;One node dies&lt;/td&gt;
&lt;td&gt;Survivor + QDevice = 2 of 3 votes, so the cluster stays quorate. &lt;span class="caps"&gt;HA&lt;/span&gt;-managed VMs restart on the survivor from their last replicated snapshot (async, so recent writes are lost).&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;QNetd &lt;span class="caps"&gt;VPS&lt;/span&gt; dies (only)&lt;/td&gt;
&lt;td&gt;Both nodes are still up and still see each other, so they hold 2 of 2 node votes and the cluster keeps running. You have simply lost the tie-breaker until QNetd returns, so &lt;em&gt;now&lt;/em&gt; a node failure would be fatal - fix it promptly.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Inter-node vSwitch path dies&lt;/td&gt;
&lt;td&gt;Both nodes alive but partitioned. Each can still reach QNetd, so the Fifty-Fifty split hands the vote to the lowest node &lt;span class="caps"&gt;ID&lt;/span&gt;; that side stays quorate, the other steps down. No split-brain.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;One OPNsense &lt;span class="caps"&gt;VM&lt;/span&gt; dies&lt;/td&gt;
&lt;td&gt;Its &lt;span class="caps"&gt;BGP&lt;/span&gt; sessions drop and routes withdraw; traffic reconverges through the surviving firewall. Because pfSync has copied connection state across, established sessions may survive where the reconverged traffic path and &lt;span class="caps"&gt;NAT&lt;/span&gt; state line up.&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;One local &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool dies&lt;/td&gt;
&lt;td&gt;Replicated copies exist on the partner, but automatic recovery depends on how the failure presents itself. If the node is fenced or the &lt;span class="caps"&gt;HA&lt;/span&gt; resource fails in a way &lt;span class="caps"&gt;HA&lt;/span&gt; can recover, the guest can be restarted from the replicated copy; a pool fault on an otherwise live node may still require operator intervention.&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="lessons-learned"&gt;Lessons&amp;nbsp;Learned&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Give corosync its own &lt;span class="caps"&gt;VLAN&lt;/span&gt; - and then go measure the path.&lt;/strong&gt; A dedicated &lt;span class="caps"&gt;VLAN&lt;/span&gt; buys you clarity, filtering, and a clean troubleshooting story. It does &lt;em&gt;not&lt;/em&gt; buy you a separate physical wire: on a Hetzner vSwitch everything still shares the one server uplink, so a big replication or migration transfer can still starve the heartbeat unless you rate-limit it. The reliability lever that actually matters is latency and jitter, so measure your inter-node &lt;span class="caps"&gt;RTT&lt;/span&gt; under load and keep it &lt;span class="caps"&gt;LAN&lt;/span&gt;-like.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A QDevice is mandatory for this design&lt;/strong&gt; if the surviving node is expected to stay quorate after the other one fails. A two-node cluster can technically exist without one - it simply cannot deliver the failure behaviour this whole article promises. Put the QNetd arbiter on a different provider so it is a genuinely independent&amp;nbsp;witness.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Do the &lt;span class="caps"&gt;MTU&lt;/span&gt; maths once, carefully.&lt;/strong&gt; 1500 down to 1400 for the vSwitch, minus 50 bytes of &lt;span class="caps"&gt;VXLAN&lt;/span&gt; overhead, gives 1350 as the calculated overlay ceiling; I run 1332 for margin. Get it wrong and you will chase intermittent stalls for days. Get it right and you never think about it&amp;nbsp;again.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The vSwitch makes location less visible to the guests - not irrelevant to you.&lt;/strong&gt; Two boxes in two cities presenting one stretched L2 to the VMs is something that used to require expensive kit, and Hetzner hands it to you as a checkbox. But geography still matters profoundly underneath: it sets your corosync latency and defines your failure domains. The guests stop caring where they run; you never get&amp;nbsp;to.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Decide what you actually care about.&lt;/strong&gt; I care about IPv6 and about being able to manage the hosts. I do not care about inbound IPv4 to one node. Being honest about that let me replace a whole layer of routing with three lines&amp;nbsp;of &lt;code&gt;iptables&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;None of this is exotic. It is two cheap servers, a layer-2 fabric I did not have to build, an overlay network, an external tie-breaker, and a firewall per node. But the sum is a resilient, geo-distributed virtualisation platform that remains cheap enough to run as a personal platform rather than an enterprise procurement exercise, and which I understand top to bottom because I assembled every piece&amp;nbsp;myself.&lt;/p&gt;
&lt;p&gt;That last part is the whole point. When something breaks at 3 a.m., I am not filing a support ticket and waiting. I am reading pvecm status, checking the vSwitch, and fixing it. Because I know exactly what every line of these configuration files&amp;nbsp;does.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://pve.proxmox.com/wiki/Cluster_Manager"&gt;Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; Cluster&amp;nbsp;Manager&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pve.proxmox.com/wiki/Cluster_Manager#_corosync_external_vote_support"&gt;Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; QDevice /&amp;nbsp;corosync-qnetd&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pve.proxmox.com/wiki/Software-Defined_Network"&gt;Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; Software-Defined Network (&lt;span class="caps"&gt;SDN&lt;/span&gt;)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.hetzner.com/robot/dedicated-server/network/vswitch/"&gt;Hetzner Docs -&amp;nbsp;vSwitch&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.opnsense.org/manual/hacarp.html"&gt;OPNsense High&amp;nbsp;Availability&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.rfc-editor.org/rfc/rfc7348"&gt;&lt;span class="caps"&gt;RFC&lt;/span&gt; 7348 - Virtual eXtensible Local Area Network (&lt;span class="caps"&gt;VXLAN&lt;/span&gt;)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="proxmox"/><category term="hetzner"/><category term="vxlan"/><category term="corosync"/><category term="opnsense"/><category term="bgp"/><category term="networking"/><category term="virtualization"/><category term="self-hosting"/></entry><entry><title>Tag-Driven Deployments: How MastoSum Ships Itself with Forgejo Actions and Rootless Podman</title><link href="https://hofstede.it/tag-driven-deployments-how-mastosum-ships-itself-with-forgejo-actions-and-rootless-podman/" rel="alternate"/><published>2026-05-24T00:00:00+02:00</published><updated>2026-05-24T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-05-24:/tag-driven-deployments-how-mastosum-ships-itself-with-forgejo-actions-and-rootless-podman/</id><summary type="html">&lt;p&gt;A walk through the MastoSum deployment pipeline: a version tag triggers Forgejo Actions, builds and signs a &lt;span class="caps"&gt;UBI&lt;/span&gt;-based image, then hands deployment to a rootless systemd/Podman host via a path unit and explicit cosign&amp;nbsp;verification.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="MastoSum deployment pipeline: Forgejo Actions workflow pushing a signed image, systemd path unit triggering the deploy" src="https://hofstede.it/images/2026-05-24-mastosum-deployment-pipeline.png" title="MastoSum deployment pipeline overview"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-stack-at-rest"&gt;The Stack at&amp;nbsp;Rest&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-pipeline-tag-build-push-sign"&gt;The Pipeline: Tag, Build, Push, Sign&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#1-build"&gt;1.&amp;nbsp;Build&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#2-push"&gt;2.&amp;nbsp;Push&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#3-sign"&gt;3.&amp;nbsp;Sign&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#4-deploy-handoff"&gt;4. Deploy&amp;nbsp;Handoff&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#signature-verification-design-and-current-reality"&gt;Signature Verification: Design and Current Reality&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#registry-authentication-operational"&gt;Registry authentication&amp;nbsp;(operational)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#host-side-cosign-verification-in-the-deploy-script-operational"&gt;Host-side cosign verification in the deploy script&amp;nbsp;(operational)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#container-policy-enforcement-via-policyjson-target-state"&gt;Container policy enforcement via policy.json (target&amp;nbsp;state)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-runner-in-the-same-user-context"&gt;The Runner in the Same User&amp;nbsp;Context&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-deploy-bridge-systemd-path-unit"&gt;The Deploy Bridge: systemd Path&amp;nbsp;Unit&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rollback"&gt;Rollback&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-full-picture"&gt;The Full&amp;nbsp;Picture&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;I&amp;#8217;ve written about &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Podman Quadlets&lt;/a&gt;, &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;rootless production Podman&lt;/a&gt;, and &lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/"&gt;Ansible-driven Quadlet deployments&lt;/a&gt; before. Those articles cover the steady state: containers running, systemd managing them, configuration under version control. What they don&amp;#8217;t cover is the moment&amp;nbsp;between &lt;code&gt;git push&lt;/code&gt; and &lt;code&gt;systemctl restart&lt;/code&gt;: the pipeline that builds the image, proves where it came from, and hands it off to the&amp;nbsp;host.&lt;/p&gt;
&lt;p&gt;MastoSum is a FastAPI app I run that watches Mastodon hashtag streams, stores matching posts, and generates news-style summaries through LLMs or a deterministic fallback. The application topology is intentionally boring: one container image, four roles selected by an environment variable&amp;nbsp;(&lt;code&gt;web&lt;/code&gt;, &lt;code&gt;worker&lt;/code&gt;, &lt;code&gt;beat&lt;/code&gt;, &lt;code&gt;flower&lt;/code&gt;), Postgres and Valkey as backing services. What&amp;#8217;s worth writing about is how it&amp;nbsp;ships.&lt;/p&gt;
&lt;p&gt;The deployment story is fully automated, tag-driven, and runs entirely rootless inside a single Podman user context. Every piece, the &lt;span class="caps"&gt;CI&lt;/span&gt; runner, the registry, the build pipeline, the signature verification, the deploy trigger, the service restarts, lives inside&amp;nbsp;the &lt;code&gt;mastosum@&lt;/code&gt; user on one &lt;span class="caps"&gt;RHEL&lt;/span&gt; host. This article walks through each link in that&amp;nbsp;chain.&lt;/p&gt;
&lt;h2 id="the-stack-at-rest"&gt;The Stack at&amp;nbsp;Rest&lt;/h2&gt;
&lt;p&gt;Before getting into the pipeline, it helps to see what the running stack looks like. On the production host, everything runs as&amp;nbsp;the &lt;code&gt;mastosum&lt;/code&gt; user:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ podman ps
CONTAINER ID  IMAGE                                           COMMAND               STATUS                   PORTS                                         NAMES
dda7c5013624  registry.redhat.io/rhel10/valkey-8:latest       run-valkey            Up 4 days (healthy)      6379/tcp                                      mastosum-valkey
13a829bea824  registry.redhat.io/rhel10/postgresql-16:latest  run-postgresql        Up 4 days (healthy)      5432/tcp                                      mastosum-postgres
bedb39c79a9d  code.forgejo.org/forgejo/runner:12              forgejo-runner da...  Up 2 hours                                                             mastosum-runner
7055f528d292  git.linuxserver.pro/chris/mastosum:stable       python3 -m mastos...  Up 25 minutes (healthy)  127.0.0.1:8000-&amp;gt;8000/tcp, 5555/tcp, 8080/tcp  mastosum-web
9d3c7666cc6a  git.linuxserver.pro/chris/mastosum:stable       python3 -m mastos...  Up 25 minutes (healthy)  5555/tcp, 8000/tcp, 8080/tcp                  mastosum-worker
b4d69e12ba35  git.linuxserver.pro/chris/mastosum:stable       python3 -m mastos...  Up 25 minutes (healthy)  5555/tcp, 8000/tcp, 8080/tcp                  mastosum-beat
51e77dea5ffb  git.linuxserver.pro/chris/mastosum:stable       celery -A mastosu...  Up 25 minutes (healthy)  127.0.0.1:5555-&amp;gt;5555/tcp, 8000/tcp, 8080/tcp  mastosum-flower
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Seven containers. Three images: the app image&amp;nbsp;(&lt;code&gt;mastosum:stable&lt;/code&gt;) running four roles, plus Postgres and Valkey from Red Hat&amp;#8217;s container catalog, plus the Forgejo Actions runner. All of them managed by systemd via rootless Podman Quadlets. The runner is a first-class member of the stack, not a separate daemon on a separate host. This setup, the runner living inside the same user context it deploys into is what makes the pipeline clean. We&amp;#8217;ll get to&amp;nbsp;why.&lt;/p&gt;
&lt;p&gt;The app image uses Red Hat&amp;#8217;s &lt;span class="caps"&gt;UBI10&lt;/span&gt; Python 3.12 minimal&amp;nbsp;base:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;registry.access.redhat.com/ubi10/python-312-minimal&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;builder&lt;/span&gt;
&lt;span class="c"&gt;# ... venv, pip install, etc.&lt;/span&gt;

&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;registry.access.redhat.com/ubi10/python-312-minimal&lt;/span&gt;
&lt;span class="c"&gt;# ... copy venv, set up /data, USER 1001&lt;/span&gt;
&lt;span class="k"&gt;CMD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;python3&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-m&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastosum.container_entrypoint&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;UBI&lt;/span&gt; minimal weighs roughly 40 &lt;span class="caps"&gt;MB&lt;/span&gt; compressed and ships with a pre-created non-root user at &lt;span class="caps"&gt;UID&lt;/span&gt; 1001. No&amp;nbsp;extra &lt;code&gt;useradd&lt;/code&gt; step,&amp;nbsp;no &lt;code&gt;shadow-utils&lt;/code&gt; dependency, and far less base-system surface area than a full distribution image. The image is signed by Red Hat and distributed&amp;nbsp;through &lt;code&gt;registry.redhat.io&lt;/code&gt;, which means the trust chain starts at a verified source before any of my code enters the&amp;nbsp;picture.&lt;/p&gt;
&lt;p&gt;The four app roles reference the moving&amp;nbsp;tag &lt;code&gt;git.linuxserver.pro/chris/mastosum:stable&lt;/code&gt; in their Quadlet files. I never edit Quadlets to deploy a new version. A release pushes both an&amp;nbsp;immutable &lt;code&gt;:v&amp;lt;version&amp;gt;&lt;/code&gt; tag (audit trail, rollback) and&amp;nbsp;re-points &lt;code&gt;:stable&lt;/code&gt; to the same digest. The Quadlets don&amp;#8217;t know or care which version is current; they just pull&amp;nbsp;whatever &lt;code&gt;:stable&lt;/code&gt; resolves&amp;nbsp;to.&lt;/p&gt;
&lt;h2 id="the-pipeline-tag-build-push-sign"&gt;The Pipeline: Tag, Build, Push,&amp;nbsp;Sign&lt;/h2&gt;
&lt;p&gt;I create releases as signed git tags. The workflow is triggered by any tag&amp;nbsp;matching &lt;code&gt;v*&lt;/code&gt;; whether the Forgejo instance or the workflow itself verifies the tag signature before building is a separate configuration choice. The important property is that the release tag carries a cryptographically verifiable identity, while the workflow that follows from it is&amp;nbsp;deterministic.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git&lt;span class="w"&gt; &lt;/span&gt;--git-dir&lt;span class="o"&gt;=&lt;/span&gt;.repo.git&lt;span class="w"&gt; &lt;/span&gt;--work-tree&lt;span class="o"&gt;=&lt;/span&gt;.&lt;span class="w"&gt; &lt;/span&gt;tag&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;v0.2.9&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Release 0.2.9&amp;quot;&lt;/span&gt;
git&lt;span class="w"&gt; &lt;/span&gt;--git-dir&lt;span class="o"&gt;=&lt;/span&gt;.repo.git&lt;span class="w"&gt; &lt;/span&gt;--work-tree&lt;span class="o"&gt;=&lt;/span&gt;.&lt;span class="w"&gt; &lt;/span&gt;push&lt;span class="w"&gt; &lt;/span&gt;origin&lt;span class="w"&gt; &lt;/span&gt;v0.2.9
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That push&amp;nbsp;triggers &lt;code&gt;.forgejo/workflows/release.yml&lt;/code&gt;, which runs on&amp;nbsp;the &lt;code&gt;mastosum-prod&lt;/code&gt; runner. The workflow has four steps, each building on the&amp;nbsp;last:&lt;/p&gt;
&lt;h3 id="1-build"&gt;1.&amp;nbsp;Build&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Build image&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;docker build \&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;--build-arg GIT_HASH=&amp;quot;$(echo &amp;quot;$GITHUB_SHA&amp;quot; | cut -c1-7)&amp;quot; \&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;-f Containerfile \&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;-t &amp;quot;${IMAGE}:${{ steps.tag.outputs.tag }}&amp;quot; \&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;-t &amp;quot;${IMAGE}:stable&amp;quot; \&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The build&amp;nbsp;uses &lt;code&gt;docker&lt;/code&gt; (the &lt;span class="caps"&gt;CLI&lt;/span&gt; binary from&amp;nbsp;the &lt;code&gt;docker.io&lt;/code&gt; package, which talks to Podman&amp;#8217;s compatible socket).&amp;nbsp;The &lt;code&gt;GIT_HASH&lt;/code&gt; build arg bakes the commit &lt;span class="caps"&gt;SHA&lt;/span&gt; into the image so&amp;nbsp;the &lt;code&gt;/health&lt;/code&gt; endpoint can report exactly which revision is running. Both tags&amp;nbsp;(&lt;code&gt;:v0.2.9&lt;/code&gt; and &lt;code&gt;:stable&lt;/code&gt;) are applied at build time to the same image. They will have the same&amp;nbsp;digest.&lt;/p&gt;
&lt;h3 id="2-push"&gt;2.&amp;nbsp;Push&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Push image&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;REGISTRY_USER&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;${{ secrets.REGISTRY_USER }}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;REGISTRY_TOKEN&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;${{ secrets.REGISTRY_TOKEN }}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;echo &amp;quot;${REGISTRY_TOKEN}&amp;quot; | docker login -u &amp;quot;${REGISTRY_USER}&amp;quot; --password-stdin git.linuxserver.pro&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;docker push &amp;quot;${IMAGE}:${{ steps.tag.outputs.tag }}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;docker push &amp;quot;${IMAGE}:stable&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two tags, same&amp;nbsp;digest. &lt;code&gt;:stable&lt;/code&gt; moves&amp;nbsp;forward; &lt;code&gt;:v0.2.9&lt;/code&gt; stays pinned forever. The registry credentials are Forgejo Actions secrets scoped to the repository, using a robot account with write-only push&amp;nbsp;access.&lt;/p&gt;
&lt;h3 id="3-sign"&gt;3.&amp;nbsp;Sign&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Sign image with cosign&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;COSIGN_KEY&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;${{ secrets.COSIGN_KEY }}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;COSIGN_PASSWORD&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;${{ secrets.COSIGN_PASSWORD }}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;umask 077&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;KEY=$(mktemp)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;printf &amp;#39;%s&amp;#39; &amp;quot;${COSIGN_KEY}&amp;quot; &amp;gt; &amp;quot;${KEY}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;DIGEST=$(docker inspect --format &amp;#39;{{ index .RepoDigests 0 }}&amp;#39; &amp;quot;${IMAGE}:${{ steps.tag.outputs.tag }}&amp;quot;)&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;cosign sign --yes --key &amp;quot;${KEY}&amp;quot; &amp;quot;${DIGEST}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;shred -u &amp;quot;${KEY}&amp;quot; 2&amp;gt;/dev/null || rm -f &amp;quot;${KEY}&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The private key material is written to a temporary file&amp;nbsp;with &lt;code&gt;umask 077&lt;/code&gt;, used exactly once, and shredded. The signature lands in the registry as&amp;nbsp;a &lt;code&gt;&amp;lt;digest&amp;gt;.sig&lt;/code&gt; tag plus a Rekor transparency-log entry. The private key itself lives only in the Forgejo Actions secret store and in an offline backup; the &lt;span class="caps"&gt;CI&lt;/span&gt; runner never sees it beyond the duration of this&amp;nbsp;step.&lt;/p&gt;
&lt;p&gt;A note on&amp;nbsp;the &lt;code&gt;docker inspect&lt;/code&gt; line: &lt;code&gt;RepoDigests[0]&lt;/code&gt; picks the first digest the local daemon knows about, which is usually correct after a fresh push but can be fragile if multiple digests are present. A safer alternative that queries the registry&amp;nbsp;directly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;DIGEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;skopeo&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{{.Digest}}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;docker://&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;TAG&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
cosign&lt;span class="w"&gt; &lt;/span&gt;sign&lt;span class="w"&gt; &lt;/span&gt;--yes&lt;span class="w"&gt; &lt;/span&gt;--key&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;KEY&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;@&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;DIGEST&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This signs the digest as observed from the registry, independent of local daemon&amp;nbsp;state.&lt;/p&gt;
&lt;h3 id="4-deploy-handoff"&gt;4. Deploy&amp;nbsp;Handoff&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Deploy to host&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;rm -f /data/deploy-status&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;printf &amp;#39;%s&amp;#39; &amp;quot;${{ steps.tag.outputs.tag }}&amp;quot; &amp;gt; /data/deploy-trigger&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;for _ in $(seq 1 60); do&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;if [ -f /data/deploy-status ]; then&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;cat /data/deploy-status&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;grep -q &amp;#39;^ok &amp;#39; /data/deploy-status &amp;amp;&amp;amp; exit 0 || exit 1&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;fi&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;sleep 5&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;done&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;echo &amp;quot;Timed out waiting for deploy completion&amp;quot; &amp;gt;&amp;amp;2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="no"&gt;exit 1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is the handoff. The runner writes the tag name&amp;nbsp;to &lt;code&gt;/data/deploy-trigger&lt;/code&gt; and polls&amp;nbsp;for &lt;code&gt;/data/deploy-status&lt;/code&gt; to appear. The file I/O is the bridge between &lt;span class="caps"&gt;CI&lt;/span&gt; and host: the runner doesn&amp;#8217;t &lt;span class="caps"&gt;SSH&lt;/span&gt; anywhere, doesn&amp;#8217;t&amp;nbsp;call &lt;code&gt;systemctl&lt;/code&gt;, doesn&amp;#8217;t touch the Podman socket for deployment purposes. It drops a file and&amp;nbsp;waits.&lt;/p&gt;
&lt;h2 id="signature-verification-design-and-current-reality"&gt;Signature Verification: Design and Current&amp;nbsp;Reality&lt;/h2&gt;
&lt;p&gt;The trust story has three layers, and it&amp;#8217;s worth separating the parts that are fully operational from the part that&amp;#8217;s waiting on registry&amp;nbsp;support.&lt;/p&gt;
&lt;h3 id="registry-authentication-operational"&gt;Registry authentication&amp;nbsp;(operational)&lt;/h3&gt;
&lt;p&gt;Forgejo requires authentication for push access. The credentials are a robot account with write-only scope on the package. The runner injects them from repository secrets. Pull access from the production host uses a read-only robot account stored&amp;nbsp;in &lt;code&gt;~/.config/containers/auth.json&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="host-side-cosign-verification-in-the-deploy-script-operational"&gt;Host-side cosign verification in the deploy script&amp;nbsp;(operational)&lt;/h3&gt;
&lt;p&gt;The deploy script (detailed in the next section)&amp;nbsp;calls &lt;code&gt;cosign verify&lt;/code&gt; against the root-installed public key before pulling the image. This is the active enforcement path: if verification fails, the deploy stops before any service is&amp;nbsp;touched.&lt;/p&gt;
&lt;h3 id="container-policy-enforcement-via-policyjson-target-state"&gt;Container policy enforcement via policy.json (target&amp;nbsp;state)&lt;/h3&gt;
&lt;p&gt;The intended long-term mechanism is&amp;nbsp;for &lt;code&gt;podman pull&lt;/code&gt; itself to reject unsigned images&amp;nbsp;via &lt;code&gt;/etc/containers/policy.json&lt;/code&gt;, without needing a&amp;nbsp;separate &lt;code&gt;cosign verify&lt;/code&gt; call:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;transports&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;docker&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;git.linuxserver.pro/chris/mastosum&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;type&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;sigstoreSigned&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;keyPath&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/etc/containers/cosign-mastosum.pub&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;signedIdentity&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;type&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;matchRepository&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The trust anchor&amp;nbsp;(&lt;code&gt;cosign-mastosum.pub&lt;/code&gt;) is installed by root&amp;nbsp;at &lt;code&gt;/etc/containers/cosign-mastosum.pub&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;mastosum&lt;/code&gt; user cannot modify it, which means even if the runner or any container in the stack is compromised, the attacker cannot swap the public key to accept a malicious image. The policy only applies&amp;nbsp;to &lt;code&gt;git.linuxserver.pro/chris/mastosum&lt;/code&gt;; the Red Hat images (Postgres, Valkey) are verified through Red Hat&amp;#8217;s own signing infrastructure via the default policy on &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;hosts.&lt;/p&gt;
&lt;p&gt;The policy is documented in the repo&amp;nbsp;(&lt;code&gt;deploy/etc/policy.json.example&lt;/code&gt;, &lt;code&gt;deploy/etc/registries.d-git.linuxserver.pro.yaml.example&lt;/code&gt;) but the actual trust material lives only on the production host, outside of version&amp;nbsp;control.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Why this isn&amp;#8217;t active yet:&lt;/strong&gt; Forgejo&amp;#8217;s container registry does not yet support the &lt;span class="caps"&gt;OCI&lt;/span&gt; 1.1 Referrers &lt;span class="caps"&gt;API&lt;/span&gt; end-to-end. cosign&amp;#8217;s &lt;span class="caps"&gt;OCI&lt;/span&gt; 1.1 referrer push fails against it,&amp;nbsp;and &lt;code&gt;use-sigstore-attachments: true&lt;/code&gt; in &lt;code&gt;registries.d&lt;/code&gt; &amp;#8212; the mechanism Podman and Skopeo use to locate sigstore signatures alongside images,  cannot locate the signature through the referrers &lt;span class="caps"&gt;API&lt;/span&gt;. In my current Forgejo&amp;nbsp;setup, &lt;code&gt;containers/image&lt;/code&gt; with &lt;code&gt;use-sigstore-attachments&lt;/code&gt; does not discover the signature reliably, even&amp;nbsp;though &lt;code&gt;cosign verify&lt;/code&gt; against the same registry succeeds. I&amp;#8217;m&amp;nbsp;treating &lt;code&gt;policy.json&lt;/code&gt; enforcement as the target state and keeping&amp;nbsp;explicit &lt;code&gt;cosign verify&lt;/code&gt; in the deploy script until the registry/referrers path works&amp;nbsp;end-to-end.&lt;/p&gt;
&lt;p&gt;In the meantime, the&amp;nbsp;explicit &lt;code&gt;cosign verify&lt;/code&gt; call in the deploy script provides equivalent enforcement. Once Forgejo&amp;#8217;s registry supports the Referrers &lt;span class="caps"&gt;API&lt;/span&gt;, the deploy script can drop that step and rely&amp;nbsp;on &lt;code&gt;policy.json&lt;/code&gt; alone. The verification setup documented above is the target state; the deploy script carries the active enforcement&amp;nbsp;today.&lt;/p&gt;
&lt;h2 id="the-runner-in-the-same-user-context"&gt;The Runner in the Same User&amp;nbsp;Context&lt;/h2&gt;
&lt;p&gt;This is the architectural decision that makes everything else simple. The Forgejo runner runs as a Quadlet container inside&amp;nbsp;the &lt;code&gt;mastosum@&lt;/code&gt; user:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# deploy/quadlet/mastosum-runner.container&lt;/span&gt;
&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;mastosum-runner&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;code.forgejo.org/forgejo/runner:12&lt;/span&gt;
&lt;span class="na"&gt;User&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;root&lt;/span&gt;
&lt;span class="na"&gt;SecurityLabelDisable&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;mastosum.network&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;%h/runner-data:/data:Z&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;%h/runner-data/config.yml:/data/config.yml:ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;%t/podman/podman.sock:%t/podman/podman.sock:rw&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;CONFIG_FILE=/data/config.yml&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;DOCKER_HOST=unix:///run/user/1000/podman/podman.sock&lt;/span&gt;
&lt;span class="na"&gt;Exec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-runner daemon --config /data/config.yml&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;User=root&lt;/code&gt; here is container root inside the rootless user namespace, not host root. The container still belongs to&amp;nbsp;the &lt;code&gt;mastosum&lt;/code&gt; user&amp;#8217;s Podman context and has no privileges beyond what that user already&amp;nbsp;holds.&lt;/p&gt;
&lt;p&gt;The runner mounts the host&amp;#8217;s rootless Podman socket at the same path it exists on the host. The&amp;nbsp;workflow&amp;#8217;s &lt;code&gt;volumes&lt;/code&gt; declaration&amp;nbsp;maps &lt;code&gt;/home/mastosum/runner-data:/data&lt;/code&gt;, and the&amp;nbsp;runner&amp;#8217;s &lt;code&gt;valid_volumes&lt;/code&gt; config permits exactly that path. The trigger file and the status file live&amp;nbsp;in &lt;code&gt;~/runner-data/&lt;/code&gt;, which the host&amp;#8217;s systemd path unit&amp;nbsp;watches.&lt;/p&gt;
&lt;p&gt;Why put the runner in the same user&amp;nbsp;context?&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;No cross-user permission problems.&lt;/strong&gt; The socket is the user&amp;#8217;s own. The data directory is the user&amp;#8217;s own. The Quadlet files are the user&amp;#8217;s own.&amp;nbsp;No &lt;code&gt;sudo&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;root&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;podman --connection&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The trigger file is local I/O.&lt;/strong&gt; The workflow doesn&amp;#8217;t need an &lt;span class="caps"&gt;SSH&lt;/span&gt; key, an &lt;span class="caps"&gt;API&lt;/span&gt; token, or a webhook receiver. It writes a file to a bind-mounted directory, and systemd picks it up. No network, no auth, no moving&amp;nbsp;parts.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;SELinux stays on.&lt;/strong&gt; The runner container&amp;nbsp;has &lt;code&gt;SecurityLabelDisable=true&lt;/code&gt; so it can talk to the Podman socket&amp;nbsp;(&lt;code&gt;container_t&lt;/code&gt; to &lt;code&gt;container_runtime_t&lt;/code&gt; is denied regardless of file labelling). The option is scoped to this one container; the host policy is untouched. Per-job containers get the same carve-out&amp;nbsp;via &lt;code&gt;container.options&lt;/code&gt; in the runner&amp;nbsp;config.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The runner is just another&amp;nbsp;service.&lt;/strong&gt; &lt;code&gt;systemctl --user restart mastosum-runner.service&lt;/code&gt; if it&amp;nbsp;hangs. &lt;code&gt;journalctl --user -u mastosum-runner.service&lt;/code&gt; if you need logs. No separate snowflake&amp;nbsp;infrastructure.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The runner does not cross into a different Unix account or host-root context, but access to the user&amp;#8217;s Podman socket is still highly privileged within&amp;nbsp;the &lt;code&gt;mastosum&lt;/code&gt; boundary: it can control every container, mount accessible user files, and affect the entire app stack. I treat the runner as part of the production trust domain, not an external&amp;nbsp;agent.&lt;/p&gt;
&lt;h2 id="the-deploy-bridge-systemd-path-unit"&gt;The Deploy Bridge: systemd Path&amp;nbsp;Unit&lt;/h2&gt;
&lt;p&gt;On the host side, a systemd path unit watches for the trigger&amp;nbsp;file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# ~/.config/systemd/user/mastosum-deploy.path&lt;/span&gt;
&lt;span class="k"&gt;[Path]&lt;/span&gt;
&lt;span class="na"&gt;PathExists&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;%h/runner-data/deploy-trigger&lt;/span&gt;
&lt;span class="na"&gt;Unit&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;mastosum-deploy.service&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;When the trigger file appears, systemd&amp;nbsp;starts &lt;code&gt;mastosum-deploy.service&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# ~/.config/systemd/user/mastosum-deploy.service&lt;/span&gt;
&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Type&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;oneshot&lt;/span&gt;
&lt;span class="na"&gt;ExecStart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/bin/bash -c &amp;#39;&lt;/span&gt;\
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;set -u; &lt;/span&gt;\
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;TAG=&amp;quot;$(cat %h/runner-data/deploy-trigger 2&amp;gt;/dev/null || echo stable)&amp;quot;; &lt;/span&gt;\
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;rm -f %h/runner-data/deploy-trigger %h/runner-data/deploy-status; &lt;/span&gt;\
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;if %h/bin/deploy &amp;quot;$TAG&amp;quot;; then &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;printf &amp;quot;ok %%s\\n&amp;quot; &amp;quot;$TAG&amp;quot; &amp;gt; %h/runner-data/deploy-status; &lt;/span&gt;\
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;else &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;rc=$?; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;printf &amp;quot;fail %%s\\n&amp;quot; &amp;quot;$TAG&amp;quot; &amp;gt; %h/runner-data/deploy-status; &lt;/span&gt;\
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s"&gt;exit &amp;quot;$rc&amp;quot;; &lt;/span&gt;\
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="s"&gt;fi&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The service reads the tag from the trigger file, removes it (so the path unit doesn&amp;#8217;t re-fire), runs the deploy script, and writes back a status file the &lt;span class="caps"&gt;CI&lt;/span&gt; workflow is polling for. If the deploy succeeds, the workflow exits 0 and the pipeline goes green. If it fails, the status file&amp;nbsp;carries &lt;code&gt;fail&lt;/code&gt; and the workflow exits&amp;nbsp;1.&lt;/p&gt;
&lt;p&gt;The deploy script&amp;nbsp;itself:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/usr/bin/env bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-euo&lt;span class="w"&gt; &lt;/span&gt;pipefail

&lt;span class="nv"&gt;TAG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;1&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;stable&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;git.linuxserver.pro/chris/mastosum&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;SERVICES&lt;/span&gt;&lt;span class="o"&gt;=(&lt;/span&gt;mastosum-web&lt;span class="w"&gt; &lt;/span&gt;mastosum-worker&lt;span class="w"&gt; &lt;/span&gt;mastosum-beat&lt;span class="w"&gt; &lt;/span&gt;mastosum-flower&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;HEALTH_TIMEOUT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;MASTOSUM_DEPLOY_HEALTH_TIMEOUT&lt;/span&gt;&lt;span class="k"&gt;:-&lt;/span&gt;&lt;span class="nv"&gt;120&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

log&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;logger&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;mastosum-deploy&lt;span class="w"&gt; &lt;/span&gt;--&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$*&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;%s\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$*&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;

log&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Starting deploy of &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;TAG&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Resolve the pushed digest and verify the cosign signature against the&lt;/span&gt;
&lt;span class="c1"&gt;# root-installed public key. If verification fails, the deploy stops&lt;/span&gt;
&lt;span class="c1"&gt;# before any service is touched.&lt;/span&gt;
&lt;span class="nv"&gt;DIGEST&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;skopeo&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{{.Digest}}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;docker://&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:stable&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;REF&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;@&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;DIGEST&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

cosign&lt;span class="w"&gt; &lt;/span&gt;verify&lt;span class="w"&gt; &lt;/span&gt;--key&lt;span class="w"&gt; &lt;/span&gt;/etc/containers/cosign-mastosum.pub&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;REF&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Pull by digest so we deploy exactly the verified reference, then tag&lt;/span&gt;
&lt;span class="c1"&gt;# it locally as :stable for the Quadlets (which reference the moving tag).&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;REF&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;tag&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;REF&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;IMAGE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;:stable&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;svc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SERVICES&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Restarting &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;svc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;svc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.service&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;

log&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Waiting up to &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;HEALTH_TIMEOUT&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;s per service for healthy status&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;svc&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SERVICES&lt;/span&gt;&lt;span class="p"&gt;[@]&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;deadline&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$((&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;+%s&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;HEALTH_TIMEOUT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;))&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;:&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{{ .State.Health.Status }}&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;svc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;missing&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;case&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;status&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;healthy&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;svc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;: healthy&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;break&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;unhealthy&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ERROR: &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;svc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; reported unhealthy&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;journalctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;svc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;.service&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--no-pager&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;logger&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;mastosum-deploy
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;*&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;+%s&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-ge&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;deadline&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;                    &lt;/span&gt;log&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ERROR: &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;svc&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; did not become healthy within &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;HEALTH_TIMEOUT&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;s&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;                    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;sleep&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="p"&gt;;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;esac&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;done&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;

log&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Deploy of &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;TAG&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt; succeeded&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few things to note about this&amp;nbsp;script:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The deploy trigger carries the release tag for logging and status reporting.&lt;/strong&gt; The script intentionally&amp;nbsp;pulls &lt;code&gt;:stable&lt;/code&gt;, not the version tag. The&amp;nbsp;immutable &lt;code&gt;:v0.2.9&lt;/code&gt; tag is the audit trail and rollback&amp;nbsp;anchor; &lt;code&gt;:stable&lt;/code&gt; is the deployment pointer. The workflow already pushed both tags to the same digest, so&amp;nbsp;pulling &lt;code&gt;:stable&lt;/code&gt; gets the same image. The trigger value&amp;nbsp;(&lt;code&gt;v0.2.9&lt;/code&gt;) appears in the deploy logs and the status file the workflow polls, but the pull target is always the moving&amp;nbsp;tag.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;cosign verification happens before the pull, and the pull targets a digest.&lt;/strong&gt; The script resolves the current digest&amp;nbsp;of &lt;code&gt;:stable&lt;/code&gt; from the registry&amp;nbsp;with &lt;code&gt;skopeo inspect&lt;/code&gt;, verifies it against the root-installed public key, pulls by that immutable digest, and then tags it locally&amp;nbsp;as &lt;code&gt;:stable&lt;/code&gt; so the Quadlets find it. This closes the race window between verification and pull: the exact verified reference is what gets deployed. If verification fails, the script exits before any service is restarted. This is the active enforcement path today; once Forgejo&amp;#8217;s registry supports &lt;span class="caps"&gt;OCI&lt;/span&gt; 1.1 referrers,&amp;nbsp;the &lt;code&gt;cosign verify&lt;/code&gt; call can be dropped in favour&amp;nbsp;of &lt;code&gt;policy.json&lt;/code&gt; enforcement at&amp;nbsp;the &lt;code&gt;podman pull&lt;/code&gt; level.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;After a successful pull, services restart in sequence&lt;/strong&gt;, and the script polls each container&amp;#8217;s health check until it&amp;nbsp;reports &lt;code&gt;healthy&lt;/code&gt; or the timeout&amp;nbsp;expires.&lt;/p&gt;
&lt;p&gt;The whole bridge &amp;#8212; workflow writes file, path unit fires, service runs script, workflow reads result &amp;#8212; is a pattern I&amp;#8217;ve come to appreciate for its simplicity. There&amp;#8217;s no webhook receiver to secure, no &lt;span class="caps"&gt;API&lt;/span&gt; to authenticate, no network port to expose. The filesystem is the &lt;span class="caps"&gt;API&lt;/span&gt;.&lt;/p&gt;
&lt;h2 id="rollback"&gt;Rollback&lt;/h2&gt;
&lt;p&gt;The immutable per-version tags make rollback&amp;nbsp;straightforward:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# From any machine with push access:&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastosum:v0.2.8
podman&lt;span class="w"&gt; &lt;/span&gt;tag&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastosum:v0.2.8&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastosum:stable
podman&lt;span class="w"&gt; &lt;/span&gt;push&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastosum:stable

&lt;span class="c1"&gt;# Then trigger a deploy from the host:&lt;/span&gt;
ssh&lt;span class="w"&gt; &lt;/span&gt;mastosum@prod&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;echo v0.2.8 &amp;gt; ~/runner-data/deploy-trigger&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Move &lt;code&gt;:stable&lt;/code&gt; back to the last known good version, write the trigger file, and the same path unit + deploy script takes over. The deploy&amp;nbsp;script&amp;#8217;s &lt;code&gt;cosign verify&lt;/code&gt; step still works because the old version tag was also signed. No special rollback procedure, no database migration reversal (the app&amp;#8217;s startup migrations are additive and forward-only by design). The Quadlets don&amp;#8217;t change. The environment file doesn&amp;#8217;t change. Only the image digest&amp;nbsp;that &lt;code&gt;:stable&lt;/code&gt; points to&amp;nbsp;changes.&lt;/p&gt;
&lt;h2 id="the-full-picture"&gt;The Full&amp;nbsp;Picture&lt;/h2&gt;
&lt;p&gt;Putting it all together, the flow&amp;nbsp;from &lt;code&gt;git push&lt;/code&gt; to healthy&amp;nbsp;containers:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;You push a version&amp;nbsp;tag.&lt;/strong&gt; &lt;code&gt;git push origin v0.2.9&lt;/code&gt;. The tag is signed locally; the workflow is triggered by its&amp;nbsp;presence.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Forgejo fires the workflow.&lt;/strong&gt;&amp;nbsp;The &lt;code&gt;mastosum-prod&lt;/code&gt; runner picks it&amp;nbsp;up.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The workflow builds.&lt;/strong&gt; &lt;span class="caps"&gt;UBI10&lt;/span&gt; base, Python venv, GIT_HASH baked&amp;nbsp;in.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The workflow&amp;nbsp;pushes&lt;/strong&gt; &lt;code&gt;:v0.2.9&lt;/code&gt; and &lt;code&gt;:stable&lt;/code&gt; to the Forgejo registry. Both tags point to the same&amp;nbsp;digest.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The workflow signs&lt;/strong&gt; the digest with cosign. Private key is shredded after use. The signature lands as&amp;nbsp;a &lt;code&gt;&amp;lt;digest&amp;gt;.sig&lt;/code&gt; tag and a Rekor transparency-log&amp;nbsp;entry.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The workflow&amp;nbsp;writes&lt;/strong&gt; &lt;code&gt;v0.2.9&lt;/code&gt; to &lt;code&gt;/home/mastosum/runner-data/deploy-trigger&lt;/code&gt; and polls for the status&amp;nbsp;file.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;systemd sees the trigger&amp;nbsp;file.&lt;/strong&gt; &lt;code&gt;mastosum-deploy.path&lt;/code&gt; fires &lt;code&gt;mastosum-deploy.service&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The deploy script runs.&lt;/strong&gt; It resolves&amp;nbsp;the &lt;code&gt;:stable&lt;/code&gt; digest, verifies the cosign signature against the root-installed public key, pulls the image by verified digest, tags it locally&amp;nbsp;as &lt;code&gt;:stable&lt;/code&gt;, restarts the four app services, and waits for each to report&amp;nbsp;healthy.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The deploy script&amp;nbsp;writes&lt;/strong&gt; &lt;code&gt;ok v0.2.9&lt;/code&gt; to &lt;code&gt;/home/mastosum/runner-data/deploy-status&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The workflow reads the status file&lt;/strong&gt;,&amp;nbsp;sees &lt;code&gt;ok&lt;/code&gt;, and exits&amp;nbsp;0.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;No manual steps. No &lt;span class="caps"&gt;SSH&lt;/span&gt; from &lt;span class="caps"&gt;CI&lt;/span&gt; to host. No webhook receiver. No additional daemon. The &lt;span class="caps"&gt;CI&lt;/span&gt; runner, the deploy machinery, and the application itself all run as the same user, in the same Podman context, managed by the same&amp;nbsp;systemd.&lt;/p&gt;
&lt;p&gt;The pieces are simple individually. The value is in how they compose: a tag push triggers a signed build, which writes a file, which systemd reacts to, which verifies the signature and pulls and restarts. Each link does one thing, and the filesystem is the interface between&amp;nbsp;them.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman in Production: Quadlets, Secrets, Auto-Updates&lt;/a&gt; &amp;#8212; the Podman architecture this deployment rests&amp;nbsp;on&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/"&gt;Ansible-Native Quadlets: Deploying a Mastodon Greeter Bot&lt;/a&gt; &amp;#8212; Quadlet generation and Podman secrets in&amp;nbsp;Ansible&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Container Forgejo Runner with Podman Quadlet&lt;/a&gt; &amp;#8212; the Quadlet pattern the runner&amp;nbsp;uses&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/speeding-up-forgejo-ci-with-a-custom-oci-image/"&gt;Speeding Up Forgejo &lt;span class="caps"&gt;CI&lt;/span&gt; with a Custom &lt;span class="caps"&gt;OCI&lt;/span&gt; Image&lt;/a&gt; &amp;#8212; Forgejo&amp;#8217;s built-in &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;registry&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.sigstore.dev/signing/quickstart/"&gt;cosign&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/containers/image/blob/main/docs/containers-policy.json.5.md"&gt;containers-policy.json(5)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="forgejo"/><category term="ci-cd"/><category term="podman"/><category term="quadlet"/><category term="containers"/><category term="deployment"/><category term="cosign"/><category term="ubi"/><category term="redhat"/><category term="systemd"/></entry><entry><title>Fedora Hummingbird: Distroless, Container-Native Linux and a Slim mastogreet</title><link href="https://hofstede.it/fedora-hummingbird-distroless-container-native-linux-and-a-slim-mastogreet/" rel="alternate"/><published>2026-05-16T00:00:00+02:00</published><updated>2026-05-16T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-05-16:/fedora-hummingbird-distroless-container-native-linux-and-a-slim-mastogreet/</id><summary type="html">&lt;p&gt;Fedora Hummingbird brings the distroless container model to a full, image-based rolling operating system, and it ships a catalog of hardened language images alongside it. I rebuild the mastogreet bot&amp;#8217;s Containerfile on top&amp;nbsp;of &lt;code&gt;registry.access.redhat.com/hi/python&lt;/code&gt; to see what changes, what shrinks, and where the commercial Red Hat Hardened Images downstream fits&amp;nbsp;in.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Header image" src="https://hofstede.it/images/2026-05-16-fedora-hummingbird-distroless-mastogreet.png" title="Fedora Hummingbird logo next to a slimmed-down Python container"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-fedora-hummingbird-actually-refers-to"&gt;What &amp;#8220;Fedora Hummingbird&amp;#8221; Actually Refers To&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#project-hummingbird-the-image-catalog"&gt;Project Hummingbird: the image&amp;nbsp;catalog&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#fedora-hummingbird-linux-the-operating-system"&gt;Fedora Hummingbird Linux: the operating&amp;nbsp;system&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-agentic-framing"&gt;The agentic&amp;nbsp;framing&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-the-model-is-useful"&gt;Why the Model Is&amp;nbsp;Useful&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-image-catalog-and-the-registry-layout"&gt;The Image Catalog and the Registry&amp;nbsp;Layout&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rebuilding-mastogreet-on-hummingbird-python"&gt;Rebuilding mastogreet on Hummingbird Python&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#array-syntax-everywhere"&gt;Array Syntax&amp;nbsp;Everywhere&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#no-useradd-because-theres-a-user-already"&gt;No useradd, Because There&amp;#8217;s a User&amp;nbsp;Already&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#virtual-environment-as-the-transport-mechanism"&gt;Virtual Environment as the Transport&amp;nbsp;Mechanism&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-state-directory-pre-owned"&gt;The State Directory,&amp;nbsp;Pre-Owned&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#selinux-and-quadlet-compatibility"&gt;SELinux and Quadlet&amp;nbsp;Compatibility&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-you-actually-get"&gt;What You Actually&amp;nbsp;Get&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rootless-smoke-test-with-the-footgun"&gt;Rootless Smoke Test, with the&amp;nbsp;Footgun&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#day-to-day-differences"&gt;Day-to-Day&amp;nbsp;Differences&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#updating-the-ansible-role"&gt;Updating the Ansible&amp;nbsp;Role&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#red-hat-hardened-images-the-commercial-downstream"&gt;Red Hat Hardened Images: The Commercial&amp;nbsp;Downstream&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-brief-note-on-fedora-hummingbird-the-os"&gt;A Brief Note on Fedora Hummingbird the &lt;span class="caps"&gt;OS&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#when-this-pays-off-and-when-it-doesnt"&gt;When This Pays Off and When It&amp;nbsp;Doesn&amp;#8217;t&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;A few weeks ago I wrote up how I deploy &lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/"&gt;&lt;code&gt;mastogreet&lt;/code&gt;&lt;/a&gt; onto my &lt;span class="caps"&gt;RHEL&lt;/span&gt; hosts with &lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/"&gt;&lt;code&gt;containers.podman&lt;/code&gt; and Quadlet generation&lt;/a&gt;. The Ansible side is genuinely tidy. The base image, on the other hand,&amp;nbsp;is &lt;code&gt;python:3.12-slim&lt;/code&gt; from Docker Hub: a Debian-flavored Python with a full&amp;nbsp;shell, &lt;code&gt;apt&lt;/code&gt;, &lt;span class="caps"&gt;GNU&lt;/span&gt; coreutils, and a long tail of binaries the bot will never touch. &amp;#8220;Slim&amp;#8221; is&amp;nbsp;generous.&lt;/p&gt;
&lt;p&gt;This article is the rebuild. I took the&amp;nbsp;same &lt;code&gt;mastogreet&lt;/code&gt; and put it on top&amp;nbsp;of &lt;code&gt;registry.access.redhat.com/hi/python:3.13&lt;/code&gt;, a distroless Python from Project Hummingbird. The runtime image came out roughly 10% smaller (126 &lt;span class="caps"&gt;MB&lt;/span&gt; versus 140 &lt;span class="caps"&gt;MB&lt;/span&gt; on amd64), which turns out to be the least interesting result. The more meaningful changes are qualitative: no shell, no package manager, a signed &lt;span class="caps"&gt;SBOM&lt;/span&gt;, and one &lt;span class="caps"&gt;UID&lt;/span&gt;-mapping footgun I didn&amp;#8217;t see coming until the first smoke-test run. The Ansible role needed a two-line variable diff and zero task&amp;nbsp;changes.&lt;/p&gt;
&lt;p&gt;In practice, &amp;#8220;distroless&amp;#8221; here means the runtime image contains only the Python interpreter and the libraries the application links against.&amp;nbsp;No &lt;code&gt;/bin/bash&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;dnf&lt;/code&gt;, no coreutils, no incidental tooling. Everything that would let an attacker spelunk after they land a code-execution primitive is simply not there to spelunk&amp;nbsp;with.&lt;/p&gt;
&lt;p&gt;The rest of this article walks through what the Hummingbird ecosystem is, the rebuild itself, the smoke-test footgun, and where Red Hat Hardened Images (the commercial downstream) fits&amp;nbsp;in.&lt;/p&gt;
&lt;h2 id="what-fedora-hummingbird-actually-refers-to"&gt;What &amp;#8220;Fedora Hummingbird&amp;#8221; Actually Refers&amp;nbsp;To&lt;/h2&gt;
&lt;p&gt;There are two related things that share the Hummingbird name. It&amp;#8217;s worth keeping them separate before going any further; the rest of the article uses both terms with&amp;nbsp;intent.&lt;/p&gt;
&lt;h3 id="project-hummingbird-the-image-catalog"&gt;Project Hummingbird: the image&amp;nbsp;catalog&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Project Hummingbird&lt;/strong&gt; is the upstream community building the image catalog. As of the Summit announcement it had roughly 49 unique minimal, hardened, distroless container images, multiplied to about 157 variants once you count &lt;span class="caps"&gt;FIPS&lt;/span&gt; builds and multi-arch (x86_64 and aarch64). Coverage spans Python, Go, Node.js, Rust, Ruby, OpenJDK, .&lt;span class="caps"&gt;NET&lt;/span&gt;, PostgreSQL, MariaDB, nginx, Caddy,&amp;nbsp;a &lt;code&gt;core-runtime&lt;/code&gt; base for compiled binaries, and a few dozen&amp;nbsp;others.&lt;/p&gt;
&lt;p&gt;The build pipeline is &lt;a href="https://konflux-ci.dev/"&gt;Konflux&lt;/a&gt;, with hermetic builds, pinned package lists, continuous scanning via Syft and Grype, &lt;span class="caps"&gt;SBOM&lt;/span&gt; generation, and image signatures. Roughly 95% of the content is Fedora Rawhide packages used unmodified, so the security story isn&amp;#8217;t &amp;#8220;we forked everything,&amp;#8221; it&amp;#8217;s &amp;#8220;we built a tight pipeline around&amp;nbsp;upstream.&amp;#8221;&lt;/p&gt;
&lt;h3 id="fedora-hummingbird-linux-the-operating-system"&gt;Fedora Hummingbird Linux: the operating&amp;nbsp;system&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Fedora Hummingbird Linux&lt;/strong&gt; is the new image-based rolling Linux distribution that takes the same model up a level. The root filesystem is read-only, writable state is confined&amp;nbsp;to &lt;code&gt;/var&lt;/code&gt; and &lt;code&gt;/etc&lt;/code&gt;, updates are atomic with rollback, and the whole &lt;span class="caps"&gt;OS&lt;/span&gt; is delivered as an &lt;span class="caps"&gt;OCI&lt;/span&gt; image you can pull and boot via &lt;a href="https://containers.github.io/bootc/"&gt;&lt;code&gt;bootc&lt;/code&gt;&lt;/a&gt;. The kernel is the &lt;a href="https://gitlab.com/cki-project"&gt;Always Ready Kernel (&lt;span class="caps"&gt;ARK&lt;/span&gt;)&lt;/a&gt; from the &lt;span class="caps"&gt;CKI&lt;/span&gt; project, tracking Linus&amp;#8217;&amp;nbsp;mainline.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;OS&lt;/span&gt; images currently live&amp;nbsp;at &lt;code&gt;quay.io/hummingbird-community/bootc-os&lt;/code&gt;. The Fedora Council has granted the trademark, and integration into Fedora proper is happening through a &lt;span class="caps"&gt;SIG&lt;/span&gt;.&lt;/p&gt;
&lt;h3 id="the-agentic-framing"&gt;The agentic&amp;nbsp;framing&lt;/h3&gt;
&lt;p&gt;The marketing layer on top of both, in Red Hat&amp;#8217;s &lt;a href="https://www.redhat.com/en/about/press-releases/fedora-hummingbird-linux-brings-agentic-linux-builders"&gt;press release&lt;/a&gt;, frames Hummingbird as the &lt;span class="caps"&gt;OS&lt;/span&gt; for &amp;#8220;agent-first builders&amp;#8221; in the &amp;#8220;agentic era.&amp;#8221; I suspect that framing is mostly aimed at making the project legible to current industry conversations, and I&amp;#8217;d argue the same properties (small attack surface, hermetic builds, fast &lt;span class="caps"&gt;CVE&lt;/span&gt; turnover, registration-free deployment, machine-readable SBOMs) are also exactly what a non-agentic ops team has wanted from container base images for a decade. So: useful regardless of whether the workload&amp;nbsp;calling &lt;code&gt;podman run&lt;/code&gt; happens to be a person, a &lt;span class="caps"&gt;CI&lt;/span&gt; job, or an &lt;span class="caps"&gt;LLM&lt;/span&gt;. The distribution itself doesn&amp;#8217;t&amp;nbsp;care.&lt;/p&gt;
&lt;h2 id="why-the-model-is-useful"&gt;Why the Model Is&amp;nbsp;Useful&lt;/h2&gt;
&lt;p&gt;The Hummingbird containers are distroless in a stronger sense&amp;nbsp;than &lt;code&gt;python:3.12-slim&lt;/code&gt;. No package manager,&amp;nbsp;no &lt;code&gt;/bin/sh&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;/bin/bash&lt;/code&gt;, no shell utilities. The runtime image carries the language interpreter, its supporting libraries, the &lt;span class="caps"&gt;RPM&lt;/span&gt; metadata for the packages that are present (so vulnerability scanners can still introspect), and almost nothing else. The often-cited nginx comparison is illustrative: the Docker&amp;nbsp;Hub &lt;code&gt;nginx&lt;/code&gt; image has roughly 270 binaries&amp;nbsp;in &lt;code&gt;/bin/&lt;/code&gt;; the&amp;nbsp;Hummingbird &lt;code&gt;nginx&lt;/code&gt; image has&amp;nbsp;31.&lt;/p&gt;
&lt;p&gt;A handful of properties fall out of&amp;nbsp;that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Smaller attack surface.&lt;/strong&gt; No shell means an attacker who lands a code execution primitive in your Python process can&amp;#8217;t reach&amp;nbsp;for &lt;code&gt;/bin/sh -c&lt;/code&gt;, can&amp;#8217;t enumerate the filesystem with familiar tooling, and&amp;nbsp;can&amp;#8217;t &lt;code&gt;curl&lt;/code&gt; a second-stage payload. None of that is a substitute for not having a code execution primitive, but it raises the&amp;nbsp;floor.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Near-zero &lt;span class="caps"&gt;CVE&lt;/span&gt; goal.&lt;/strong&gt; Because the package list is pinned and the pipeline scans continuously with Syft and Grype, the catalog&amp;#8217;s stated objective is &amp;#8220;zero &lt;span class="caps"&gt;CVE&lt;/span&gt; reports in every image&amp;#8221; at any given point in time. In practice that means &lt;span class="caps"&gt;CVE&lt;/span&gt; fixes flow through within hours to days, not the weeks-or-months you sometimes see on community&amp;nbsp;images.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;SBOMs and signatures by default.&lt;/strong&gt; Each image ships an &lt;span class="caps"&gt;SBOM&lt;/span&gt; you can pull&amp;nbsp;with &lt;code&gt;cosign download sbom&lt;/code&gt;, and images are signed. For supply-chain-conscious environments this is the difference between &amp;#8220;trust me&amp;#8221; and&amp;nbsp;&amp;#8220;verify.&amp;#8221;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Hermetic, reproducible builds.&lt;/strong&gt; Builds happen in isolation from arbitrary network access, against pinned package lists. The same input produces the same&amp;nbsp;output.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Chunkah re-layering.&lt;/strong&gt; Traditional &lt;span class="caps"&gt;OCI&lt;/span&gt; images bundle many packages into a handful of large filesystem layers, which means changing one package invalidates the whole layer&amp;#8217;s cache. Hummingbird&amp;#8217;s chunkah tool re-cuts those layers into smaller content-addressable chunks grouped by package origin, so updates and related images share more cached content. The visible side of this is the 33-layer pull you&amp;#8217;ll see in the build output below; the practical side is that a &lt;span class="caps"&gt;CVE&lt;/span&gt; refresh only re-downloads the chunks that actually&amp;nbsp;changed.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The trade-off is real: you&amp;nbsp;can&amp;#8217;t &lt;code&gt;podman exec -it&lt;/code&gt; into a running container and poke around with a shell, because there is no shell. You also&amp;nbsp;can&amp;#8217;t &lt;code&gt;RUN apt-get install ...&lt;/code&gt; your way to a dependency in production. Both of those are intentional, and both of them push you toward multi-stage builds and proper observability. We&amp;#8217;ll get to&amp;nbsp;that.&lt;/p&gt;
&lt;h2 id="the-image-catalog-and-the-registry-layout"&gt;The Image Catalog and the Registry&amp;nbsp;Layout&lt;/h2&gt;
&lt;p&gt;The catalog is&amp;nbsp;at &lt;code&gt;registry.access.redhat.com/hi/&lt;/code&gt;. Python&amp;nbsp;is &lt;code&gt;registry.access.redhat.com/hi/python&lt;/code&gt;. Tags work the way you&amp;#8217;d&amp;nbsp;hope:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;:latest&lt;/code&gt; follows the newest supported Python (currently&amp;nbsp;3.14)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;:3.13&lt;/code&gt;, &lt;code&gt;:3.12&lt;/code&gt;, &lt;code&gt;:3.11&lt;/code&gt; stay pinned to those&amp;nbsp;branches&lt;/li&gt;
&lt;li&gt;&lt;code&gt;:3.13-builder&lt;/code&gt; is the same Python but&amp;nbsp;with &lt;code&gt;dnf&lt;/code&gt;, &lt;code&gt;bash&lt;/code&gt;, and friends, intended for multi-stage build&amp;nbsp;stages&lt;/li&gt;
&lt;li&gt;&lt;code&gt;:3.13-fips&lt;/code&gt; is the &lt;span class="caps"&gt;FIPS&lt;/span&gt;-validated variant for compliance-bound&amp;nbsp;environments&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Equivalents exist for Node.js&amp;nbsp;(&lt;code&gt;hi/nodejs&lt;/code&gt;), Go&amp;nbsp;(&lt;code&gt;hi/go&lt;/code&gt;), Java/OpenJDK&amp;nbsp;(&lt;code&gt;hi/openjdk&lt;/code&gt;), and so on. There&amp;#8217;s&amp;nbsp;also &lt;code&gt;hi/core-runtime&lt;/code&gt; for compiled binaries that don&amp;#8217;t need a language runtime at&amp;nbsp;all.&lt;/p&gt;
&lt;p&gt;A few alternative registries exist for the same content with different trust&amp;nbsp;properties:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;quay.io/hummingbird/&lt;/code&gt; mirrors the catalog without Red Hat&amp;#8217;s&amp;nbsp;signing&lt;/li&gt;
&lt;li&gt;&lt;code&gt;quay.io/hummingbird-community/&lt;/code&gt; is the community-supported entry&amp;nbsp;point&lt;/li&gt;
&lt;li&gt;&lt;code&gt;quay.io/hummingbird-rawhide/&lt;/code&gt; follows Fedora Rawhide&amp;nbsp;directly&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For anything I run on real hosts I want the signed copies,&amp;nbsp;so &lt;code&gt;registry.access.redhat.com/hi/&lt;/code&gt; it&amp;nbsp;is.&lt;/p&gt;
&lt;h2 id="rebuilding-mastogreet-on-hummingbird-python"&gt;Rebuilding mastogreet on Hummingbird&amp;nbsp;Python&lt;/h2&gt;
&lt;p&gt;Time for a concrete example. The &lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/#the-container-briefly"&gt;original &lt;code&gt;mastogreet&lt;/code&gt; Containerfile&lt;/a&gt; is&amp;nbsp;straightforward:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;python:3.12-slim&lt;/span&gt;

&lt;span class="k"&gt;ARG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;UID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10001&lt;/span&gt;
&lt;span class="k"&gt;ARG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;GID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10001&lt;/span&gt;

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;groupadd&lt;span class="w"&gt; &lt;/span&gt;--system&lt;span class="w"&gt; &lt;/span&gt;--gid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;GID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;mastogreet&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;useradd&lt;span class="w"&gt;  &lt;/span&gt;--system&lt;span class="w"&gt; &lt;/span&gt;--uid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;UID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--gid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;GID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--home-dir&lt;span class="w"&gt; &lt;/span&gt;/app&lt;span class="w"&gt; &lt;/span&gt;--no-create-home&lt;span class="w"&gt; &lt;/span&gt;mastogreet

&lt;span class="k"&gt;WORKDIR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/app&lt;/span&gt;

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;requirements.txt&lt;span class="w"&gt; &lt;/span&gt;.
&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--no-cache-dir&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;requirements.txt

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;mastowelcome.py&lt;span class="w"&gt; &lt;/span&gt;.

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;mastogreet&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;mastogreet&lt;span class="w"&gt; &lt;/span&gt;/data
&lt;span class="k"&gt;VOLUME&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/data&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;USER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet&lt;/span&gt;

&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PYTHONUNBUFFERED&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_CONFIG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/data/config.yaml&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_DB&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/data/state.db
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;It works. It builds in seconds. The image weighs about 140 &lt;span class="caps"&gt;MB&lt;/span&gt; on disk&amp;nbsp;because &lt;code&gt;python:3.12-slim&lt;/code&gt; carries Debian&amp;#8217;s&amp;nbsp;userland, &lt;code&gt;pip&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s build tooling, libraries the bot never imports, and a full shell so&amp;nbsp;the &lt;code&gt;RUN groupadd &amp;amp;&amp;amp; useradd&lt;/code&gt; line above can even&amp;nbsp;exist.&lt;/p&gt;
&lt;p&gt;Here is the same bot on top&amp;nbsp;of &lt;code&gt;hi/python&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c"&gt;# Build stage: includes dnf, bash, build tools. Never ships to production.&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;registry.access.redhat.com/hi/python:3.13-builder&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;builder&lt;/span&gt;

&lt;span class="k"&gt;USER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;python3&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-m&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;venv&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/opt/venv&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/opt/venv/bin:&lt;/span&gt;&lt;span class="nv"&gt;$PATH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;requirements.txt&lt;span class="w"&gt; &lt;/span&gt;/tmp/requirements.txt
&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pip3&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;install&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--no-cache-dir&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-r&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/tmp/requirements.txt&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;

&lt;span class="c"&gt;# Pre-create the /data skeleton with the right ownership so the&lt;/span&gt;
&lt;span class="c"&gt;# runtime stage doesn&amp;#39;t need a shell or coreutils to do it itself.&lt;/span&gt;
&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;install&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-d&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-o&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;65532&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-g&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;65532&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/data-skel&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;

&lt;span class="c"&gt;# Runtime stage: distroless. No shell, no dnf, no surprises.&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;registry.access.redhat.com/hi/python:3.13&lt;/span&gt;

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--from&lt;span class="o"&gt;=&lt;/span&gt;builder&lt;span class="w"&gt; &lt;/span&gt;/opt/venv&lt;span class="w"&gt; &lt;/span&gt;/opt/venv
&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--from&lt;span class="o"&gt;=&lt;/span&gt;builder&lt;span class="w"&gt; &lt;/span&gt;--chown&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;65532&lt;/span&gt;:65532&lt;span class="w"&gt; &lt;/span&gt;/data-skel&lt;span class="w"&gt; &lt;/span&gt;/data
&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--chown&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;65532&lt;/span&gt;:65532&lt;span class="w"&gt; &lt;/span&gt;mastowelcome.py&lt;span class="w"&gt; &lt;/span&gt;/app/mastowelcome.py

&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PATH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/opt/venv/bin:&lt;/span&gt;&lt;span class="nv"&gt;$PATH&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;PYTHONUNBUFFERED&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;PYTHONDONTWRITEBYTECODE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_CONFIG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/data/config.yaml&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_DB&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/data/state.db

&lt;span class="k"&gt;WORKDIR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/app&lt;/span&gt;
&lt;span class="k"&gt;USER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;65532&lt;/span&gt;
&lt;span class="k"&gt;VOLUME&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/data&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;CMD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;python3&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;mastowelcome.py&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few details are worth pulling&amp;nbsp;apart.&lt;/p&gt;
&lt;h3 id="array-syntax-everywhere"&gt;Array Syntax&amp;nbsp;Everywhere&lt;/h3&gt;
&lt;p&gt;Every &lt;code&gt;RUN&lt;/code&gt; instruction uses the &lt;span class="caps"&gt;JSON&lt;/span&gt; array form, not the shell-string form. The shell form&amp;nbsp;(&lt;code&gt;RUN pip install -r requirements.txt&lt;/code&gt;) implicitly&amp;nbsp;invokes &lt;code&gt;/bin/sh -c&lt;/code&gt;. The Hummingbird runtime image doesn&amp;#8217;t have a shell, and the builder image has one but it&amp;#8217;s good practice to write the Containerfile so that the same instructions would work if you moved them to the runtime stage. Same logic&amp;nbsp;for &lt;code&gt;CMD ["python3", "mastowelcome.py"]&lt;/code&gt;: array form, no shell&amp;nbsp;needed.&lt;/p&gt;
&lt;p&gt;This is the most common porting mistake people hit: a perfectly&amp;nbsp;reasonable &lt;code&gt;RUN echo "PATH=/opt/venv/bin" &amp;gt;&amp;gt; /etc/profile&lt;/code&gt; falls over in the runtime image, and the error message is the&amp;nbsp;cheerful &lt;code&gt;executable file not found in $PATH: bash&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="no-useradd-because-theres-a-user-already"&gt;No &lt;code&gt;useradd&lt;/code&gt;, Because There&amp;#8217;s a User&amp;nbsp;Already&lt;/h3&gt;
&lt;p&gt;The original Containerfile created a&amp;nbsp;custom &lt;code&gt;mastogreet:10001&lt;/code&gt; user. The Hummingbird images already ship a non-root user at &lt;span class="caps"&gt;UID&lt;/span&gt; 65532 by default, which is the convention Google&amp;#8217;s distroless project popularised and which Hummingbird&amp;nbsp;adopted.&lt;/p&gt;
&lt;p&gt;Adding a second non-root user is only possible in the builder stage (since the runtime has&amp;nbsp;no &lt;code&gt;useradd&lt;/code&gt;), and would force me to either&amp;nbsp;copy &lt;code&gt;/etc/passwd&lt;/code&gt; and &lt;code&gt;/etc/group&lt;/code&gt; across stages or&amp;nbsp;set &lt;code&gt;USER&lt;/code&gt; numerically in the runtime&amp;nbsp;image.&lt;/p&gt;
&lt;p&gt;Easier and cleaner: drop the custom &lt;span class="caps"&gt;UID&lt;/span&gt;, use the one the image gives me, and chown&amp;nbsp;the &lt;code&gt;/data&lt;/code&gt; skeleton to 65532 in the builder stage. On the Ansible&amp;nbsp;side, &lt;code&gt;mastogreet_container_uid: 65532&lt;/code&gt; falls out as a single variable change and everything else still&amp;nbsp;composes.&lt;/p&gt;
&lt;h3 id="virtual-environment-as-the-transport-mechanism"&gt;Virtual Environment as the Transport&amp;nbsp;Mechanism&lt;/h3&gt;
&lt;p&gt;The builder stage installs dependencies&amp;nbsp;into &lt;code&gt;/opt/venv&lt;/code&gt;. The runtime stage&amp;nbsp;receives &lt;code&gt;/opt/venv&lt;/code&gt; whole,&amp;nbsp;via &lt;code&gt;COPY --from=builder&lt;/code&gt;. This is the idiomatic Hummingbird Python pattern, and it&amp;#8217;s there for a&amp;nbsp;reason: &lt;code&gt;pip&lt;/code&gt; is technically retained in the runtime image (the catalog kept it for compatibility with how interpreted-language workflows expect to behave), but you really don&amp;#8217;t want to invoke it there, because you have no shell to wrap it in and no compilers to build native extensions. The venv keeps all the work in the builder stage where the tools exist, and ships a self-contained directory the runtime can just&amp;nbsp;point &lt;code&gt;PATH&lt;/code&gt; at.&lt;/p&gt;
&lt;p&gt;If a dependency needs C compilation, the builder stage is also&amp;nbsp;where &lt;code&gt;dnf install -y gcc python3-devel libffi-devel&lt;/code&gt; happens; none of that ships to&amp;nbsp;production.&lt;/p&gt;
&lt;h3 id="the-state-directory-pre-owned"&gt;The State Directory,&amp;nbsp;Pre-Owned&lt;/h3&gt;
&lt;p&gt;In the original&amp;nbsp;Containerfile, &lt;code&gt;RUN install -d -o mastogreet -g mastogreet /data&lt;/code&gt; ran in the final image because Debian&amp;nbsp;has &lt;code&gt;install(1)&lt;/code&gt; everywhere. In Hummingbird, that command runs in the builder stage and writes to&amp;nbsp;a &lt;code&gt;/data-skel&lt;/code&gt; path, which then gets copied&amp;nbsp;to &lt;code&gt;/data&lt;/code&gt; in the runtime stage&amp;nbsp;with &lt;code&gt;COPY --from=builder --chown=65532:65532&lt;/code&gt;. Same effect, but the runtime image never had to host&amp;nbsp;coreutils.&lt;/p&gt;
&lt;h3 id="selinux-and-quadlet-compatibility"&gt;SELinux and Quadlet&amp;nbsp;Compatibility&lt;/h3&gt;
&lt;p&gt;Nothing about the bind-mount story changes. The host still&amp;nbsp;mounts &lt;code&gt;/opt/mastogreet/&lt;/code&gt; to &lt;code&gt;/data&lt;/code&gt; with&amp;nbsp;the &lt;code&gt;:Z&lt;/code&gt; flag, and the Ansible role still ensures the host-side directory is owned by 65532 instead of 10001. The Quadlet unit generated&amp;nbsp;by &lt;code&gt;containers.podman.podman_container&lt;/code&gt; is identical in shape to the one in &lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/#step-6-generating-the-quadlet"&gt;the previous article&lt;/a&gt;; only the variables&amp;nbsp;move.&lt;/p&gt;
&lt;h3 id="what-you-actually-get"&gt;What You Actually&amp;nbsp;Get&lt;/h3&gt;
&lt;p&gt;Building it for&amp;nbsp;real:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastogreet:hummingbird&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;Containerfile-hi&lt;span class="w"&gt; &lt;/span&gt;.
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;/2&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;STEP&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;/7:&lt;span class="w"&gt; &lt;/span&gt;FROM&lt;span class="w"&gt; &lt;/span&gt;registry.access.redhat.com/hi/python:3.13-builder&lt;span class="w"&gt; &lt;/span&gt;AS&lt;span class="w"&gt; &lt;/span&gt;builder
Trying&lt;span class="w"&gt; &lt;/span&gt;to&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;registry.access.redhat.com/hi/python:3.13-builder...
...&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="m"&gt;33&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;chunkah&lt;span class="w"&gt; &lt;/span&gt;layers&lt;span class="w"&gt; &lt;/span&gt;stream&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;/2&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;STEP&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;6&lt;/span&gt;/7:&lt;span class="w"&gt; &lt;/span&gt;RUN&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;pip3&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;install&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;--no-cache-dir&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-r&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/tmp/requirements.txt&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
Collecting&lt;span class="w"&gt; &lt;/span&gt;Mastodon.py&amp;lt;&lt;span class="m"&gt;2&lt;/span&gt;,&amp;gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;.8&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;from&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;/tmp/requirements.txt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;line&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="o"&gt;))&lt;/span&gt;
...
Successfully&lt;span class="w"&gt; &lt;/span&gt;installed&lt;span class="w"&gt; &lt;/span&gt;Mastodon.py-1.8.1&lt;span class="w"&gt; &lt;/span&gt;PyYAML-6.0.3&lt;span class="w"&gt; &lt;/span&gt;blurhash-1.1.5
certifi-2026.4.22&lt;span class="w"&gt; &lt;/span&gt;charset_normalizer-3.4.7&lt;span class="w"&gt; &lt;/span&gt;decorator-5.2.1&lt;span class="w"&gt; &lt;/span&gt;idna-3.15
python-dateutil-2.9.0.post0&lt;span class="w"&gt; &lt;/span&gt;python-magic-0.4.27&lt;span class="w"&gt; &lt;/span&gt;requests-2.34.2
six-1.17.0&lt;span class="w"&gt; &lt;/span&gt;urllib3-2.7.0
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;/2&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;STEP&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;7&lt;/span&gt;/7:&lt;span class="w"&gt; &lt;/span&gt;RUN&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;install&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-d&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-o&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;65532&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;-g&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;65532&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/data-skel&amp;quot;&lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;/2&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;STEP&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;/9:&lt;span class="w"&gt; &lt;/span&gt;FROM&lt;span class="w"&gt; &lt;/span&gt;registry.access.redhat.com/hi/python:3.13
...
&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;/2&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;COMMIT&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastogreet:hummingbird
Successfully&lt;span class="w"&gt; &lt;/span&gt;tagged&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastogreet:hummingbird
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The build is uneventful. The interesting bit is what comes out the other&amp;nbsp;end:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;images&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;mastogreet
git.linuxserver.pro/chris/mastogreet&lt;span class="w"&gt;  &lt;/span&gt;hummingbird&lt;span class="w"&gt;  &lt;/span&gt;e97c9dafdcc9&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;21&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;seconds&lt;span class="w"&gt; &lt;/span&gt;ago&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;126&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;MB
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A head-to-head against the same bot built&amp;nbsp;on &lt;code&gt;python:3.12-slim&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;images&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;mastogreet
git.linuxserver.pro/chris/mastogreet&lt;span class="w"&gt;  &lt;/span&gt;latest&lt;span class="w"&gt;       &lt;/span&gt;e835eccb3d62&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;140&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;MB
git.linuxserver.pro/chris/mastogreet&lt;span class="w"&gt;  &lt;/span&gt;hummingbird&lt;span class="w"&gt;  &lt;/span&gt;e97c9dafdcc9&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;126&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;MB
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;126 &lt;span class="caps"&gt;MB&lt;/span&gt; versus 140 &lt;span class="caps"&gt;MB&lt;/span&gt;. That&amp;#8217;s a roughly 10% shrink, which is worth saying clearly: if you came here expecting Google-style 30 &lt;span class="caps"&gt;MB&lt;/span&gt; distroless images, you won&amp;#8217;t find&amp;nbsp;them. &lt;code&gt;hi/python&lt;/code&gt; is a full Python 3.13 runtime with the Fedora-derived support libraries it depends on, just without the shell, package manager, and incidental tooling. The size win is real but it isn&amp;#8217;t the headline, and anyone selling Hummingbird on file-size alone is selling it&amp;nbsp;wrong.&lt;/p&gt;
&lt;p&gt;The headline is everything&amp;nbsp;else:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The image has&amp;nbsp;no &lt;code&gt;bash&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;dash&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;sh&lt;/code&gt;. &lt;code&gt;podman exec -it mastogreet sh&lt;/code&gt; exits&amp;nbsp;with &lt;code&gt;executable file 'sh' not found in $PATH&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;It has&amp;nbsp;no &lt;code&gt;apt&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;dnf&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;rpm&lt;/code&gt; binary. The &lt;span class="caps"&gt;RPM&lt;/span&gt; database is preserved as data for scanners, but the executable that would manipulate it isn&amp;#8217;t&amp;nbsp;there.&lt;/li&gt;
&lt;li&gt;The chunkah re-layering is visible in the pull: 33 small layers rather than half a dozen fat ones. Repeated pulls of related images deduplicate heavily, and &lt;span class="caps"&gt;CVE&lt;/span&gt; refreshes only re-pull the chunks that&amp;nbsp;changed.&lt;/li&gt;
&lt;li&gt;The image is signed, and ships with an &lt;span class="caps"&gt;SBOM&lt;/span&gt; you can&amp;nbsp;pull:&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;code&gt;sh
  cosign download sbom --platform linux/amd64 \
    registry.access.redhat.com/hi/python:3.13 &amp;gt; python.spdx.json&lt;/code&gt;&lt;/p&gt;
&lt;h3 id="rootless-smoke-test-with-the-footgun"&gt;Rootless Smoke Test, with the&amp;nbsp;Footgun&lt;/h3&gt;
&lt;p&gt;Before deploying the image to a host, I always smoke-test the container locally. With Hummingbird this surfaces a &lt;span class="caps"&gt;UID&lt;/span&gt; gotcha worth covering, because it&amp;#8217;s the one I expect most readers will hit on their first&amp;nbsp;run.&lt;/p&gt;
&lt;p&gt;The naive command, copied over from the previous&amp;nbsp;bot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--userns&lt;span class="o"&gt;=&lt;/span&gt;keep-id:uid&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10001&lt;/span&gt;,gid&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10001&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_ACCESS_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;...&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_INSTANCE_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;https://burningboard.net&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PWD&lt;/span&gt;&lt;span class="s2"&gt;/smoke:/data:Z&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;git.linuxserver.pro/chris/mastogreet:hummingbird

Traceback&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;most&lt;span class="w"&gt; &lt;/span&gt;recent&lt;span class="w"&gt; &lt;/span&gt;call&lt;span class="w"&gt; &lt;/span&gt;last&lt;span class="o"&gt;)&lt;/span&gt;:
&lt;span class="w"&gt;  &lt;/span&gt;File&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/app/mastowelcome.py&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;line&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;28&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;init_db
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;conn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sqlite3.connect&lt;span class="o"&gt;(&lt;/span&gt;path&lt;span class="o"&gt;)&lt;/span&gt;
sqlite3.OperationalError:&lt;span class="w"&gt; &lt;/span&gt;unable&lt;span class="w"&gt; &lt;/span&gt;to&lt;span class="w"&gt; &lt;/span&gt;open&lt;span class="w"&gt; &lt;/span&gt;database&lt;span class="w"&gt; &lt;/span&gt;file
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;What happened: the&amp;nbsp;image&amp;#8217;s &lt;code&gt;USER 65532&lt;/code&gt; would normally run the process as &lt;span class="caps"&gt;UID&lt;/span&gt; 65532 inside the container,&amp;nbsp;and &lt;code&gt;/data&lt;/code&gt; inside the image is owned by 65532 from the builder skeleton copy.&amp;nbsp;But &lt;code&gt;--userns=keep-id:uid=10001,gid=10001&lt;/code&gt; overrode that, mapping the host user to &lt;span class="caps"&gt;UID&lt;/span&gt; 10001 inside instead. The bot process therefore runs as&amp;nbsp;10001, &lt;code&gt;/data&lt;/code&gt; is owned by 65532, and the open-for-write&amp;nbsp;on &lt;code&gt;state.db&lt;/code&gt; fails with &lt;span class="caps"&gt;EACCES&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The fix is to match the userns mapping to the image&amp;#8217;s actual &lt;span class="caps"&gt;UID&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--userns&lt;span class="o"&gt;=&lt;/span&gt;keep-id:uid&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;65532&lt;/span&gt;,gid&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;65532&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_ACCESS_TOKEN&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;...&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_INSTANCE_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;https://burningboard.net&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$PWD&lt;/span&gt;&lt;span class="s2"&gt;/smoke:/data:Z&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;git.linuxserver.pro/chris/mastogreet:hummingbird

Traceback&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;most&lt;span class="w"&gt; &lt;/span&gt;recent&lt;span class="w"&gt; &lt;/span&gt;call&lt;span class="w"&gt; &lt;/span&gt;last&lt;span class="o"&gt;)&lt;/span&gt;:
&lt;span class="w"&gt;  &lt;/span&gt;File&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/app/mastowelcome.py&amp;quot;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;line&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;257&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;main
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;me&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;mastodon.me&lt;span class="o"&gt;()&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;...
mastodon.errors.MastodonUnauthorizedError:&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Mastodon API returned error&amp;#39;&lt;/span&gt;,
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="m"&gt;401&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Unauthorized&amp;#39;&lt;/span&gt;,&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Der Zugriffstoken ist ungültig&amp;#39;&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That looks like a failure, but it&amp;#8217;s actually the smoke test passing: the bot started, sqlite&amp;nbsp;opened &lt;code&gt;/data/state.db&lt;/code&gt; cleanly, the Mastodon client built and made an authenticated &lt;span class="caps"&gt;HTTPS&lt;/span&gt; call&amp;nbsp;to &lt;code&gt;burningboard.net&lt;/code&gt;, and the server replied 401 because I supplied a placeholder token. Everything up to &amp;#8220;real &lt;span class="caps"&gt;API&lt;/span&gt; conversation&amp;#8221;&amp;nbsp;works.&lt;/p&gt;
&lt;p&gt;This wrinkle is specifically a rootless smoke-test thing. The production path through the Ansible role is unaffected: the&amp;nbsp;host-side &lt;code&gt;/opt/mastogreet/&lt;/code&gt; is chowned to 65532 as root by&amp;nbsp;the &lt;code&gt;file&lt;/code&gt; task, the Quadlet runs the container without rootless userns remapping, and the in-container &lt;span class="caps"&gt;UID&lt;/span&gt; matches the host directory ownership without&amp;nbsp;any &lt;code&gt;keep-id&lt;/code&gt; gymnastics. Worth being explicit about, because someone will copy the smoke-test snippet from the previous article verbatim, hit this error, and spend twenty minutes wondering whether Hummingbird broke their bot. It didn&amp;#8217;t; the &lt;span class="caps"&gt;UID&lt;/span&gt; just&amp;nbsp;moved.&lt;/p&gt;
&lt;h3 id="day-to-day-differences"&gt;Day-to-Day&amp;nbsp;Differences&lt;/h3&gt;
&lt;p&gt;The bot behaves identically once&amp;nbsp;running. &lt;code&gt;podman logs mastogreet&lt;/code&gt; shows the same authentication line, the same backfill cutoff, the same poll loop. The only operational difference is&amp;nbsp;that &lt;code&gt;podman exec -it mastogreet sh&lt;/code&gt; no longer drops me into a shell, which is fine because I don&amp;#8217;t actually need to do that. When I do want to inspect the running container, I either&amp;nbsp;use &lt;code&gt;podman exec mastogreet python3 -c '...'&lt;/code&gt; or run&amp;nbsp;the &lt;code&gt;:3.13-builder&lt;/code&gt; image side-by-side for poking&amp;nbsp;purposes.&lt;/p&gt;
&lt;h2 id="updating-the-ansible-role"&gt;Updating the Ansible&amp;nbsp;Role&lt;/h2&gt;
&lt;p&gt;The role from the &lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/#the-ansible-layout"&gt;previous article&lt;/a&gt; needs two trivial&amp;nbsp;changes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# group_vars/all.yml&lt;/span&gt;
&lt;span class="nt"&gt;mastogreet_image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;git.linuxserver.pro/chris/mastogreet:hi-20260516&lt;/span&gt;
&lt;span class="nt"&gt;mastogreet_container_uid&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;65532&lt;/span&gt;
&lt;span class="nt"&gt;mastogreet_container_gid&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;65532&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;#8217;s it.&amp;nbsp;The &lt;code&gt;Ensure state directory exists with container UID ownership&lt;/code&gt; task picks up the new &lt;span class="caps"&gt;UID&lt;/span&gt;, the&amp;nbsp;templated &lt;code&gt;config.yaml&lt;/code&gt; lands with the right ownership, and the Quadlet unit gets re-rendered&amp;nbsp;because &lt;code&gt;mastogreet_image&lt;/code&gt; changed. Handler chain reloads systemd, restarts the service, and the next container creation pulls the Hummingbird-based image. Zero changes to the role&amp;#8217;s tasks or&amp;nbsp;handlers.&lt;/p&gt;
&lt;p&gt;This is also a decent illustration of why the role-plus-variables approach pays off: the entire base-image migration is a two-line variable diff&amp;nbsp;in &lt;code&gt;group_vars/all.yml&lt;/code&gt;. No host-side commands, no&amp;nbsp;manual &lt;code&gt;daemon-reload&lt;/code&gt;, no remembered&amp;nbsp;steps.&lt;/p&gt;
&lt;h2 id="red-hat-hardened-images-the-commercial-downstream"&gt;Red Hat Hardened Images: The Commercial&amp;nbsp;Downstream&lt;/h2&gt;
&lt;p&gt;The article up to this point has been about the Fedora-community side: free, vendor-neutral, available to anyone who&amp;nbsp;can &lt;code&gt;podman pull&lt;/code&gt;. The images&amp;nbsp;at &lt;code&gt;registry.access.redhat.com/hi/&lt;/code&gt; are themselves free to consume. The commercial wrapper around the same artefacts is &lt;a href="https://www.redhat.com/en/products/hardened-images"&gt;Red Hat Hardened Images&lt;/a&gt;, and the separation is fairly&amp;nbsp;clean.&lt;/p&gt;
&lt;p&gt;What you get when you buy&amp;nbsp;in:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Support under your existing &lt;span class="caps"&gt;RHEL&lt;/span&gt; or OpenShift subscription.&lt;/strong&gt; Hardened-image consumption is covered; you can open a case if something breaks. The community images you pull anonymously are&amp;nbsp;best-effort.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The Calunga trusted-libraries index.&lt;/strong&gt; A vetted Python package index&amp;nbsp;at &lt;code&gt;packages.redhat.com/trusted-libraries/python/&lt;/code&gt;, accessed via a&amp;nbsp;mounted &lt;code&gt;pip.conf&lt;/code&gt; secret in the builder stage. The point isn&amp;#8217;t faster downloads; the point is that somebody has signed and stands behind the packages. For regulated environments this matters more than for a Mastodon&amp;nbsp;bot.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;FIPS&lt;/span&gt; variants and validated cryptography.&lt;/strong&gt;&amp;nbsp;The &lt;code&gt;:3.13-fips&lt;/code&gt; tag points at images linked against &lt;span class="caps"&gt;FIPS&lt;/span&gt;-validated OpenSSL, with the validation paperwork&amp;nbsp;attached.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Contractual &lt;span class="caps"&gt;CVE&lt;/span&gt; response times.&lt;/strong&gt; Same near-zero &lt;span class="caps"&gt;CVE&lt;/span&gt; design goal as the community catalog, with SLAs around&amp;nbsp;it.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The underlying images are built by the same Konflux pipeline. The commercial offer adds support and contractual guarantees; it does not gatekeep the artefacts themselves. If you want to evaluate and contribute, you do that entirely from the community side. If you need someone to call at 2am, you&amp;nbsp;upgrade.&lt;/p&gt;
&lt;h2 id="a-brief-note-on-fedora-hummingbird-the-os"&gt;A Brief Note on Fedora Hummingbird the &lt;span class="caps"&gt;OS&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;I&amp;#8217;ve focused on the image catalog because that&amp;#8217;s the part I&amp;#8217;m using in anger today. The &lt;span class="caps"&gt;OS&lt;/span&gt; side is interesting independently. A bootable container that becomes a full Linux host, with atomic updates, rollback, and a read-only root, is a meaningful shift for the kind of small-fleet operator this blog gets read by. I&amp;#8217;m running it in a &lt;span class="caps"&gt;VM&lt;/span&gt; on my workstation to see how it behaves; the experience so far is &amp;#8220;Fedora, but the upgrade story&amp;nbsp;is &lt;code&gt;bootc upgrade &amp;amp;&amp;amp; systemctl reboot&lt;/code&gt;, and there&amp;#8217;s&amp;nbsp;no &lt;code&gt;dnf upgrade&lt;/code&gt; to forget about.&amp;#8221; I&amp;#8217;ll write that up properly once I have more than a few weeks&amp;nbsp;of &lt;code&gt;bootc&lt;/code&gt; time on&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;The relevant point for this article is that the &lt;span class="caps"&gt;OS&lt;/span&gt; and the image catalog share machinery: the same Konflux pipeline produces both, the same hermetic-build guarantees apply, and the same SBOMs ship for both. If you&amp;#8217;re already running Hummingbird containers, the &lt;span class="caps"&gt;OS&lt;/span&gt; is a natural extension. If you&amp;#8217;re not, the &lt;span class="caps"&gt;OS&lt;/span&gt; is also a perfectly good place to&amp;nbsp;start.&lt;/p&gt;
&lt;h2 id="when-this-pays-off-and-when-it-doesnt"&gt;When This Pays Off and When It&amp;nbsp;Doesn&amp;#8217;t&lt;/h2&gt;
&lt;p&gt;Hummingbird images are worth reaching for&amp;nbsp;when:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The workload runs in production and survives reinstalls. The cost of a multi-stage Containerfile is paid&amp;nbsp;once.&lt;/li&gt;
&lt;li&gt;Security review will ask for an &lt;span class="caps"&gt;SBOM&lt;/span&gt;, signatures, or a &lt;span class="caps"&gt;CVE&lt;/span&gt;-response timeline. Showing up&amp;nbsp;with &lt;code&gt;cosign verify&lt;/code&gt; and a Konflux attestation is a much shorter conversation than promising to &amp;#8220;look into&amp;nbsp;it.&amp;#8221;&lt;/li&gt;
&lt;li&gt;The container is part of a fleet, not a one-off. The compounding benefit of small, fast-updating images is real once you have more than a handful of&amp;nbsp;hosts.&lt;/li&gt;
&lt;li&gt;You already accept the trade-offs of distroless (no shell access, multi-stage build mandatory, debugging shifts to logs and structured&amp;nbsp;tooling).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I&amp;#8217;d skip them&amp;nbsp;for:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Throw-away development containers where I want&amp;nbsp;to &lt;code&gt;podman exec sh&lt;/code&gt; and iterate. The builder variant is closer to that experience than the runtime one,&amp;nbsp;but &lt;code&gt;python:3.12-slim&lt;/code&gt; is still faster to spin&amp;nbsp;up.&lt;/li&gt;
&lt;li&gt;Workloads that genuinely depend on tools the distroless image doesn&amp;#8217;t ship (custom shell-script&amp;nbsp;entrypoints, &lt;code&gt;tini&lt;/code&gt;-style &lt;span class="caps"&gt;PID&lt;/span&gt;-1 wrappers that aren&amp;#8217;t already in the base, things that&amp;nbsp;expect &lt;code&gt;getent&lt;/code&gt; at runtime). You can usually work around this, but if the workaround dominates the Containerfile, the gain is&amp;nbsp;gone.&lt;/li&gt;
&lt;li&gt;Anything where the operator workflow assumes a shell is&amp;nbsp;one &lt;code&gt;podman exec&lt;/code&gt; away. Worth adopting eventually, but not as a same-day&amp;nbsp;forklift.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For&amp;nbsp;the &lt;code&gt;mastogreet&lt;/code&gt; case, the rebuild took about half an hour: drafting the multi-stage Containerfile, fixing&amp;nbsp;one &lt;code&gt;RUN&lt;/code&gt; line that I left in shell form, bumping the &lt;span class="caps"&gt;UID&lt;/span&gt;&amp;nbsp;in &lt;code&gt;group_vars/all.yml&lt;/code&gt;, and untangling the rootless smoke-test footgun once. Since cutting over to the Hummingbird-based image the journal output has been indistinguishable from the previous deployment. The next rebuild will move&amp;nbsp;from &lt;code&gt;:3.13&lt;/code&gt; to &lt;code&gt;:3.14&lt;/code&gt; so I stay on a refreshed &lt;span class="caps"&gt;CVE&lt;/span&gt; baseline rather than pin too&amp;nbsp;tightly.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://fedoramagazine.org/fedora-hummingbird-linux-taking-the-hummingbird-model-to-the-full-os/"&gt;Fedora Magazine: Fedora Hummingbird Linux, Taking the Hummingbird model to the full &lt;span class="caps"&gt;OS&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.redhat.com/en/about/press-releases/fedora-hummingbird-linux-brings-agentic-linux-builders"&gt;Red Hat press release: Fedora Hummingbird Linux brings agentic Linux to&amp;nbsp;builders&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.redhat.com/en/products/hardened-images"&gt;Red Hat Hardened Images product&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hummingbird-project.io/docs/using/overview/"&gt;Project Hummingbird: Using a container&amp;nbsp;image&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developers.redhat.com/articles/2026/04/28/exploring-distroless-containers-project-hummingbird"&gt;Red Hat Developers: Exploring distroless containers with Project&amp;nbsp;Hummingbird&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://developers.redhat.com/articles/2026/05/05/build-trusted-python-containers-project-hummingbird-and-calunga"&gt;Red Hat Developers: Build trusted Python containers with Project Hummingbird and&amp;nbsp;Calunga&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/"&gt;Ansible-Native Quadlets: Deploying a Mastodon Greeter Bot with containers.podman&lt;/a&gt; - the original mastogreet writeup this article&amp;nbsp;extends&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Production-Grade Container Deployment with Podman Quadlets&lt;/a&gt; - the Quadlet&amp;nbsp;primer&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman in Production: Quadlets, Secrets, Auto-Updates, and Docker Compatibility&lt;/a&gt; - broader Podman&amp;nbsp;context&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="fedora"/><category term="hummingbird"/><category term="podman"/><category term="containers"/><category term="distroless"/><category term="security"/><category term="sbom"/><category term="redhat"/><category term="python"/></entry><entry><title>Red Hat Offline Knowledge Portal: All the Docs, Air-Gapped, On Your Laptop</title><link href="https://hofstede.it/red-hat-offline-knowledge-portal-all-the-docs-air-gapped-on-your-laptop/" rel="alternate"/><published>2026-05-14T00:00:00+02:00</published><updated>2026-05-14T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-05-14:/red-hat-offline-knowledge-portal-all-the-docs-air-gapped-on-your-laptop/</id><summary type="html">&lt;p&gt;The entire Red Hat documentation site and the full Knowledgebase fit into a single &lt;span class="caps"&gt;OCI&lt;/span&gt; container that updates weekly, runs locally with a web &lt;span class="caps"&gt;UI&lt;/span&gt; and Solr search, and is included in every &lt;span class="caps"&gt;RHEL&lt;/span&gt; subscription that bundles Satellite. I have it on my laptop. I use it daily. Almost nobody I talk to knows it exists. This post is my small contribution to fixing&amp;nbsp;that.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-it-actually-is"&gt;What it actually&amp;nbsp;is&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-it-matters"&gt;Why it&amp;nbsp;matters&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-word-on-the-marketing-gap"&gt;A word on the marketing&amp;nbsp;gap&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#practical-onboarding"&gt;Practical&amp;nbsp;onboarding&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#closing-thoughts"&gt;Closing&amp;nbsp;thoughts&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#further-reading"&gt;Further&amp;nbsp;reading&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;The entire Red Hat documentation site and the full Knowledgebase fit into a single &lt;span class="caps"&gt;OCI&lt;/span&gt; container that updates weekly, runs locally with a web &lt;span class="caps"&gt;UI&lt;/span&gt; and Solr search, and is included in every &lt;span class="caps"&gt;RHEL&lt;/span&gt; subscription that bundles Red Hat Satellite. I have it on my laptop. I use it daily. Almost nobody I talk to knows it&amp;nbsp;exists.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Red Hat Offline Knowledge Portal" src="https://hofstede.it/images/2026-05-14-red-hat-offline-knowledge-portal.png" title="The RHOKP landing page running locally on 127.0.0.1:8080"&gt;&lt;/p&gt;
&lt;p&gt;That paragraph is the mission statement of this post. The product is called the Red Hat Offline Knowledge Portal, &lt;span class="caps"&gt;RHOKP&lt;/span&gt; for short, and it is one of those quiet, slightly-too-modest pieces of software that a customer-facing engineer like me runs into, falls in love with, and then realises that most of the people who would benefit from it have never been told it is there. This post is my attempt to help with&amp;nbsp;that.&lt;/p&gt;
&lt;h2 id="what-it-actually-is"&gt;What it actually&amp;nbsp;is&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;RHOKP&lt;/span&gt; is a single container image, published&amp;nbsp;on &lt;code&gt;registry.redhat.io&lt;/code&gt;, that contains a self-hosted, locally-searchable copy of essentially the entire Red Hat customer-facing knowledge corpus. Concretely, what is inside the&amp;nbsp;box:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;All product documentation.&lt;/strong&gt; Every product, every supported version, the full &lt;span class="caps"&gt;HTML&lt;/span&gt; rendering you would otherwise read&amp;nbsp;on &lt;code&gt;docs.redhat.com&lt;/code&gt;. Documentation is included for all versions of each&amp;nbsp;product.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The full Knowledgebase.&lt;/strong&gt; Every solution article and every troubleshooting writeup that lives&amp;nbsp;behind &lt;code&gt;access.redhat.com/solutions&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Errata and the &lt;span class="caps"&gt;CVE&lt;/span&gt; database.&lt;/strong&gt; The Red Hat Product Errata catalogue and the &lt;span class="caps"&gt;CVE&lt;/span&gt; database, locally searchable. Comprehensive information for the CVEs that affect Red Hat products, basic information for the&amp;nbsp;rest.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The Security Data &lt;span class="caps"&gt;API&lt;/span&gt;.&lt;/strong&gt; A machine-consumable &lt;span class="caps"&gt;JSON&lt;/span&gt; feed of &lt;span class="caps"&gt;CVE&lt;/span&gt; and errata data, available locally, so security automation that normally&amp;nbsp;hits &lt;code&gt;access.redhat.com/security/data/&lt;/code&gt; keeps working in an&amp;nbsp;air-gap.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Product lifecycle pages and selected Customer Portal pages&lt;/strong&gt; for the support resources that matter when you are&amp;nbsp;offline.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;An Apache Solr 9.8 index&lt;/strong&gt; in front of all of it, exposed through a clean web &lt;span class="caps"&gt;UI&lt;/span&gt; that looks and feels like a stripped-down,&amp;nbsp;faster &lt;code&gt;access.redhat.com&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The whole thing is a single appliance image built&amp;nbsp;on &lt;code&gt;ubi9/httpd-24&lt;/code&gt;. As of today, the size on my&amp;nbsp;laptop:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;images&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;rhokp
registry.redhat.io/offline-knowledge-portal/rhokp-rhel9&lt;span class="w"&gt;  &lt;/span&gt;latest&lt;span class="w"&gt;  &lt;/span&gt;a7c87cff0ff7&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;40&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;hours&lt;span class="w"&gt; &lt;/span&gt;ago&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="m"&gt;9&lt;/span&gt;.68&lt;span class="w"&gt; &lt;/span&gt;GB
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;9.68 &lt;span class="caps"&gt;GB&lt;/span&gt; for the sum of Red Hat&amp;#8217;s public technical knowledge is, when you stop and think about it, an absurdly good deal. Red Hat publishes a new snapshot roughly weekly (the &amp;#8220;40 hours ago&amp;#8221; line above is typical), so once you have it pulled,&amp;nbsp;a &lt;code&gt;podman pull&lt;/code&gt; once a week keeps you within days of the live&amp;nbsp;site.&lt;/p&gt;
&lt;p&gt;A quick clarification on entitlement, because this catches almost everyone. Red Hat Satellite is not sold as a stand-alone subscription. It comes bundled into certain &lt;span class="caps"&gt;RHEL&lt;/span&gt; subscriptions (the Smart Management and equivalent add-ons), and it is that bundled entitlement that unlocks &lt;span class="caps"&gt;RHOKP&lt;/span&gt;. If your &lt;span class="caps"&gt;RHEL&lt;/span&gt; subscription includes Satellite, you have access to &lt;span class="caps"&gt;RHOKP&lt;/span&gt; at no additional cost, whether or not you have actually deployed a Satellite&amp;nbsp;server.&lt;/p&gt;
&lt;p&gt;Two things then gate access. The first is the image itself: you need&amp;nbsp;to &lt;code&gt;podman login registry.redhat.io&lt;/code&gt; against an account whose subscription includes the Satellite entitlement. The second is an &lt;strong&gt;access key&lt;/strong&gt;, which is a small personal token you generate against your Red Hat account and pass into the container at launch as&amp;nbsp;the &lt;code&gt;ACCESS_KEY&lt;/code&gt; environment variable. Without the access key, the portal launches in a degraded mode: you get the Documentation tile and can browse the docs, but the search bar, the Solutions, and the Articles are unavailable. With the access key passed in, the full portal is unlocked. The key never leaves the host you generated it on, and it is what tells the container that the running instance is bound to an entitled&amp;nbsp;human.&lt;/p&gt;
&lt;p&gt;One important boundary up front: the &lt;span class="caps"&gt;RHOKP&lt;/span&gt; &lt;span class="caps"&gt;EULA&lt;/span&gt; explicitly forbids publishing or sharing the image, its content, or your access key. The product is licensed for your own internal use. Treat the image like any other entitled artefact, treat the key like a password, and keep both off the public internet. None of the screenshots or commands in this post leak&amp;nbsp;either.&lt;/p&gt;
&lt;h2 id="why-it-matters"&gt;Why it&amp;nbsp;matters&lt;/h2&gt;
&lt;p&gt;The obvious use case is air-gapped environments, and that is genuinely the headline feature: financial-services back offices, regulated industrial networks, defence, classified work, anywhere connecting a host to the public internet is a serious decision. For those operators, &lt;span class="caps"&gt;RHOKP&lt;/span&gt; is the difference between &amp;#8220;we have the documentation&amp;#8221; and &amp;#8220;we have a stale &lt;span class="caps"&gt;PDF&lt;/span&gt; someone downloaded in 2023&amp;#8221;. That alone justifies the&amp;nbsp;product.&lt;/p&gt;
&lt;p&gt;But the air-gap framing undersells it, because the same properties that make &lt;span class="caps"&gt;RHOKP&lt;/span&gt; useful behind a Faraday cage also make it useful in a long list of much more mundane&amp;nbsp;scenarios.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Customer-site engagements.&lt;/strong&gt; As a consultant I spend a non-trivial amount of my working life inside customer networks where outbound traffic is often restricted, sometimes via a corporate proxy that requires Kerberos against an &lt;span class="caps"&gt;AD&lt;/span&gt; I am not a member of, sometimes via a &amp;#8220;we will whitelist that domain by Wednesday&amp;#8221; workflow that takes until the following Friday. In those situations, the difference between a productive day and a frustrating one is whether the docs I need to consult are reachable from the bastion I have been given. With &lt;span class="caps"&gt;RHOKP&lt;/span&gt; running on my laptop and the laptop bridged into the customer network on whatever terms they allow, the docs come with&amp;nbsp;me.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Trains and planes.&lt;/strong&gt; This is half the reason I keep a copy on my work laptop. The &lt;span class="caps"&gt;ICE&lt;/span&gt; between Frankfurt and Hamburg has WiFi that is best described as aspirational, and I have spent more billable hours than I would like to admit reading product documentation between Mannheim and Köln. Local docs survive tunnels, dead spots, and the airline WiFi tier that lets you message but not stream. The docs are always there because they live two centimetres below my&amp;nbsp;keyboard.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Bandwidth-constrained sites.&lt;/strong&gt; Branch offices on satellite uplinks. Edge deployments. Anywhere the round trip&amp;nbsp;to &lt;code&gt;docs.redhat.com&lt;/code&gt; is fast enough to be usable but not fast enough to be pleasant, a local mirror with a real search index is qualitatively better than the live&amp;nbsp;site.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Privacy.&lt;/strong&gt; Every search you run&amp;nbsp;on &lt;code&gt;access.redhat.com&lt;/code&gt; is, reasonably, logged. That is how a vendor learns what their customers are confused about, and the data is used to improve products and prioritise documentation. It is also, depending on your operational context, information you may not want a third party to have. Running &lt;span class="caps"&gt;RHOKP&lt;/span&gt; means your search history for &amp;#8220;how do I rotate the IdM &lt;span class="caps"&gt;CA&lt;/span&gt; in a disaster-recovery scenario&amp;#8221; stays on your laptop. For some people that matters a&amp;nbsp;lot.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Speed.&lt;/strong&gt; Less philosophical, but worth saying out loud: local Solr against a 10 &lt;span class="caps"&gt;GB&lt;/span&gt; corpus on an NVMe drive is faster&amp;nbsp;than &lt;code&gt;access.redhat.com&lt;/code&gt;. Not faster in the sense of &amp;#8220;the network is the bottleneck&amp;#8221;, faster in the sense of &amp;#8220;you stop noticing the search round trip exists&amp;#8221;. After a few weeks of using it, going back to the live site feels slow in a way you cannot quite articulate until you measure&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="a-word-on-the-marketing-gap"&gt;A word on the marketing&amp;nbsp;gap&lt;/h2&gt;
&lt;p&gt;Before we get to how to run it, a brief observation about&amp;nbsp;visibility.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;RHOKP&lt;/span&gt; is, in my honest opinion, one of the best products Red Hat ships that almost nobody talks about. It made a short appearance at Red Hat Summit and surfaces occasionally in disconnected-environments documentation, but it does not show up in the places customers usually look. Most of the people I have shown it to had no idea it existed, and a non-trivial number of Red Hatters I have shown it to had not heard of it either, which surprised me. That is a shame, because the people who would benefit from this product are not a niche: they are anyone whose &lt;span class="caps"&gt;RHEL&lt;/span&gt; subscription includes Satellite, which is a large fraction of the enterprise customer&amp;nbsp;base.&lt;/p&gt;
&lt;p&gt;I want to be precise about my framing here. This post is not a critique of anyone&amp;#8217;s work and not an argument that anyone has dropped the ball. I think this product deserves more attention than it currently receives, and I am going to spend some of my own writing&amp;#8217;s reach on giving it that attention. There is a long and honourable tradition at Red Hat of employees publicly enthusing about parts of the portfolio that they personally find excellent, and this post is meant to sit in that tradition. If even a handful of customers end up running &lt;span class="caps"&gt;RHOKP&lt;/span&gt; because of it, the post will have done its small&amp;nbsp;job.&lt;/p&gt;
&lt;h2 id="practical-onboarding"&gt;Practical&amp;nbsp;onboarding&lt;/h2&gt;
&lt;p&gt;The fastest way to be productive with &lt;span class="caps"&gt;RHOKP&lt;/span&gt;, on&amp;nbsp;an &lt;code&gt;x86_64&lt;/code&gt; or &lt;code&gt;aarch64&lt;/code&gt; host with Podman installed, a Red Hat account on a &lt;span class="caps"&gt;RHEL&lt;/span&gt; subscription that includes Satellite, and a few gigabytes of free disk space (the docs ask for 50 &lt;span class="caps"&gt;GB&lt;/span&gt; minimum, 75 &lt;span class="caps"&gt;GB&lt;/span&gt; recommended, to leave headroom for the image plus the running container&amp;#8217;s working&amp;nbsp;state):&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;1. Log into the Red Hat&amp;nbsp;registry.&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;login&lt;span class="w"&gt; &lt;/span&gt;registry.redhat.io
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Use your Customer Portal, Red Hat Developer, or registry service account credentials. On a Fedora workstation, the entitlement certificates Podman needs are auto-mounted&amp;nbsp;via &lt;code&gt;/usr/share/containers/mounts.conf&lt;/code&gt;, so the registry login is the only thing you have to do by&amp;nbsp;hand.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;2. Pull the image.&lt;/strong&gt; Roughly 10 &lt;span class="caps"&gt;GB&lt;/span&gt; on the wire, so set this off and go make&amp;nbsp;coffee.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;3. Generate your access key.&lt;/strong&gt; This is the step that surprised me the first time I deployed &lt;span class="caps"&gt;RHOKP&lt;/span&gt;. Open the &lt;em&gt;Red Hat Offline Knowledge Portal Access Key Generator&lt;/em&gt; page on the Customer Portal (the installation guide linked at the bottom of this post has the direct link), sign in, and click &lt;em&gt;Generate key&lt;/em&gt;. Copy the resulting string somewhere safe. The key is stored against your Red Hat account, and the same page will display it again if you need it later. If it leaks, file a bug through the same page to unbind it and generate a new&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;4. Launch the container with the key.&lt;/strong&gt; This is the canonical run command from the documentation, almost&amp;nbsp;verbatim:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8080&lt;/span&gt;:8080&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8443&lt;/span&gt;:8443&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--env&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ACCESS_KEY=&amp;lt;your_personal_access_key&amp;gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Give the container about thirty seconds to come up. Then point a browser&amp;nbsp;at &lt;code&gt;http://127.0.0.1:8080&lt;/code&gt; or &lt;code&gt;https://127.0.0.1:8443&lt;/code&gt; (the &lt;span class="caps"&gt;HTTPS&lt;/span&gt; endpoint uses a self-signed certificate, so expect the usual browser warning the first time). You should land on the portal, with the date of the content extract shown in the upper-right corner of the&amp;nbsp;page.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;5. Optional, but worth knowing about: tune it with environment variables.&lt;/strong&gt; The container reads a handful of additional env&amp;nbsp;vars:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;ONLINE_VIEW=true&lt;/code&gt; adds an &lt;em&gt;Online View&lt;/em&gt; button on every page that links to the&amp;nbsp;live &lt;code&gt;access.redhat.com&lt;/code&gt; or &lt;code&gt;docs.redhat.com&lt;/code&gt; equivalent. Useful on a laptop that is sometimes&amp;nbsp;connected.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;SOLR_MEM=4g&lt;/code&gt; raises the Solr &lt;span class="caps"&gt;JVM&lt;/span&gt; heap from the&amp;nbsp;default &lt;code&gt;1g&lt;/code&gt;. On a workstation with spare memory, this makes the search noticeably&amp;nbsp;snappier.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;UNCLASSIFIED_BANNER=true&lt;/code&gt; paints a&amp;nbsp;green &lt;code&gt;UNCLASSIFIED&lt;/code&gt; bar across the top of every page. This one is aimed at government and defence users who run &lt;span class="caps"&gt;RHOKP&lt;/span&gt; alongside classified material and want a constant visual reminder that this particular tab is the unclassified one. I have never personally needed it, but the fact that it ships in the product tells you something about the audience Red Hat is serving&amp;nbsp;here.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;You can also bring your own &lt;span class="caps"&gt;TLS&lt;/span&gt; certificate by mounting a directory with the expected layout&amp;nbsp;at &lt;code&gt;/opt/app-root/httpd-ssl/&lt;/code&gt;. The official guide has the exact directory structure; for a laptop deployment, the self-signed default is&amp;nbsp;fine.&lt;/p&gt;
&lt;p&gt;A few practical notes that the docs do not spell&amp;nbsp;out:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Run it as a Quadlet if you want it always-on.&lt;/strong&gt; The docs&amp;nbsp;show &lt;code&gt;--rm&lt;/code&gt; because they assume an ad-hoc launch, but for a workstation it makes more sense to convert that into&amp;nbsp;a &lt;code&gt;systemd --user&lt;/code&gt; Quadlet so the portal survives reboots and is there before you are. The unit is unremarkable:&amp;nbsp;one &lt;code&gt;Image=&lt;/code&gt;, the&amp;nbsp;two &lt;code&gt;PublishPort=&lt;/code&gt; lines,&amp;nbsp;the &lt;code&gt;Environment=ACCESS_KEY=...&lt;/code&gt; for your key,&amp;nbsp;and &lt;code&gt;Restart=always&lt;/code&gt;. Same shape as any other long-running container, with the access key kept in&amp;nbsp;a &lt;code&gt;systemd-creds&lt;/code&gt; blob or a&amp;nbsp;mode-&lt;code&gt;600&lt;/code&gt; file rather than in shell&amp;nbsp;history.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Schedule a weekly refresh.&lt;/strong&gt; I have a tiny systemd timer that&amp;nbsp;runs &lt;code&gt;podman pull&lt;/code&gt; against the image once a week and restarts the unit if a new digest arrived. The image tag&amp;nbsp;is &lt;code&gt;latest&lt;/code&gt;, so the upstream weekly rebuild is what drives freshness. If you forget to refresh for a month, the world does not end, but you will start to notice gaps as newer errata land in the live site without yet being in your local&amp;nbsp;copy.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Try the search first.&lt;/strong&gt; The Solr front-end is the part that surprises people. It is fast, it understands Red Hat&amp;#8217;s domain vocabulary&amp;nbsp;(&lt;code&gt;pacemaker&lt;/code&gt;, &lt;code&gt;gpfs&lt;/code&gt;, &lt;code&gt;osp&lt;/code&gt;, &lt;code&gt;ipa&lt;/code&gt;), and the results are ranked sensibly. The first thing I tell people to do after they get the container running is to search for something they searched for&amp;nbsp;on &lt;code&gt;access.redhat.com&lt;/code&gt; last week and compare. The local result is usually as good and arrives&amp;nbsp;faster.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;For genuinely disconnected environments, save and ship the image.&lt;/strong&gt; The documented air-gap flow is straightforward: on a connected host, after you have pulled the image,&amp;nbsp;run&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;save&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;oci-archive&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;rhokp.tar&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;registry.redhat.io/offline-knowledge-portal/rhokp-rhel9:latest
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;transfer the&amp;nbsp;resulting &lt;code&gt;rhokp.tar&lt;/code&gt; to the disconnected side using whatever your organisation&amp;#8217;s approved method is (sneakernet, data diode, one-way file mover), and on the offline host&amp;nbsp;run &lt;code&gt;podman load -i rhokp.tar&lt;/code&gt; followed by the&amp;nbsp;same &lt;code&gt;podman run&lt;/code&gt; command shown above. The access key still has to be generated against a Red Hat account from a connected host first, then copied across with the&amp;nbsp;image.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Combine it with your local Insights&amp;nbsp;or &lt;code&gt;rhc&lt;/code&gt; setup if you are fully disconnected.&lt;/strong&gt; &lt;span class="caps"&gt;RHOKP&lt;/span&gt; gives you the knowledge corpus locally; the host-side automation tooling on Satellite gives you the operations that consume it. In a properly built air-gap the two form a closed loop: hosts report locally, knowledge is searched locally, recommendations apply locally, and the only thing that crosses the boundary is a periodic image&amp;nbsp;refresh.&lt;/p&gt;
&lt;h2 id="closing-thoughts"&gt;Closing&amp;nbsp;thoughts&lt;/h2&gt;
&lt;p&gt;There is a quiet principle in this product that I think is worth&amp;nbsp;preserving.&lt;/p&gt;
&lt;p&gt;Documentation and operational knowledge are most useful when they do not require connectivity to be reached. The most important reference materials in any operational toolkit are the ones you can still get to when nothing else works: when the corporate &lt;span class="caps"&gt;VPN&lt;/span&gt; is down, when the customer&amp;#8217;s egress firewall has a typo, when the train is in a tunnel, when the building you are sitting in does not allow outbound traffic on principle. A container that holds everything you might need to know about the platform you are operating, that you can pull once and reach forever, is a small but real act of operational&amp;nbsp;resilience.&lt;/p&gt;
&lt;p&gt;It is also, quietly, an act of trust in the customer. The implicit message of a product like &lt;span class="caps"&gt;RHOKP&lt;/span&gt; is &amp;#8220;here is the knowledge, take it with you, you do not need to come back to us to retrieve it.&amp;#8221; That is a posture I wish more vendors took. The default in this industry has drifted, slowly, towards services and portals and consoles and search bars that live on someone else&amp;#8217;s infrastructure, and the cumulative effect on operational autonomy is greater than any single product makes it look. A 10 &lt;span class="caps"&gt;GB&lt;/span&gt; tarball you can carry around in your pocket is the opposite of that drift. It is not a revolutionary product. It is a small, deliberate vote in favour of the customer&amp;#8217;s ability to function on their&amp;nbsp;own.&lt;/p&gt;
&lt;p&gt;That is the part I will keep paying for, even if every other vendor in my stack eventually decides documentation belongs in a chatbot behind a login wall. The &lt;span class="caps"&gt;RHOKP&lt;/span&gt; container&amp;nbsp;in &lt;code&gt;~/.local/share/containers/&lt;/code&gt; is, in its small way, infrastructure I&amp;nbsp;own.&lt;/p&gt;
&lt;p&gt;If your &lt;span class="caps"&gt;RHEL&lt;/span&gt; subscription includes Satellite, you have this already. Pull it, run it, and the next time you are on a train without WiFi and a customer pings you about a sosreport, you will be glad you&amp;nbsp;did.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_offline_knowledge_portal/1/html-single/what_is_the_red_hat_offline_knowledge_portal/index"&gt;What is the Red Hat Offline Knowledge Portal&lt;/a&gt; - the official&amp;nbsp;one-pager.&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.redhat.com/en/documentation/red_hat_offline_knowledge_portal/1/html-single/making_red_hat_expert_knowledge_available_in_offline_environments/index"&gt;Making Red Hat Expert Knowledge Available in Offline Environments&lt;/a&gt; - the full installation and air-gap deployment&amp;nbsp;guide.&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="red hat"/><category term="rhel"/><category term="satellite"/><category term="documentation"/><category term="offline"/><category term="air-gap"/><category term="podman"/><category term="containers"/><category term="knowledge-base"/></entry><entry><title>Distrobox: Different Distributions in a Box, Powered by Podman</title><link href="https://hofstede.it/distrobox-different-distributions-in-a-box-powered-by-podman/" rel="alternate"/><published>2026-05-10T00:00:00+02:00</published><updated>2026-05-10T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-05-10:/distrobox-different-distributions-in-a-box-powered-by-podman/</id><summary type="html">&lt;p&gt;A &amp;#8220;real&amp;#8221; &lt;span class="caps"&gt;RHEL&lt;/span&gt; container on a Fedora laptop, an Arch box for the &lt;span class="caps"&gt;AUR&lt;/span&gt;, an Ubuntu box for that one vendor tool that ships .deb files only, all of them with your home directory mounted, your shell history shared, and &lt;span class="caps"&gt;GUI&lt;/span&gt; apps that integrate into your desktop as if they were native. Distrobox uses Podman to make that boring instead of clever, and it pairs especially well with Atomic distributions where the host is meant to stay&amp;nbsp;immutable.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Distrobox" src="https://hofstede.it/images/2026-05-10-distrobox-podman-dev-environments.png" title="Distrobox: Different Distributions in a Box"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-distrobox-actually-does"&gt;What Distrobox Actually&amp;nbsp;Does&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-i-use-it"&gt;Why I Use It&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#multiple-distributions"&gt;Multiple&amp;nbsp;Distributions&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#per-project-toolchains"&gt;Per-Project&amp;nbsp;Toolchains&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#vscode-integration-is-excellent"&gt;VSCode Integration is&amp;nbsp;Excellent&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#atomic-desktops-mutability-where-you-need-it"&gt;Atomic Desktops: Mutability Where You Need&amp;nbsp;It&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-actually-stays-on-the-host"&gt;What Actually Stays on the&amp;nbsp;Host&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rhel-userspace-on-fedora"&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; Userspace on&amp;nbsp;Fedora&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-few-patterns-worth-knowing"&gt;A Few Patterns Worth Knowing&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#exporting-applications-and-binaries-to-the-host"&gt;Exporting Applications and Binaries to the&amp;nbsp;Host&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#init-files-and-reproducible-boxes"&gt;Init Files and Reproducible&amp;nbsp;Boxes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#btrfs-subvolumes-for-per-box-snapshots"&gt;Btrfs Subvolumes for Per-Box&amp;nbsp;Snapshots&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#nesting-and-rootful-boxes"&gt;Nesting and Rootful&amp;nbsp;Boxes&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#where-distrobox-stops-being-the-right-tool"&gt;Where Distrobox Stops Being the Right&amp;nbsp;Tool&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#further-reading"&gt;Further&amp;nbsp;Reading&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;Most developers already solve environment contamination manually: Python venvs, custom toolchain directories, shell glue, version managers, and pages of setup notes. &lt;a href="https://distrobox.it/"&gt;Distrobox&lt;/a&gt; replaces much of that with a simpler idea: run the toolchain in another Linux distribution container that behaves like a normal shell&amp;nbsp;session.&lt;/p&gt;
&lt;p&gt;It is a thin wrapper around &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman&lt;/a&gt; (it can also drive Docker, but we don&amp;#8217;t use the Docker word here) that creates long-lived, tightly integrated containers, with your home directory mounted in, your user mapped through, and the desktop bus wired up so that &lt;span class="caps"&gt;GUI&lt;/span&gt; applications and editors &lt;em&gt;just work&lt;/em&gt;. A Fedora laptop with an Arch development environment for one project, a &lt;span class="caps"&gt;RHEL&lt;/span&gt; 9 environment for another, and an Ubuntu environment for the binary the vendor only ships as&amp;nbsp;a &lt;code&gt;.deb&lt;/code&gt; becomes a fairly normal setup rather than a heroic&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;Unlike a &lt;span class="caps"&gt;VM&lt;/span&gt;, a Distrobox container shares the host kernel, so startup is effectively instant and memory overhead is small. You get userspace separation, not hardware virtualization. Conceptually it sits between&amp;nbsp;a &lt;code&gt;chroot&lt;/code&gt; (same kernel, almost no isolation, painful to set up) and a full &lt;span class="caps"&gt;VM&lt;/span&gt; (separate kernel, real isolation, real overhead): same kernel, useful userspace separation, and a shell session that feels&amp;nbsp;native.&lt;/p&gt;
&lt;p&gt;This article is the case I&amp;#8217;d make to a colleague who has heard of Distrobox but hasn&amp;#8217;t pulled the trigger yet, plus the patterns I actually use day to&amp;nbsp;day.&lt;/p&gt;
&lt;h2 id="what-distrobox-actually-does"&gt;What Distrobox Actually&amp;nbsp;Does&lt;/h2&gt;
&lt;p&gt;Strip the marketing layer off and Distrobox is a shell script (a substantial one, but still a shell script) that&amp;nbsp;runs &lt;code&gt;podman create&lt;/code&gt; with a long list of flags you would not enjoy typing by hand. Those flags are what make a Distrobox container behave like a development sandbox instead of a server&amp;nbsp;workload:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Your&amp;nbsp;host &lt;code&gt;$HOME&lt;/code&gt; is bind-mounted into the container at the same path. Files you edit in the container are the same files on the&amp;nbsp;host.&lt;/li&gt;
&lt;li&gt;Your user is mapped into the container with the same &lt;span class="caps"&gt;UID&lt;/span&gt; and &lt;span class="caps"&gt;GID&lt;/span&gt;, so file ownership is consistent across the&amp;nbsp;boundary.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;/dev&lt;/code&gt;, &lt;code&gt;/tmp&lt;/code&gt;, the user runtime directory, the X11 and Wayland sockets, and the user systemd session are passed&amp;nbsp;through.&lt;/li&gt;
&lt;li&gt;Common host paths that matter for desktop integration&amp;nbsp;(&lt;code&gt;/run/user/&amp;lt;uid&amp;gt;&lt;/code&gt;, &lt;code&gt;/etc/hosts&lt;/code&gt;, the keyring socket) are wired up so that &lt;span class="caps"&gt;GUI&lt;/span&gt; applications, &lt;span class="caps"&gt;GPG&lt;/span&gt;, &lt;span class="caps"&gt;SSH&lt;/span&gt; agents and so on transparently see the host&amp;nbsp;versions.&lt;/li&gt;
&lt;li&gt;Distrobox can run a lightweight init or, if you ask for it, a full systemd session inside the container, so the box behaves more like a persistent user environment than a&amp;nbsp;disposable &lt;code&gt;podman run --rm&lt;/code&gt; shell.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In other words, it is &lt;em&gt;not&lt;/em&gt; an ephemeral container that throws away state when you exit. It is a long-lived container with your identity inside it. Closer in spirit to a chroot or a &lt;span class="caps"&gt;VM&lt;/span&gt; than to&amp;nbsp;a &lt;code&gt;podman run --rm&lt;/code&gt; one-off.&lt;/p&gt;
&lt;p&gt;Concretely, the basic workflow&amp;nbsp;is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# create a Fedora 43 box called &amp;quot;fedora-dev&amp;quot;&lt;/span&gt;
distrobox&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;fedora-dev&lt;span class="w"&gt; &lt;/span&gt;--image&lt;span class="w"&gt; &lt;/span&gt;registry.fedoraproject.org/fedora-toolbox:43

&lt;span class="c1"&gt;# step into it&lt;/span&gt;
distrobox&lt;span class="w"&gt; &lt;/span&gt;enter&lt;span class="w"&gt; &lt;/span&gt;fedora-dev

&lt;span class="c1"&gt;# inside, you are &amp;lt;your-username&amp;gt; on Fedora 43, with $HOME mounted&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;rustup&lt;span class="w"&gt; &lt;/span&gt;gcc-c++&lt;span class="w"&gt; &lt;/span&gt;openssl-devel
rustup&lt;span class="w"&gt; &lt;/span&gt;default&lt;span class="w"&gt; &lt;/span&gt;stable
cargo&lt;span class="w"&gt; &lt;/span&gt;build
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Exit, come back tomorrow, and the box is still there with the same Rust toolchain. List your boxes&amp;nbsp;with &lt;code&gt;distrobox list&lt;/code&gt;, remove one&amp;nbsp;with &lt;code&gt;distrobox rm &amp;lt;name&amp;gt;&lt;/code&gt;, upgrade what&amp;#8217;s inside&amp;nbsp;with &lt;code&gt;distrobox upgrade &amp;lt;name&amp;gt;&lt;/code&gt; (which runs the distribution&amp;#8217;s package manager&amp;nbsp;non-interactively).&lt;/p&gt;
&lt;h2 id="why-i-use-it"&gt;Why I Use&amp;nbsp;It&lt;/h2&gt;
&lt;h3 id="multiple-distributions"&gt;Multiple&amp;nbsp;Distributions&lt;/h3&gt;
&lt;p&gt;The headline feature is also the most useful one in practice. On a single Fedora workstation I have, at various times,&amp;nbsp;run:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;fedora-toolbox:43&lt;/code&gt; for the default development&amp;nbsp;environment.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;registry.access.redhat.com/ubi9/ubi&lt;/code&gt; for testing playbooks against a &lt;span class="caps"&gt;RHEL&lt;/span&gt; 9 control&amp;nbsp;node.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;registry.access.redhat.com/ubi10/ubi&lt;/code&gt; for the same against &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;10.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;docker.io/library/archlinux&lt;/code&gt; for a project where someone insisted on an &lt;span class="caps"&gt;AUR&lt;/span&gt;&amp;nbsp;dependency.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;quay.io/toolbx/ubuntu-toolbox:24.04&lt;/code&gt; for one vendor binary shipped only as&amp;nbsp;a &lt;code&gt;.deb&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;registry.opensuse.org/opensuse/distrobox:tumbleweed&lt;/code&gt; to reproduce a Tumbleweed bug&amp;nbsp;report.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Each one is a separate box with its&amp;nbsp;own &lt;code&gt;dnf&lt;/code&gt; / &lt;code&gt;apt&lt;/code&gt; / &lt;code&gt;pacman&lt;/code&gt; / &lt;code&gt;zypper&lt;/code&gt;. None of them touch the host&amp;#8217;s package database. None of them care that the host is&amp;nbsp;Fedora.&lt;/p&gt;
&lt;p&gt;The list of &amp;#8220;supported&amp;#8221; base images is long but unfussy: anything with a working package manager and a reasonable init can be made to work, and the &lt;a href="https://distrobox.it/compatibility/"&gt;project documentation&lt;/a&gt; catalogues which images Distrobox knows how to bootstrap&amp;nbsp;automatically.&lt;/p&gt;
&lt;h3 id="per-project-toolchains"&gt;Per-Project&amp;nbsp;Toolchains&lt;/h3&gt;
&lt;p&gt;This is where Distrobox earns its keep on a daily basis. Each project gets a box. The box gets exactly the toolchain that project needs, pinned to the distribution where that toolchain is least painful (Arch for bleeding-edge, &lt;span class="caps"&gt;RHEL&lt;/span&gt; for &amp;#8220;this customer is on &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8221;, Ubuntu for &amp;#8220;this vendor only tests on Ubuntu&amp;#8221;). When the project&amp;nbsp;ends, &lt;code&gt;distrobox rm&lt;/code&gt; and the entire toolchain is gone with it. No &amp;#8220;I think I can uninstall this now&amp;#8221; archaeology,&amp;nbsp;no &lt;code&gt;pip uninstall&lt;/code&gt; chains, no leftover systemd&amp;nbsp;units.&lt;/p&gt;
&lt;p&gt;There is a real productivity payoff to being unafraid to throw an environment away. I create boxes more readily now than I ever created venvs, because the cost of recreation is measured in a&amp;nbsp;single &lt;code&gt;distrobox create&lt;/code&gt; plus whatever&amp;nbsp;your &lt;code&gt;dnf install&lt;/code&gt; line&amp;nbsp;is.&lt;/p&gt;
&lt;h3 id="vscode-integration-is-excellent"&gt;VSCode Integration is&amp;nbsp;Excellent&lt;/h3&gt;
&lt;p&gt;This is the one I genuinely did not expect to like as much as I do. VSCode&amp;#8217;s Remote Containers / Remote &lt;span class="caps"&gt;SSH&lt;/span&gt; stack treats a Distrobox container as a first-class remote target. From the&amp;nbsp;host:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;distrobox&lt;span class="w"&gt; &lt;/span&gt;enter&lt;span class="w"&gt; &lt;/span&gt;fedora-dev&lt;span class="w"&gt; &lt;/span&gt;--&lt;span class="w"&gt; &lt;/span&gt;code&lt;span class="w"&gt; &lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;VSCode launches on the host (with all your host-level settings and accounts) but the language servers, debuggers, build tools and integrated terminal all run &lt;em&gt;inside&lt;/em&gt; the container. The C++ extension finds the&amp;nbsp;container&amp;#8217;s &lt;code&gt;gcc&lt;/code&gt;. The Python extension finds the container&amp;#8217;s interpreter. The Rust analyzer runs against the&amp;nbsp;container&amp;#8217;s &lt;code&gt;cargo&lt;/code&gt;. No &amp;#8220;Dev&amp;nbsp;Containers&amp;#8221; &lt;code&gt;devcontainer.json&lt;/code&gt; ceremony required, no rebuilds when you change a setting; it is just a remote, and the remote happens to share your home directory so saving a file is&amp;nbsp;instantaneous.&lt;/p&gt;
&lt;p&gt;Worth being honest about the trade-off here. Unlike Dev Containers, the environment is stateful and long-lived by default, so you do not get the strict reproducibility guarantees of&amp;nbsp;a &lt;code&gt;devcontainer.json&lt;/code&gt;-driven build. You are choosing convenience and integration over a hermetic, repo-pinned environment. For a project where reproducibility is the goal (a &lt;span class="caps"&gt;CI&lt;/span&gt; image, an Ansible Execution Environment, a shipping artifact), build a real &lt;span class="caps"&gt;OCI&lt;/span&gt; image. For everyday development, the convenience usually&amp;nbsp;wins.&lt;/p&gt;
&lt;p&gt;For me this killed the last argument for installing development toolchains on the host at&amp;nbsp;all.&lt;/p&gt;
&lt;h3 id="atomic-desktops-mutability-where-you-need-it"&gt;Atomic Desktops: Mutability Where You Need&amp;nbsp;It&lt;/h3&gt;
&lt;p&gt;If you are running an Atomic distribution (Fedora Silverblue, Kinoite, Sericea, Onyx, or Fedora CoreOS / IoT for the server case), Distrobox stops being a nice-to-have and becomes the &lt;em&gt;intended&lt;/em&gt; path for development work. I covered the broader Atomic story in &lt;a href="https://hofstede.it/immutable-linux-desktops-universal-blue-ostree-and-the-future-of-desktop-linux/"&gt;Universal Blue and Atomic Desktops&lt;/a&gt;; the short version is that the host filesystem is read-only and assembled from an OSTree commit, so you do not install development packages on it. You install them inside a Distrobox container, which is mutable, snapshottable, and&amp;nbsp;disposable.&lt;/p&gt;
&lt;p&gt;The mental model that finally clicked for me on Silverblue: the host is the appliance; Distrobox boxes are the workstations. You stop treating the host as a place where things accumulate, and you stop being precious about any single&amp;nbsp;box.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;rpm-ostree&lt;/code&gt; layering is still there as an escape hatch, but every package you layer is one more thing you are signing up to maintain across upgrades. Distrobox lets you keep the layering set as small as it should be: usually just shell, editor, and the handful of host tools that genuinely need to be on the&amp;nbsp;host.&lt;/p&gt;
&lt;h3 id="what-actually-stays-on-the-host"&gt;What Actually Stays on the&amp;nbsp;Host&lt;/h3&gt;
&lt;p&gt;A reasonable question once you start moving toolchains into boxes: what is the host &lt;em&gt;for&lt;/em&gt;, then? My layered-on-the-host set on Silverblue and Kinoite is short and&amp;nbsp;stable:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;The shell and core terminal tooling&amp;nbsp;(&lt;code&gt;fish&lt;/code&gt; / &lt;code&gt;zsh&lt;/code&gt;, &lt;code&gt;tmux&lt;/code&gt;, &lt;code&gt;git&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;The editor &lt;em&gt;frontend&lt;/em&gt;&amp;nbsp;(&lt;code&gt;code&lt;/code&gt;, &lt;code&gt;neovim&lt;/code&gt;), even though the language servers and build tools live in the&amp;nbsp;boxes.&lt;/li&gt;
&lt;li&gt;A browser, where it is not already a&amp;nbsp;Flatpak.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;podman&lt;/code&gt; and &lt;code&gt;distrobox&lt;/code&gt; themselves (these come on the base image on Atomic spins&amp;nbsp;anyway).&lt;/li&gt;
&lt;li&gt;The virtualization stack if I run VMs&amp;nbsp;(&lt;code&gt;virt-manager&lt;/code&gt;, &lt;code&gt;libvirt&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;A small amount of desktop glue&amp;nbsp;(&lt;code&gt;gnome-tweaks&lt;/code&gt;, codecs, drivers that genuinely need host kernel&amp;nbsp;access).&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;That is roughly it. Compilers, language runtimes, database clients, infrastructure tools, anything project-shaped, lives in a box. The host stays close to the upstream image, which is the entire point of an Atomic&amp;nbsp;distribution.&lt;/p&gt;
&lt;h3 id="rhel-userspace-on-fedora"&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; Userspace on&amp;nbsp;Fedora&lt;/h3&gt;
&lt;p&gt;This is the trick that surprised me the first time it worked. On a Fedora host that is &lt;em&gt;itself&lt;/em&gt; registered&amp;nbsp;with &lt;code&gt;subscription-manager&lt;/code&gt;, you can create a Distrobox container from a Red Hat &lt;span class="caps"&gt;UBI&lt;/span&gt; image, and the resulting container behaves much like a regular &lt;span class="caps"&gt;RHEL&lt;/span&gt; userspace with access to the standard &lt;span class="caps"&gt;RHEL&lt;/span&gt; repositories. No flags, no manual bind&amp;nbsp;mounts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# one-time, on the host&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;subscription-manager
sudo&lt;span class="w"&gt; &lt;/span&gt;subscription-manager&lt;span class="w"&gt; &lt;/span&gt;register

&lt;span class="c1"&gt;# then, as your normal user&lt;/span&gt;
distrobox&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;rhel10&lt;span class="w"&gt; &lt;/span&gt;--image&lt;span class="w"&gt; &lt;/span&gt;registry.access.redhat.com/ubi10/ubi
distrobox&lt;span class="w"&gt; &lt;/span&gt;enter&lt;span class="w"&gt; &lt;/span&gt;rhel10
sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;ansible-core&lt;span class="w"&gt; &lt;/span&gt;gcc&lt;span class="w"&gt; &lt;/span&gt;make&lt;span class="w"&gt; &lt;/span&gt;tmux
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The reason this just works is that Fedora ships Podman with a default&amp;nbsp;in &lt;code&gt;/usr/share/containers/mounts.conf&lt;/code&gt; that propagates the host&amp;#8217;s &lt;span class="caps"&gt;RHEL&lt;/span&gt; entitlements into every container&amp;nbsp;automatically:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ cat /usr/share/containers/mounts.conf
/usr/share/rhel/secrets:/run/secrets

$ ls -l /usr/share/rhel/secrets
etc-pki-entitlement -&amp;gt; ../../../../etc/pki/entitlement
rhsm                -&amp;gt; ../../../../etc/rhsm
redhat.repo         -&amp;gt; ../../../../etc/yum.repos.d/redhat.repo
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Those symlinks point at the&amp;nbsp;host&amp;#8217;s &lt;code&gt;/etc/pki/entitlement&lt;/code&gt;, &lt;code&gt;/etc/rhsm&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;redhat.repo&lt;/code&gt;. Podman binds the whole tree into the container&amp;nbsp;at &lt;code&gt;/run/secrets&lt;/code&gt;, &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s &lt;code&gt;subscription-manager&lt;/code&gt; plugin&amp;nbsp;for &lt;code&gt;dnf&lt;/code&gt; knows to look there, and the&amp;nbsp;container&amp;#8217;s &lt;code&gt;dnf&lt;/code&gt; quietly starts seeing the&amp;nbsp;host&amp;#8217;s &lt;code&gt;BaseOS&lt;/code&gt;, &lt;code&gt;AppStream&lt;/code&gt; and &lt;code&gt;CodeReady&lt;/code&gt; repositories. Distrobox inherits this for free because it is&amp;nbsp;just &lt;code&gt;podman create&lt;/code&gt; with extra mounts; it doesn&amp;#8217;t fight the&amp;nbsp;default.&lt;/p&gt;
&lt;p&gt;The userspace is &lt;span class="caps"&gt;RHEL&lt;/span&gt;, the package set is &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s, and the kernel is whatever the host runs. That is enough to reproduce most customer issues, build against the same library versions production will run, or&amp;nbsp;run &lt;code&gt;ansible-navigator&lt;/code&gt; inside the same &lt;span class="caps"&gt;OS&lt;/span&gt; your control plane&amp;nbsp;uses.&lt;/p&gt;
&lt;p&gt;If you do not have a host subscription, &lt;span class="caps"&gt;UBI&lt;/span&gt; on its own is still useful (it will get&amp;nbsp;you &lt;code&gt;ansible-core&lt;/code&gt;, &lt;code&gt;python3&lt;/code&gt;, the standard build toolchain), it just won&amp;#8217;t pull from the full repository set. On a non-Fedora-derivative host that doesn&amp;#8217;t ship the&amp;nbsp;same &lt;code&gt;mounts.conf&lt;/code&gt; default, you can either drop in an equivalent file yourself, register the container directly&amp;nbsp;with &lt;code&gt;subscription-manager&lt;/code&gt; inside it, or pass the entitlement directories in by&amp;nbsp;hand.&lt;/p&gt;
&lt;h2 id="a-few-patterns-worth-knowing"&gt;A Few Patterns Worth&amp;nbsp;Knowing&lt;/h2&gt;
&lt;h3 id="exporting-applications-and-binaries-to-the-host"&gt;Exporting Applications and Binaries to the&amp;nbsp;Host&lt;/h3&gt;
&lt;p&gt;A box that you can only use by&amp;nbsp;running &lt;code&gt;distrobox enter&lt;/code&gt; is fine for shell work, but for desktop apps it would be tedious. Distrobox solves this&amp;nbsp;with &lt;code&gt;distrobox-export&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# inside the box&lt;/span&gt;
distrobox-export&lt;span class="w"&gt; &lt;/span&gt;--app&lt;span class="w"&gt; &lt;/span&gt;gimp&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="c1"&gt;# adds GIMP to the host&amp;#39;s application menu&lt;/span&gt;
distrobox-export&lt;span class="w"&gt; &lt;/span&gt;--bin&lt;span class="w"&gt; &lt;/span&gt;/usr/bin/cargo&lt;span class="w"&gt; &lt;/span&gt;--export-path&lt;span class="w"&gt; &lt;/span&gt;~/.local/bin
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The exported launcher is just a shim that&amp;nbsp;calls &lt;code&gt;distrobox enter &amp;lt;name&amp;gt; -- &amp;lt;command&amp;gt;&lt;/code&gt;. Your desktop sees it as a&amp;nbsp;normal &lt;code&gt;.desktop&lt;/code&gt; file or a normal binary&amp;nbsp;on &lt;code&gt;$PATH&lt;/code&gt;, and clicking the icon enters the right container automatically. This is the bit that makes having the Inkscape from one box and the Krita from another feel transparent rather than&amp;nbsp;fiddly.&lt;/p&gt;
&lt;h3 id="init-files-and-reproducible-boxes"&gt;Init Files and Reproducible&amp;nbsp;Boxes&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;distrobox create&lt;/code&gt; accepts &lt;code&gt;--init-hooks&lt;/code&gt; and &lt;code&gt;--pre-init-hooks&lt;/code&gt; for one-off creation-time setup, but the more useful pattern for me has been an &lt;a href="https://distrobox.it/usage/distrobox-assemble/"&gt;&lt;code&gt;assemble&lt;/code&gt;-style manifest&lt;/a&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# ~/.config/distrobox/distrobox.ini&lt;/span&gt;
&lt;span class="k"&gt;[fedora-dev]&lt;/span&gt;
&lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry.fedoraproject.org/fedora-toolbox:43&lt;/span&gt;
&lt;span class="na"&gt;additional_packages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;git tmux gcc rustup&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;init_hooks&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;rustup default stable&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;[rhel10]&lt;/span&gt;
&lt;span class="na"&gt;image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry.access.redhat.com/ubi10/ubi&lt;/span&gt;
&lt;span class="na"&gt;additional_flags&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;--volume /etc/pki/entitlement:/etc/pki/entitlement:ro --volume /etc/rhsm:/etc/rhsm:ro&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;additional_packages&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;ansible-core git tmux&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;distrobox&lt;span class="w"&gt; &lt;/span&gt;assemble&lt;span class="w"&gt; &lt;/span&gt;create
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now a fresh laptop is&amp;nbsp;one &lt;code&gt;distrobox assemble create&lt;/code&gt; away from having every box I rely on, with the right packages pre-installed. This is what I sync between machines, not the boxes&amp;nbsp;themselves.&lt;/p&gt;
&lt;h3 id="btrfs-subvolumes-for-per-box-snapshots"&gt;Btrfs Subvolumes for Per-Box&amp;nbsp;Snapshots&lt;/h3&gt;
&lt;p&gt;If&amp;nbsp;your &lt;code&gt;containers/storage&lt;/code&gt; lives on Btrfs (the default on modern Fedora and Atomic spins), Podman uses&amp;nbsp;the &lt;code&gt;overlay&lt;/code&gt; driver on top of it, and the entire container graph sits inside a subvolume tree&amp;nbsp;at &lt;code&gt;~/.local/share/containers/storage/&lt;/code&gt;. Two consequences worth&amp;nbsp;knowing:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;You can take&amp;nbsp;a &lt;code&gt;btrfs subvolume snapshot&lt;/code&gt; of the whole container store before a major change (a system upgrade, an&amp;nbsp;experimental &lt;code&gt;dnf install&lt;/code&gt; spree inside a box) and roll back if it goes&amp;nbsp;wrong.&lt;/li&gt;
&lt;li&gt;The lightweight version of this, which is what I actually do most days,&amp;nbsp;is &lt;code&gt;Timeshift&lt;/code&gt;-style hourly snapshots&amp;nbsp;of &lt;code&gt;~/.local/share/containers/&lt;/code&gt;. Disk is cheap; an evening of lost configuration inside a box is&amp;nbsp;not.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;If you want per-box granularity (snapshotting one important box without touching the rest), that gets&amp;nbsp;into &lt;code&gt;containers.conf&lt;/code&gt; and&amp;nbsp;Podman&amp;#8217;s &lt;code&gt;additionalimagestores&lt;/code&gt;, which is its own small adventure. Worth the effort for the long-lived &lt;span class="caps"&gt;RHEL&lt;/span&gt; box you do not want to rebuild, probably not worth it for the throwaway boxes that make up most of the&amp;nbsp;list.&lt;/p&gt;
&lt;h3 id="nesting-and-rootful-boxes"&gt;Nesting and Rootful&amp;nbsp;Boxes&lt;/h3&gt;
&lt;p&gt;Two common&amp;nbsp;gotchas:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Nested containers.&lt;/strong&gt; Running Podman or building &lt;span class="caps"&gt;OCI&lt;/span&gt; images &lt;em&gt;inside&lt;/em&gt; a Distrobox box works, but the inner Podman needs the right namespaces. The most reliable path I have found is to&amp;nbsp;pass &lt;code&gt;--unshare-netns --unshare-ipc&lt;/code&gt; at create time and to use&amp;nbsp;the &lt;code&gt;vfs&lt;/code&gt; storage driver for the inner Podman&amp;nbsp;(set &lt;code&gt;storage.driver = "vfs"&lt;/code&gt; in the&amp;nbsp;box&amp;#8217;s &lt;code&gt;~/.config/containers/storage.conf&lt;/code&gt;). It is slower than overlay, but it sidesteps the rootless overlay-on-overlay problem. For &amp;#8220;build and test a container image&amp;#8221; workflows where speed matters, just run Podman on the host and use the box for the&amp;nbsp;toolchain.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rootful&amp;nbsp;boxes.&lt;/strong&gt; &lt;code&gt;distrobox create --root&lt;/code&gt; runs the container as the host&amp;#8217;s root user. You almost never want this for development; it exists for cases where the workload genuinely needs root capabilities. For the curious: it interacts cleanly&amp;nbsp;with &lt;code&gt;sudo&lt;/code&gt;, but you lose some of the seamless home-directory&amp;nbsp;integration.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="where-distrobox-stops-being-the-right-tool"&gt;Where Distrobox Stops Being the Right&amp;nbsp;Tool&lt;/h2&gt;
&lt;p&gt;Distrobox is for development environments and developer-adjacent workflows. It is not what I reach for when I&amp;nbsp;want:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Production workloads.&lt;/strong&gt; Use Podman directly, with &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Quadlets&lt;/a&gt;, or Kubernetes, or whatever your platform of choice is. A Distrobox container has too many host things mounted into it to be a sensible production&amp;nbsp;unit.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Reproducible &lt;span class="caps"&gt;CI&lt;/span&gt;.&lt;/strong&gt;&amp;nbsp;Use &lt;code&gt;ansible-builder&lt;/code&gt; and an &lt;a href="https://hofstede.it/reproducible-ansible-with-execution-environments/"&gt;Execution Environment&lt;/a&gt;, or a regular &lt;span class="caps"&gt;OCI&lt;/span&gt; build, where the artifact is the image. Distrobox is for stateful, interactive boxes, the opposite of what you want in a &lt;span class="caps"&gt;CI&lt;/span&gt;&amp;nbsp;runner.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Strong isolation.&lt;/strong&gt; Distrobox shares your home directory with the box. A compromised box can read&amp;nbsp;your &lt;code&gt;~/.ssh&lt;/code&gt; and your browser profile. If you need to sandbox something hostile, use a &lt;span class="caps"&gt;VM&lt;/span&gt; or a properly locked-down rootless container, not&amp;nbsp;Distrobox.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Within the development niche it is aimed at, I struggle to think of a tool I&amp;#8217;d rather have on a Linux laptop right now. Especially on Atomic systems, where it stops being a productivity nicety and becomes the way you do&amp;nbsp;work.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://distrobox.it/"&gt;Distrobox project&amp;nbsp;site&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://distrobox.it/compatibility/"&gt;Distrobox compatibility matrix&lt;/a&gt; (which images Distrobox knows how to&amp;nbsp;bootstrap)&lt;/li&gt;
&lt;li&gt;&lt;a href="https://containertoolbx.org/"&gt;Toolbx&lt;/a&gt;, the older Fedora project Distrobox was inspired by; if you only ever use Fedora-flavoured boxes on Fedora, it is a smaller, simpler&amp;nbsp;alternative.&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/immutable-linux-desktops-universal-blue-ostree-and-the-future-of-desktop-linux/"&gt;Universal Blue and Atomic Desktops&lt;/a&gt;, for the Atomic side of the&amp;nbsp;story.&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman in Production&lt;/a&gt;, for the engine&amp;nbsp;underneath.&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="distrobox"/><category term="podman"/><category term="containers"/><category term="linux"/><category term="fedora"/><category term="rhel"/><category term="silverblue"/><category term="kinoite"/><category term="atomic"/><category term="btrfs"/><category term="vscode"/><category term="development"/></entry><entry><title>Reproducible Ansible with Execution Environments</title><link href="https://hofstede.it/reproducible-ansible-with-execution-environments/" rel="alternate"/><published>2026-05-06T00:00:00+02:00</published><updated>2026-05-06T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-05-06:/reproducible-ansible-with-execution-environments/</id><summary type="html">&lt;p&gt;A &amp;#8220;control node&amp;#8221; with a hand-curated venv works. Until it doesn&amp;#8217;t. A colleague has different collection versions, &lt;span class="caps"&gt;CI&lt;/span&gt; installs a slightly newer Python library, and a playbook that ran yesterday now fails. Execution Environments turn the control node into a versioned container image. ansible-builder produces it, ansible-navigator runs against it, and the same artifact ships from your laptop into &lt;span class="caps"&gt;AAP&lt;/span&gt;/&lt;span class="caps"&gt;AWX&lt;/span&gt; without&amp;nbsp;modification.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-an-execution-environment-actually-contains"&gt;What an Execution Environment Actually&amp;nbsp;Contains&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-containerize-ansible-at-all"&gt;Why Containerize Ansible at&amp;nbsp;All&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-pieces"&gt;The&amp;nbsp;Pieces&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-worked-example-an-ee-for-homelab-automation"&gt;A Worked Example: An &lt;span class="caps"&gt;EE&lt;/span&gt; for Homelab Automation&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#execution-environmentyml"&gt;execution-environment.yml&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#requirementsyml"&gt;requirements.yml&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#requirementstxt"&gt;requirements.txt&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#bindeptxt"&gt;bindep.txt&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#building-it"&gt;Building&amp;nbsp;It&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#ansible-navigator-running-plays-against-the-ee"&gt;ansible-navigator: Running Plays Against the &lt;span class="caps"&gt;EE&lt;/span&gt;&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#other-navigator-subcommands-worth-knowing"&gt;Other Navigator Subcommands Worth&amp;nbsp;Knowing&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#ci-pinning-the-image-not-the-tool"&gt;&lt;span class="caps"&gt;CI&lt;/span&gt;: Pinning the Image, Not the&amp;nbsp;Tool&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#updating-an-ee"&gt;Updating an &lt;span class="caps"&gt;EE&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#trade-offs-and-when-its-overkill"&gt;Trade-offs and When It&amp;#8217;s&amp;nbsp;Overkill&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#summary"&gt;Summary&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#common-first-time-pitfalls"&gt;Common First-Time&amp;nbsp;Pitfalls&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;If you&amp;#8217;ve run Ansible for any non-trivial length of time, you&amp;#8217;ve hit the control-node problem. A laptop accumulates&amp;nbsp;a &lt;code&gt;~/.ansible/collections/&lt;/code&gt; tree, a venv with a dozen Python libraries pinned to&amp;nbsp;whatever &lt;code&gt;pip install&lt;/code&gt; gave you that afternoon, and a handful of distro packages&amp;nbsp;(&lt;code&gt;sshpass&lt;/code&gt;, &lt;code&gt;git&lt;/code&gt;,&amp;nbsp;sometimes &lt;code&gt;kerberos&lt;/code&gt; headers) that you only remember are there when a teammate clones the repo and the same playbook fails for them. &lt;span class="caps"&gt;CI&lt;/span&gt; papers over part of that with a fresh container per job, but only as long as you keep the install steps in lock-step with what the laptops are running, which is where it usually falls&amp;nbsp;apart.&lt;/p&gt;
&lt;p&gt;Execution Environments are the answer the Ansible project converged on for that whole class of problem. An Execution Environment (&lt;span class="caps"&gt;EE&lt;/span&gt;) is a container image that&amp;nbsp;bundles &lt;code&gt;ansible-core&lt;/code&gt;, &lt;code&gt;ansible-runner&lt;/code&gt;, your collections, your Python dependencies, and any system packages you need into a single artifact with a tag you can&amp;nbsp;pin. &lt;code&gt;ansible-builder&lt;/code&gt; is the tool that produces it from a declarative&amp;nbsp;definition; &lt;code&gt;ansible-navigator&lt;/code&gt; is what you run instead&amp;nbsp;of &lt;code&gt;ansible-playbook&lt;/code&gt; so the playbook executes inside that image. The same image is what &lt;span class="caps"&gt;AAP&lt;/span&gt; and &lt;span class="caps"&gt;AWX&lt;/span&gt; use natively, so the &amp;#8220;works on my laptop&amp;#8221; problem and the &amp;#8220;works in dev but not in production&amp;#8221; problem collapse into the same problem and get solved&amp;nbsp;once.&lt;/p&gt;
&lt;p&gt;This article is the practical walkthrough I&amp;#8217;d give a teammate who&amp;#8217;s just been told &amp;#8220;we&amp;#8217;re moving to EEs&amp;#8221;. What they are, why you&amp;#8217;d want one, how the two tools fit together, and the gotchas that aren&amp;#8217;t obvious from the&amp;nbsp;docs.&lt;/p&gt;
&lt;h2 id="what-an-execution-environment-actually-contains"&gt;What an Execution Environment Actually&amp;nbsp;Contains&lt;/h2&gt;
&lt;p&gt;Strip the marketing away and an &lt;span class="caps"&gt;EE&lt;/span&gt; is a fairly thin &lt;span class="caps"&gt;OCI&lt;/span&gt; image. Looking inside one of&amp;nbsp;mine:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;ansible&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;playbook&lt;/span&gt;&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ansible&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;core&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;CLI&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;rarely&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;called&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;directly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;EE&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;workflows&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;ansible&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;runner&lt;/span&gt;&lt;span class="w"&gt;           &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ansible&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;runner&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;process&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;supervisor&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;share&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;ansible&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;collections&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;collections&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;installed&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;at&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;build&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;time&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;lib&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;python3&lt;/span&gt;&lt;span class="m m-Double"&gt;.11&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;site&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;packages&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Python&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;deps&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;requests&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;kubernetes&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;etc&lt;/span&gt;&lt;span class="p"&gt;.)&lt;/span&gt;
&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;git&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;sshpass&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;system&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;packages&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;#8217;s it. There&amp;#8217;s no daemon, no Ansible &amp;#8220;engine&amp;#8221; beyond the standard tooling, and no special runtime.&amp;nbsp;When &lt;code&gt;ansible-navigator&lt;/code&gt; invokes the image, it bind-mounts your project directory and your inventory in,&amp;nbsp;runs &lt;code&gt;ansible-runner&lt;/code&gt; inside the container, streams stdout back, and exits. The image is the artifact; the tools around it are&amp;nbsp;scaffolding.&lt;/p&gt;
&lt;p&gt;The Python version is part of the contract too. If your collection only works on 3.11, the &lt;span class="caps"&gt;EE&lt;/span&gt; is where you lock that in, not a comment in the &lt;span class="caps"&gt;README&lt;/span&gt; that hopes everyone is paying&amp;nbsp;attention.&lt;/p&gt;
&lt;p&gt;Two things follow from&amp;nbsp;that:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;You own the image.&lt;/strong&gt; Whatever your playbooks need (a&amp;nbsp;specific &lt;code&gt;kubernetes&lt;/code&gt; Python client version,&amp;nbsp;the &lt;code&gt;community.crypto&lt;/code&gt; collection at a specific&amp;nbsp;tag, &lt;code&gt;openssl&lt;/code&gt; for ad-hoc certificate generation in a play) gets baked in once and shipped as a&amp;nbsp;unit.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The image is the contract.&lt;/strong&gt; Pin the image tag in &lt;span class="caps"&gt;CI&lt;/span&gt;, in &lt;span class="caps"&gt;AAP&lt;/span&gt;, and in your&amp;nbsp;local &lt;code&gt;ansible-navigator.yml&lt;/code&gt; and you&amp;#8217;ve eliminated most of the version-drift surface area between&amp;nbsp;environments.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="why-containerize-ansible-at-all"&gt;Why Containerize Ansible at&amp;nbsp;All&lt;/h2&gt;
&lt;p&gt;A reasonable objection: &amp;#8220;I already use a venv and&amp;nbsp;a &lt;code&gt;requirements.txt&lt;/code&gt;, why go further?&amp;#8221; A venv solves Python drift. It does nothing for system packages, and only partially solves collection drift. An &lt;span class="caps"&gt;EE&lt;/span&gt; is the first point where all three are versioned together. Three reasons that show up in&amp;nbsp;practice.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Collections drag in Python.&lt;/strong&gt; A modern Ansible playbook isn&amp;#8217;t just &lt;span class="caps"&gt;YAML&lt;/span&gt; calling &lt;span class="caps"&gt;SSH&lt;/span&gt;.&amp;nbsp;The &lt;code&gt;kubernetes.core&lt;/code&gt; collection&amp;nbsp;wants &lt;code&gt;kubernetes&lt;/code&gt;, &lt;code&gt;PyYAML&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;openshift&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;community.crypto&lt;/code&gt; collection&amp;nbsp;wants &lt;code&gt;cryptography&lt;/code&gt; at a version that does or doesn&amp;#8217;t have a particular &lt;span class="caps"&gt;API&lt;/span&gt;.&amp;nbsp;The &lt;code&gt;containers.podman&lt;/code&gt; collection (covered in the &lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/"&gt;Mastodon greeter article&lt;/a&gt;)&amp;nbsp;wants &lt;code&gt;requests&lt;/code&gt;. Multiply that across a real fleet&amp;#8217;s worth of collections and &amp;#8220;I&amp;#8217;ll&amp;nbsp;just &lt;code&gt;pip install&lt;/code&gt; what I need&amp;#8221; becomes a coordination problem. Bake it into an image and the coordination is a Git&amp;nbsp;diff.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;System packages&amp;nbsp;exist.&lt;/strong&gt; &lt;code&gt;community.crypto&lt;/code&gt; shells out&amp;nbsp;to &lt;code&gt;openssl&lt;/code&gt; for some operations. Anything touching &lt;span class="caps"&gt;IPMI&lt;/span&gt;&amp;nbsp;wants &lt;code&gt;ipmitool&lt;/code&gt;. &lt;code&gt;community.general.kerberos_ticket&lt;/code&gt; wants the &lt;span class="caps"&gt;MIT&lt;/span&gt; Kerberos client libraries. Putting those into a venv is impossible; putting them into an &lt;span class="caps"&gt;EE&lt;/span&gt; is one line in a build&amp;nbsp;manifest.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;AAP&lt;/span&gt;/&lt;span class="caps"&gt;AWX&lt;/span&gt; speak EEs natively.&lt;/strong&gt; If you ship work to a controller, the controller already runs your playbooks inside an &lt;span class="caps"&gt;EE&lt;/span&gt;; the only question is whose. Either you produce it (and your laptop runs the same image you deploy) or Red Hat&amp;#8217;s default does (and your laptop drifts from production). The first option is strictly better and not measurably more work once you&amp;#8217;ve done it&amp;nbsp;once.&lt;/p&gt;
&lt;p&gt;The flip side, which I&amp;#8217;ll come back to under trade-offs: an &lt;span class="caps"&gt;EE&lt;/span&gt; adds a container start to every play, which is on the order of hundreds of milliseconds. For interactive ad-hoc work it&amp;#8217;s noticeable. For anything reviewed and merged, it&amp;#8217;s&amp;nbsp;free.&lt;/p&gt;
&lt;h2 id="the-pieces"&gt;The&amp;nbsp;Pieces&lt;/h2&gt;
&lt;p&gt;Three tools, each with a specific&amp;nbsp;job:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible-builder&lt;/code&gt;&lt;/strong&gt; consumes&amp;nbsp;an &lt;code&gt;execution-environment.yml&lt;/code&gt; definition and produces a container image. It&amp;#8217;s a Python package; on the host it relies&amp;nbsp;on &lt;code&gt;podman&lt;/code&gt; to do the actual&amp;nbsp;building.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible-navigator&lt;/code&gt;&lt;/strong&gt; is the local replacement&amp;nbsp;for &lt;code&gt;ansible-playbook&lt;/code&gt;. It runs your playbook inside a chosen &lt;span class="caps"&gt;EE&lt;/span&gt; image, with either a &lt;span class="caps"&gt;TUI&lt;/span&gt; or plain stdout. It also&amp;nbsp;wraps &lt;code&gt;ansible-doc&lt;/code&gt;, &lt;code&gt;ansible-config&lt;/code&gt;, and a few inspection commands so you can poke at collections inside the image without dropping into a&amp;nbsp;shell.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible-runner&lt;/code&gt;&lt;/strong&gt; lives inside the &lt;span class="caps"&gt;EE&lt;/span&gt; image and is the process that actually drives a play. You rarely interact with it&amp;nbsp;directly; &lt;code&gt;ansible-navigator&lt;/code&gt; calls it for you, and &lt;span class="caps"&gt;AAP&lt;/span&gt;/&lt;span class="caps"&gt;AWX&lt;/span&gt; do the same on the controller&amp;nbsp;side.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A practical mental&amp;nbsp;model: &lt;code&gt;ansible-builder&lt;/code&gt; is &lt;code&gt;podman build&lt;/code&gt; for Ansible (with a&amp;nbsp;generated &lt;code&gt;Containerfile&lt;/code&gt; you don&amp;#8217;t have to maintain&amp;nbsp;yourself), &lt;code&gt;ansible-navigator&lt;/code&gt; is &lt;code&gt;podman run&lt;/code&gt; for Ansible,&amp;nbsp;and &lt;code&gt;ansible-runner&lt;/code&gt; is the entrypoint inside the container that does the&amp;nbsp;work.&lt;/p&gt;
&lt;h2 id="a-worked-example-an-ee-for-homelab-automation"&gt;A Worked Example: An &lt;span class="caps"&gt;EE&lt;/span&gt; for Homelab&amp;nbsp;Automation&lt;/h2&gt;
&lt;p&gt;To make this concrete, here&amp;#8217;s the &lt;span class="caps"&gt;EE&lt;/span&gt; I use for the small fleet that runs the &lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/"&gt;mastogreet&lt;/a&gt; bot, the Quadlet-managed Forgejo from the &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;original Quadlet article&lt;/a&gt;, and a handful of other &lt;span class="caps"&gt;RHEL&lt;/span&gt; boxes. It&amp;#8217;s not large; that&amp;#8217;s the&amp;nbsp;point.&lt;/p&gt;
&lt;p&gt;The directory&amp;nbsp;layout:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;ansible&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;ee&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;homelab&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;execution&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;environment&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;yml&lt;/span&gt;
&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;requirements&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;yml&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Galaxy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;collections&lt;/span&gt;
&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;requirements&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;txt&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Python&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;packages&lt;/span&gt;
&lt;span class="err"&gt;└──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;bindep&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nx"&gt;txt&lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;system&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;packages&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="execution-environmentyml"&gt;&lt;code&gt;execution-environment.yml&lt;/code&gt;&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;3&lt;/span&gt;

&lt;span class="nt"&gt;images&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;base_image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;quay.io/centos/centos:stream9&lt;/span&gt;

&lt;span class="nt"&gt;dependencies&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible_core&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;package_pip&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ansible-core&amp;gt;=2.17,&amp;lt;2.18&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible_runner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;package_pip&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ansible-runner&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;galaxy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;requirements.yml&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;python&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;requirements.txt&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;system&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;bindep.txt&lt;/span&gt;

&lt;span class="nt"&gt;additional_build_steps&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;append_final&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;LABEL org.opencontainers.image.source=&amp;quot;https://git.linuxserver.pro/chris/ansible-ee-homelab&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;LABEL org.opencontainers.image.description=&amp;quot;Homelab Ansible EE&lt;/span&gt;&lt;span class="p p-Indicator"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;containers.podman, community.general, ansible.posix&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few things worth pointing&amp;nbsp;out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;version: 3&lt;/code&gt;&lt;/strong&gt; is the current schema (introduced&amp;nbsp;with &lt;code&gt;ansible-builder&lt;/code&gt; 3.x). v1 is gone, v2 still works but lacks the per-step granularity v3 gives you. New EEs should always be&amp;nbsp;v3.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;base_image&lt;/code&gt;&lt;/strong&gt;. CentOS Stream 9 here because it tracks the same &lt;span class="caps"&gt;RHEL&lt;/span&gt; minor that my hosts run, which keeps the system-package versions inside the &lt;span class="caps"&gt;EE&lt;/span&gt; close to what&amp;#8217;s actually deployed. Red&amp;nbsp;Hat&amp;#8217;s &lt;code&gt;registry.redhat.io/ansible-automation-platform-25/ee-minimal-rhel9&lt;/code&gt; is the supported equivalent if you have a subscription; the Ansible community&amp;nbsp;publishes &lt;code&gt;quay.io/ansible/community-ee-minimal&lt;/code&gt; and &lt;code&gt;community-ee-base&lt;/code&gt; as smaller starting points if you want a curated base. All three work. I use the CentOS Stream image because the &lt;a href="https://hofstede.it/why-i-prefer-centos-stream-over-old-centos/"&gt;Stream-better-than-old-CentOS&lt;/a&gt; reasoning applies here too: a stable &lt;span class="caps"&gt;RHEL&lt;/span&gt;-equivalent base without a subscription&amp;nbsp;requirement.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible_core&lt;/code&gt; pinned to a minor&amp;nbsp;range.&lt;/strong&gt; &lt;code&gt;&amp;gt;=2.17,&amp;lt;2.18&lt;/code&gt; means I get patch updates within the line but not surprise behaviour changes from a major bump. Pin tighter&amp;nbsp;(&lt;code&gt;==2.17.4&lt;/code&gt;) if you have &lt;span class="caps"&gt;AAP&lt;/span&gt; version constraints to&amp;nbsp;match.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;additional_build_steps&lt;/code&gt;&lt;/strong&gt; is the escape hatch. Anything you&amp;#8217;d put into&amp;nbsp;a &lt;code&gt;Containerfile&lt;/code&gt; (&lt;span class="caps"&gt;LABEL&lt;/span&gt;, &lt;span class="caps"&gt;RUN&lt;/span&gt;, &lt;span class="caps"&gt;ENV&lt;/span&gt;) goes here, scoped by stage&amp;nbsp;(&lt;code&gt;prepend_base&lt;/code&gt;, &lt;code&gt;append_base&lt;/code&gt;, &lt;code&gt;prepend_galaxy&lt;/code&gt;, &lt;code&gt;append_galaxy&lt;/code&gt;, &lt;code&gt;prepend_final&lt;/code&gt;, &lt;code&gt;append_final&lt;/code&gt;). The v3 schema&amp;#8217;s stage granularity is the main reason to be on&amp;nbsp;it.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="requirementsyml"&gt;&lt;code&gt;requirements.yml&lt;/code&gt;&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="nt"&gt;collections&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;containers.podman&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;=1.16.0,&amp;lt;2.0.0&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;community.general&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;=10.0.0&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ansible.posix&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;=1.6.0&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;community.crypto&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;version&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;&amp;gt;=2.20.0&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Standard Galaxy requirements. Pinning the major helps; pinning the patch is overkill unless you&amp;#8217;re chasing a specific&amp;nbsp;bug.&lt;/p&gt;
&lt;h3 id="requirementstxt"&gt;&lt;code&gt;requirements.txt&lt;/code&gt;&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;requests&amp;gt;=2.32
cryptography&amp;gt;=43
PyYAML&amp;gt;=6.0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The Python deps the collections actually&amp;nbsp;use. &lt;code&gt;containers.podman&lt;/code&gt; calls out&amp;nbsp;to &lt;code&gt;requests&lt;/code&gt; for some&amp;nbsp;operations, &lt;code&gt;community.crypto&lt;/code&gt; wants a&amp;nbsp;recent &lt;code&gt;cryptography&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;PyYAML&lt;/code&gt; is so universally needed that listing it explicitly is a form of&amp;nbsp;documentation.&lt;/p&gt;
&lt;h3 id="bindeptxt"&gt;&lt;code&gt;bindep.txt&lt;/code&gt;&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;git [platform:rpm]
openssh-clients [platform:rpm]
sshpass [platform:rpm]
openssl [platform:rpm]
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;bindep&lt;/code&gt; (a small dependency specification format Ansible uses for system packages) is&amp;nbsp;what &lt;code&gt;ansible-builder&lt;/code&gt; consumes here, with platform tags so the same file works&amp;nbsp;for &lt;code&gt;apt&lt;/code&gt;, &lt;code&gt;dnf&lt;/code&gt;, &lt;code&gt;apk&lt;/code&gt;, etc. A homelab &lt;span class="caps"&gt;EE&lt;/span&gt; almost always&amp;nbsp;wants &lt;code&gt;git&lt;/code&gt; (for &lt;code&gt;ansible.builtin.git&lt;/code&gt;), &lt;code&gt;openssh-clients&lt;/code&gt; (for &lt;span class="caps"&gt;SSH&lt;/span&gt; against odd-keyed&amp;nbsp;hosts), &lt;code&gt;sshpass&lt;/code&gt; (for the rare case you can&amp;#8217;t avoid password auth),&amp;nbsp;and &lt;code&gt;openssl&lt;/code&gt; (for &lt;code&gt;community.crypto&lt;/code&gt; operations that shell&amp;nbsp;out).&lt;/p&gt;
&lt;p&gt;One more entry that bites people on minimal base images: if your playbooks talk to internal &lt;span class="caps"&gt;HTTPS&lt;/span&gt; endpoints (a Forgejo registry, an internal Vault, anything behind a private &lt;span class="caps"&gt;CA&lt;/span&gt;),&amp;nbsp;add &lt;code&gt;ca-certificates&lt;/code&gt; and, if you use a private &lt;span class="caps"&gt;CA&lt;/span&gt;, install the cert&amp;nbsp;during &lt;code&gt;additional_build_steps&lt;/code&gt;. The &amp;#8220;why&amp;nbsp;does &lt;code&gt;requests&lt;/code&gt; fail with a &lt;span class="caps"&gt;TLS&lt;/span&gt; error inside the &lt;span class="caps"&gt;EE&lt;/span&gt; but not on my laptop&amp;#8221; debugging session is one of the most common first &lt;span class="caps"&gt;EE&lt;/span&gt;&amp;nbsp;experiences.&lt;/p&gt;
&lt;h3 id="building-it"&gt;Building&amp;nbsp;It&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible-builder&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--tag&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/ansible-ee-homelab:2026-05-06&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--container-runtime&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--verbosity&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;ansible-builder&lt;/code&gt; materialises&amp;nbsp;a &lt;code&gt;Containerfile&lt;/code&gt; in &lt;code&gt;context/&lt;/code&gt;, then hands it&amp;nbsp;to &lt;code&gt;podman build&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;--verbosity 2&lt;/code&gt; flag echoes each step so you can see exactly what it&amp;#8217;s doing; first-time builders should run it at least once to demystify the&amp;nbsp;process.&lt;/p&gt;
&lt;p&gt;The first build is slow (pulls the base image, runs Galaxy, pip, dnf). Subsequent builds are fast as long&amp;nbsp;as &lt;code&gt;requirements.yml&lt;/code&gt;, &lt;code&gt;requirements.txt&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;bindep.txt&lt;/code&gt; haven&amp;#8217;t changed; the v3 stage layout means only the layers downstream of the change rebuild. In practice that&amp;#8217;s the difference between a two-minute rebuild and a twenty-minute&amp;nbsp;one.&lt;/p&gt;
&lt;p&gt;After build, push to the&amp;nbsp;registry:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;push&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/ansible-ee-homelab:2026-05-06
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now the image is the artifact. Anyone with credentials to the registry can run playbooks against the same versioned bundle of&amp;nbsp;dependencies.&lt;/p&gt;
&lt;h2 id="ansible-navigator-running-plays-against-the-ee"&gt;ansible-navigator: Running Plays Against the &lt;span class="caps"&gt;EE&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;ansible-navigator&lt;/code&gt; is the new front door. It looks superficially&amp;nbsp;like &lt;code&gt;ansible-playbook&lt;/code&gt;, but with a couple of important differences. The simplest&amp;nbsp;invocation:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible-navigator&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;playbook.yml&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--execution-environment-image&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/ansible-ee-homelab:2026-05-06&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--pull-policy&lt;span class="w"&gt; &lt;/span&gt;missing&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--mode&lt;span class="w"&gt; &lt;/span&gt;stdout
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;--mode stdout&lt;/code&gt; produces output that looks exactly&amp;nbsp;like &lt;code&gt;ansible-playbook&lt;/code&gt; (which is what you want in &lt;span class="caps"&gt;CI&lt;/span&gt; and what most people want locally). Without it, navigator launches its &lt;span class="caps"&gt;TUI&lt;/span&gt;, which is genuinely useful for browsing task results across a multi-host run but a poor fit for terminals being scraped by a&amp;nbsp;tool.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;--pull-policy missing&lt;/code&gt; pulls the image if it&amp;#8217;s not already present locally and otherwise reuses it. The other policies&amp;nbsp;(&lt;code&gt;always&lt;/code&gt;, &lt;code&gt;never&lt;/code&gt;, &lt;code&gt;tag&lt;/code&gt;) cover the obvious&amp;nbsp;cases; &lt;code&gt;missing&lt;/code&gt; is the right default for almost&amp;nbsp;everyone.&lt;/p&gt;
&lt;p&gt;In practice, you don&amp;#8217;t pass these flags every&amp;nbsp;time. &lt;code&gt;ansible-navigator&lt;/code&gt; reads &lt;code&gt;ansible-navigator.yml&lt;/code&gt; from the project&amp;nbsp;directory:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nn"&gt;---&lt;/span&gt;
&lt;span class="nt"&gt;ansible-navigator&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;execution-environment&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;git.linuxserver.pro/chris/ansible-ee-homelab:2026-05-06&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;pull&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;policy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;missing&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;container-engine&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;podman&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;container-options&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;--net=host&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;stdout&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;playbook-artifact&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;enable&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few notes on this&amp;nbsp;config:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;container-options: --net=host&lt;/code&gt;&lt;/strong&gt; is sometimes necessary when your inventory uses host-local &lt;span class="caps"&gt;DNS&lt;/span&gt; or you&amp;#8217;re &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;#8217;ing to addresses only resolvable on the host&amp;#8217;s network namespace. It&amp;#8217;s a sledgehammer; if you can avoid it (by&amp;nbsp;feeding &lt;code&gt;/etc/hosts&lt;/code&gt; entries or using FQDNs that resolve via public &lt;span class="caps"&gt;DNS&lt;/span&gt;),&amp;nbsp;do.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;playbook-artifact: enable: false&lt;/code&gt;&lt;/strong&gt;. By&amp;nbsp;default &lt;code&gt;ansible-navigator&lt;/code&gt; writes a &lt;span class="caps"&gt;JSON&lt;/span&gt; artifact of every run into the project directory. Useful for the &lt;span class="caps"&gt;TUI&lt;/span&gt;&amp;#8217;s &amp;#8220;replay&amp;#8221; feature, noisy in a Git repo. Turn it off unless you&amp;#8217;re using replay&amp;nbsp;deliberately.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;mode: stdout&lt;/code&gt;&lt;/strong&gt; as the project default means you don&amp;#8217;t have to&amp;nbsp;pass &lt;code&gt;--mode stdout&lt;/code&gt; every time. The &lt;span class="caps"&gt;TUI&lt;/span&gt; is&amp;nbsp;one &lt;code&gt;--mode interactive&lt;/code&gt; away when you want&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; agent forwarding.&lt;/strong&gt; If you rely&amp;nbsp;on &lt;code&gt;ssh-agent&lt;/code&gt; for key access (the common case), the agent socket lives on the host and is invisible to the &lt;span class="caps"&gt;EE&lt;/span&gt; container by default. Pass it in&amp;nbsp;via &lt;code&gt;container-options&lt;/code&gt;: &lt;code&gt;-v $SSH_AUTH_SOCK:/ssh-agent&lt;/code&gt;,&amp;nbsp;then &lt;code&gt;-e SSH_AUTH_SOCK=/ssh-agent&lt;/code&gt;. The path inside the container is arbitrary; it just needs to match the environment variable. Without forwarding, the very first task that opens an &lt;span class="caps"&gt;SSH&lt;/span&gt; connection fails with no obviously-related error&amp;nbsp;message.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;With that file in place, the invocation collapses&amp;nbsp;to:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ansible-navigator&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;inventory&lt;span class="w"&gt; &lt;/span&gt;playbook.yml
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And the rest of the command-line surface&amp;nbsp;(&lt;code&gt;--limit&lt;/code&gt;, &lt;code&gt;--tags&lt;/code&gt;, &lt;code&gt;--ask-vault-pass&lt;/code&gt;, &lt;code&gt;--check&lt;/code&gt;, &lt;code&gt;--diff&lt;/code&gt;) works exactly the same&amp;nbsp;as &lt;code&gt;ansible-playbook&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="other-navigator-subcommands-worth-knowing"&gt;Other Navigator Subcommands Worth&amp;nbsp;Knowing&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;ansible-navigator&lt;/code&gt; also wraps a handful of Ansible&amp;#8217;s introspection tools, all of which run inside the &lt;span class="caps"&gt;EE&lt;/span&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible-navigator collections&lt;/code&gt;&lt;/strong&gt; lists the collections present in the image. This is the fastest way to verify your &lt;span class="caps"&gt;EE&lt;/span&gt; actually got built with what you asked&amp;nbsp;for.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible-navigator doc &amp;lt;module&amp;gt;&lt;/code&gt;&lt;/strong&gt;&amp;nbsp;runs &lt;code&gt;ansible-doc&lt;/code&gt; inside the &lt;span class="caps"&gt;EE&lt;/span&gt;. Useful when the module behaviour you care about is the version your &lt;span class="caps"&gt;EE&lt;/span&gt; has, not the version your laptop&amp;#8217;s outer Ansible&amp;nbsp;has.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible-navigator config&lt;/code&gt;&lt;/strong&gt; dumps the&amp;nbsp;effective &lt;code&gt;ansible.cfg&lt;/code&gt; as the &lt;span class="caps"&gt;EE&lt;/span&gt; sees it. Handy for &amp;#8220;why is this variable not being picked up in the&amp;nbsp;controller&amp;#8221;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;ansible-navigator inventory -i inventory&lt;/code&gt;&lt;/strong&gt; parses your inventory inside the &lt;span class="caps"&gt;EE&lt;/span&gt; and shows the merged result. Catches a fair number of &amp;#8220;the dynamic inventory plugin needs a Python lib I forgot to add&amp;nbsp;to &lt;code&gt;requirements.txt&lt;/code&gt;&amp;#8221;&amp;nbsp;issues.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The pattern is consistent: anything you used to invoke as a top-level Ansible command, navigator runs for you inside the chosen&amp;nbsp;image.&lt;/p&gt;
&lt;h2 id="ci-pinning-the-image-not-the-tool"&gt;&lt;span class="caps"&gt;CI&lt;/span&gt;: Pinning the Image, Not the&amp;nbsp;Tool&lt;/h2&gt;
&lt;p&gt;The &lt;span class="caps"&gt;CI&lt;/span&gt; story is where EEs really pay off. The &lt;a href="https://hofstede.it/speeding-up-forgejo-ci-with-a-custom-oci-image/"&gt;Forgejo Actions custom &lt;span class="caps"&gt;OCI&lt;/span&gt; image walkthrough&lt;/a&gt; covered the general &amp;#8220;build once, use in &lt;span class="caps"&gt;CI&lt;/span&gt;&amp;#8221; pattern; for Ansible specifically, an &lt;span class="caps"&gt;EE&lt;/span&gt; is the version of that pattern that &lt;span class="caps"&gt;AAP&lt;/span&gt; also&amp;nbsp;understands.&lt;/p&gt;
&lt;p&gt;A typical Forgejo Actions&amp;nbsp;step:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Run Ansible playbook&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;uses&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;https://code.forgejo.org/actions/run@v1&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;with&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;git.linuxserver.pro/chris/ansible-ee-homelab:2026-05-06&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;run&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;ansible-playbook -i inventory playbook.yml&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note that in &lt;span class="caps"&gt;CI&lt;/span&gt; you can&amp;nbsp;call &lt;code&gt;ansible-playbook&lt;/code&gt; directly inside the &lt;span class="caps"&gt;EE&lt;/span&gt; image, since the runner is already executing inside the container. Navigator adds no value here: you&amp;#8217;re already &amp;#8220;inside&amp;#8221; the &lt;span class="caps"&gt;EE&lt;/span&gt;. The indirection it provides locally is exactly what your &lt;span class="caps"&gt;CI&lt;/span&gt; runner is already doing. Navigator&amp;#8217;s role is local-developer ergonomics; in &lt;span class="caps"&gt;CI&lt;/span&gt;, the &lt;span class="caps"&gt;EE&lt;/span&gt; image is the runner, and the steps inside it are just&amp;nbsp;plays.&lt;/p&gt;
&lt;p&gt;The win is the pinned tag. The&amp;nbsp;same &lt;code&gt;2026-05-06&lt;/code&gt; image that the developer ran against locally is the one &lt;span class="caps"&gt;CI&lt;/span&gt; runs against, which is the one the controller in &lt;span class="caps"&gt;AAP&lt;/span&gt; runs against if you also use &lt;span class="caps"&gt;AAP&lt;/span&gt;. Three places, one artifact, no&amp;nbsp;drift.&lt;/p&gt;
&lt;h2 id="updating-an-ee"&gt;Updating an &lt;span class="caps"&gt;EE&lt;/span&gt;&lt;/h2&gt;
&lt;p&gt;Updates are a Git change and a rebuild. Bump a collection&amp;nbsp;in &lt;code&gt;requirements.yml&lt;/code&gt;, change the date in your image tag,&amp;nbsp;run &lt;code&gt;ansible-builder build&lt;/code&gt;, push, and point&amp;nbsp;your &lt;code&gt;ansible-navigator.yml&lt;/code&gt; and &lt;span class="caps"&gt;CI&lt;/span&gt; at the new tag in a pull request. Reviewers see the diff (which collections moved, which Python pins changed) and the build is reproducible because the previous tag still exists in the registry. Rolling back&amp;nbsp;is &lt;code&gt;git revert&lt;/code&gt; on the tag&amp;nbsp;pointer.&lt;/p&gt;
&lt;p&gt;The two patterns I rotate&amp;nbsp;between:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Date-tagged immutable images&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;...:2026-05-06&lt;/code&gt;). Rebuild on a cadence (monthly, or when something I depend on cuts a release I want), tag with the date, never re-tag. Slightly more friction; significantly more&amp;nbsp;auditability.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Channel tags&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;...:stable&lt;/code&gt;, &lt;code&gt;...:edge&lt;/code&gt;). Convenient for fast iteration, but you give up &amp;#8220;the image you ran against last week is still pullable&amp;#8221;. I use channel tags for a personal sandbox &lt;span class="caps"&gt;EE&lt;/span&gt; and date tags for anything&amp;nbsp;reviewed.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;For most people, start with date tags. Move to channel tags only if you have a clear reason&amp;nbsp;to.&lt;/p&gt;
&lt;p&gt;A subtle reproducibility point worth flagging: if&amp;nbsp;your &lt;code&gt;requirements.txt&lt;/code&gt; and &lt;code&gt;requirements.yml&lt;/code&gt; aren&amp;#8217;t tightly pinned, two rebuilds a week apart from the same Git tree can produce slightly different images. Galaxy and PyPI keep moving even if your repo doesn&amp;#8217;t. For the homelab &lt;span class="caps"&gt;EE&lt;/span&gt; I&amp;#8217;m fine with the loose pins above, because the date in the image tag captures &amp;#8220;what was current that day&amp;#8221;. For a regulated environment, pin to exact versions and treat the build as deterministic in the strict sense, not just &amp;#8220;repeatable&amp;nbsp;enough&amp;#8221;.&lt;/p&gt;
&lt;h2 id="trade-offs-and-when-its-overkill"&gt;Trade-offs and When It&amp;#8217;s&amp;nbsp;Overkill&lt;/h2&gt;
&lt;p&gt;The honest list of what an &lt;span class="caps"&gt;EE&lt;/span&gt;&amp;nbsp;costs:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Build infrastructure.&lt;/strong&gt; You need somewhere to&amp;nbsp;run &lt;code&gt;ansible-builder&lt;/code&gt;, somewhere to push the image, and credentials for both. For one person, this is a laptop and a free Quay account. For a team, it&amp;#8217;s a &lt;span class="caps"&gt;CI&lt;/span&gt; job and an internal&amp;nbsp;registry.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Container start latency.&lt;/strong&gt; Every play pays a one-shot cost (a few hundred milliseconds) for the &lt;span class="caps"&gt;EE&lt;/span&gt; container to start. Imperceptible on a real playbook; annoying on a one-liner&amp;nbsp;ad-hoc.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Indirection.&lt;/strong&gt; When something fails, you&amp;#8217;re now debugging a play running inside a container running on a&amp;nbsp;host. &lt;code&gt;ansible-navigator --mode stdout&lt;/code&gt; keeps the surface flat for the common case, but the occasional weird interaction (SELinux on a bind mount, an &lt;span class="caps"&gt;SSH&lt;/span&gt; agent socket not visible inside the &lt;span class="caps"&gt;EE&lt;/span&gt;) does cost some debugging time the first time it bites&amp;nbsp;you.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Discipline.&lt;/strong&gt; An &lt;span class="caps"&gt;EE&lt;/span&gt; only helps if you actually use it. Letting &amp;#8220;I&amp;#8217;ll&amp;nbsp;just &lt;code&gt;pip install&lt;/code&gt; this collection&amp;#8217;s new dep on my laptop and not bump the &lt;span class="caps"&gt;EE&lt;/span&gt;&amp;#8221; become a habit gives you the worst of both worlds: an &lt;span class="caps"&gt;EE&lt;/span&gt; that&amp;#8217;s notionally pinned but in practice a moving&amp;nbsp;target.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;The fastest way to break EEs is bypassing them.&lt;/strong&gt; Worth keeping front-of-mind, because the temptation to skip a rebuild for &amp;#8220;just this one tiny thing&amp;#8221; is exactly how the laptop-versus-&lt;span class="caps"&gt;CI&lt;/span&gt;-versus-&lt;span class="caps"&gt;AAP&lt;/span&gt; drift you bought the &lt;span class="caps"&gt;EE&lt;/span&gt; to avoid creeps back&amp;nbsp;in.&lt;/p&gt;
&lt;p&gt;When I&amp;#8217;d skip an &lt;span class="caps"&gt;EE&lt;/span&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Single-person scripting, two playbooks, one host. Just use a&amp;nbsp;venv.&lt;/li&gt;
&lt;li&gt;Throwaway experiments where the friction of a rebuild outweighs the audit&amp;nbsp;value.&lt;/li&gt;
&lt;li&gt;Teams that genuinely don&amp;#8217;t have collection sprawl: a couple&amp;nbsp;of &lt;code&gt;ansible.builtin&lt;/code&gt; modules,&amp;nbsp;no &lt;code&gt;kubernetes&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;crypto&lt;/code&gt;, no third-party Galaxy content. The drift surface isn&amp;#8217;t there to begin&amp;nbsp;with.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;When I&amp;#8217;d insist on&amp;nbsp;one:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Anything reviewed by more than one person. The &lt;span class="caps"&gt;EE&lt;/span&gt; is the &amp;#8220;this is what we agreed Ansible looks like&amp;#8221;&amp;nbsp;artifact.&lt;/li&gt;
&lt;li&gt;Anything running in &lt;span class="caps"&gt;CI&lt;/span&gt; and on a developer laptop. The image collapses the version&amp;nbsp;surface.&lt;/li&gt;
&lt;li&gt;Anything heading toward &lt;span class="caps"&gt;AAP&lt;/span&gt;/&lt;span class="caps"&gt;AWX&lt;/span&gt;. You&amp;#8217;ll need an &lt;span class="caps"&gt;EE&lt;/span&gt; there anyway; producing it yourself rather than inheriting a default is mostly free once the build pipeline&amp;nbsp;exists.&lt;/li&gt;
&lt;li&gt;Anything&amp;nbsp;touching &lt;code&gt;kubernetes.core&lt;/code&gt;, &lt;code&gt;community.crypto&lt;/code&gt;, &lt;code&gt;containers.podman&lt;/code&gt;, or any other collection with a non-trivial Python or system dependency. Those are the cases where &amp;#8220;works on my laptop&amp;#8221; most reliably&amp;nbsp;bites.&lt;/li&gt;
&lt;li&gt;Air-gapped or regulated environments with restricted outbound access. Pre-building the &lt;span class="caps"&gt;EE&lt;/span&gt; on a build host with internet access and shipping the image inward avoids runtime dependency installation entirely, which is often the only viable path through change-control&amp;nbsp;review.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="summary"&gt;Summary&lt;/h2&gt;
&lt;p&gt;Execution Environments are the version of &amp;#8220;containerize your toolchain&amp;#8221; that the Ansible project already uses internally. The deal is straightforward: write a small declarative manifest,&amp;nbsp;let &lt;code&gt;ansible-builder&lt;/code&gt; produce a versioned image, and run plays against it&amp;nbsp;with &lt;code&gt;ansible-navigator&lt;/code&gt; locally and the same image in &lt;span class="caps"&gt;CI&lt;/span&gt; and on &lt;span class="caps"&gt;AAP&lt;/span&gt;. In return you get a single artifact whose version is the version of your automation, and the ability to talk about &amp;#8220;the Ansible we use&amp;#8221; as a thing that exists in a registry rather than as the loose union of whatever everyone&amp;#8217;s laptop has installed&amp;nbsp;today.&lt;/p&gt;
&lt;p&gt;For a small fleet, the upfront investment is one afternoon. For anything bigger, it&amp;#8217;s the cheapest way to stop debugging environment&amp;nbsp;drift.&lt;/p&gt;
&lt;h2 id="common-first-time-pitfalls"&gt;Common First-Time&amp;nbsp;Pitfalls&lt;/h2&gt;
&lt;p&gt;A short list of the things that bite first-time &lt;span class="caps"&gt;EE&lt;/span&gt; users, gathered from teammates&amp;#8217; debugging sessions over a few of these&amp;nbsp;rollouts:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Missing Python deps cause module import errors.&lt;/strong&gt; A collection&amp;#8217;s docs list what it needs; if it isn&amp;#8217;t&amp;nbsp;in &lt;code&gt;requirements.txt&lt;/code&gt;, the play fails with a&amp;nbsp;Python &lt;code&gt;ImportError&lt;/code&gt; that doesn&amp;#8217;t always say which collection asked for the&amp;nbsp;library.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Missing system packages cause runtime failures.&lt;/strong&gt;&amp;nbsp;A &lt;code&gt;community.crypto&lt;/code&gt; task that&amp;nbsp;calls &lt;code&gt;openssl&lt;/code&gt;, a Kerberos task that&amp;nbsp;wants &lt;code&gt;kinit&lt;/code&gt;. The error often surfaces as a generic non-zero exit code from the underlying binary. Add&amp;nbsp;to &lt;code&gt;bindep.txt&lt;/code&gt; and&amp;nbsp;rebuild.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; agent socket not visible inside the &lt;span class="caps"&gt;EE&lt;/span&gt;.&lt;/strong&gt; Pass it in&amp;nbsp;via &lt;code&gt;container-options&lt;/code&gt; (&lt;code&gt;-v $SSH_AUTH_SOCK:/ssh-agent&lt;/code&gt;, &lt;code&gt;-e SSH_AUTH_SOCK=/ssh-agent&lt;/code&gt;) or fall back to a key-file mount. Without one of those two, the first &lt;span class="caps"&gt;SSH&lt;/span&gt; connection fails with a confusing &amp;#8220;permission&amp;nbsp;denied&amp;#8221;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;DNS&lt;/span&gt; or routing surprises.&lt;/strong&gt; Hosts addressable from your laptop but not from the &lt;span class="caps"&gt;EE&lt;/span&gt;&amp;#8217;s container network. The pragmatic fix&amp;nbsp;is &lt;code&gt;--net=host&lt;/code&gt; in &lt;code&gt;container-options&lt;/code&gt;; the cleaner fix is to make the inventory addressable independently of host-local&amp;nbsp;resolution.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;CA&lt;/span&gt; certificates missing on minimal base images.&lt;/strong&gt; Anything pulling from an internal &lt;span class="caps"&gt;HTTPS&lt;/span&gt; endpoint fails at &lt;span class="caps"&gt;TLS&lt;/span&gt; validation.&amp;nbsp;Add &lt;code&gt;ca-certificates&lt;/code&gt; to &lt;code&gt;bindep.txt&lt;/code&gt; and, for a private &lt;span class="caps"&gt;CA&lt;/span&gt;, drop the cert into place&amp;nbsp;during &lt;code&gt;additional_build_steps&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;File ownership surprises on bind mounts.&lt;/strong&gt; Files created by the &lt;span class="caps"&gt;EE&lt;/span&gt; container may be owned by a different &lt;span class="caps"&gt;UID&lt;/span&gt;/&lt;span class="caps"&gt;GID&lt;/span&gt; than your host user, depending on the container runtime configuration (rootless Podman with userns mapping is the usual culprit). Usually harmless, occasionally confusing when cleaning up artifacts or trying to read a generated file from your&amp;nbsp;editor.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Each of these is a one-line fix once you know which line. The first time, they&amp;#8217;re an evening of &amp;#8220;but it works on my&amp;nbsp;laptop&amp;#8221;.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://ansible.readthedocs.io/projects/builder/en/latest/"&gt;Ansible Builder documentation&lt;/a&gt; - the canonical guide&amp;nbsp;for &lt;code&gt;execution-environment.yml&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://ansible.readthedocs.io/projects/navigator/"&gt;Ansible Navigator documentation&lt;/a&gt; - configuration reference and subcommand&amp;nbsp;walkthroughs&lt;/li&gt;
&lt;li&gt;&lt;a href="https://ansible.readthedocs.io/projects/runner/en/latest/"&gt;Ansible Runner documentation&lt;/a&gt; - the inner process supervisor that EEs&amp;nbsp;ship&lt;/li&gt;
&lt;li&gt;&lt;a href="https://quay.io/organization/ansible"&gt;Community Execution Environment images on Quay&lt;/a&gt;&amp;nbsp;- &lt;code&gt;community-ee-minimal&lt;/code&gt; and &lt;code&gt;community-ee-base&lt;/code&gt; as starting&amp;nbsp;points&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.redhat.com/en/blog/whats-in-an-execution-environment"&gt;Red Hat: What&amp;#8217;s in an Execution Environment&lt;/a&gt; - Red Hat&amp;#8217;s own&amp;nbsp;primer&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/"&gt;Ansible-native Quadlets: Mastogreet&lt;/a&gt; - the homelab Ansible role this &lt;span class="caps"&gt;EE&lt;/span&gt; is sized&amp;nbsp;for&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/speeding-up-forgejo-ci-with-a-custom-oci-image/"&gt;Forgejo &lt;span class="caps"&gt;CI&lt;/span&gt; with a Custom &lt;span class="caps"&gt;OCI&lt;/span&gt; Image&lt;/a&gt; - the same pinned-image pattern for &lt;span class="caps"&gt;CI&lt;/span&gt; runners more&amp;nbsp;broadly&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/why-i-prefer-centos-stream-over-old-centos/"&gt;CentOS Stream as a base&lt;/a&gt; - reasoning for&amp;nbsp;why &lt;code&gt;centos:stream9&lt;/code&gt; is a sensible &lt;span class="caps"&gt;EE&lt;/span&gt;&amp;nbsp;base&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="ansible"/><category term="execution-environments"/><category term="ansible-builder"/><category term="ansible-navigator"/><category term="containers"/><category term="podman"/><category term="automation"/><category term="rhel"/></entry><entry><title>Ansible-Native Quadlets: Deploying a Mastodon Greeter Bot with containers.podman</title><link href="https://hofstede.it/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/" rel="alternate"/><published>2026-04-27T00:00:00+02:00</published><updated>2026-04-27T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-04-27:/ansible-native-quadlets-deploying-a-mastodon-greeter-bot-with-containerspodman/</id><summary type="html">&lt;p&gt;Hand-writing Quadlet files works for one host. For a fleet, the containers.podman collection&amp;#8217;s native Quadlet generation lets you describe containers as Ansible state, including secrets, registry logins, and templated configuration. A small Mastodon welcome bot makes a concrete&amp;nbsp;example.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Header image" src="https://hofstede.it/images/2026-04-27-ansible-podman-quadlets-mastogreet.png" title="Ansible deploying a Quadlet container"&gt;&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-ansible-and-not-just-scp-plus-systemctl"&gt;Why Ansible (and Not Just scp Plus&amp;nbsp;systemctl)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-bot-in-one-paragraph"&gt;The Bot in One&amp;nbsp;Paragraph&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-container-briefly"&gt;The Container,&amp;nbsp;Briefly&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-ansible-layout"&gt;The Ansible&amp;nbsp;Layout&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-1-fail-fast-on-missing-configuration"&gt;Step 1: Fail Fast on Missing&amp;nbsp;Configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-2-host-side-state-directory"&gt;Step 2: Host-Side State&amp;nbsp;Directory&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-3-templated-configuration"&gt;Step 3: Templated&amp;nbsp;Configuration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-4-optional-registry-login"&gt;Step 4: Optional Registry&amp;nbsp;Login&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-5-the-podman-secret"&gt;Step 5: The Podman&amp;nbsp;Secret&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-6-generating-the-quadlet"&gt;Step 6: Generating the&amp;nbsp;Quadlet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#step-7-the-handler-chain"&gt;Step 7: The Handler&amp;nbsp;Chain&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-you-get"&gt;What You&amp;nbsp;Get&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#updates-and-secret-rotation"&gt;Updates and Secret&amp;nbsp;Rotation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#when-this-pattern-pays-off-and-when-its-overkill"&gt;When This Pattern Pays Off (and When It&amp;#8217;s&amp;nbsp;Overkill)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;p&gt;I&amp;#8217;ve covered &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Podman Quadlets&lt;/a&gt; and the broader &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman in Production story&lt;/a&gt; already. The pattern is straightforward: drop&amp;nbsp;a &lt;code&gt;.container&lt;/code&gt; file&amp;nbsp;into &lt;code&gt;/etc/containers/systemd/&lt;/code&gt;,&amp;nbsp;run &lt;code&gt;systemctl daemon-reload&lt;/code&gt;, and you have a container managed as a first-class systemd service. No daemon, no Compose runtime, no orchestrator. That works beautifully on a single host; once you have more than one, you want the deployment in version control and applied by a tool that can detect&amp;nbsp;drift.&lt;/p&gt;
&lt;p&gt;The &lt;a href="https://galaxy.ansible.com/ui/repo/published/containers/podman/"&gt;&lt;code&gt;containers.podman&lt;/code&gt;&lt;/a&gt; Ansible Collection has quietly become a very nice answer to that. As of recent releases it can generate Quadlet files directly from&amp;nbsp;a &lt;code&gt;podman_container&lt;/code&gt; task: you declare the desired container, the module writes the unit file, and the role wraps the rest of the deployment around it. Ansible&amp;#8217;s idempotency takes care of the&amp;nbsp;rest.&lt;/p&gt;
&lt;p&gt;The worked example is a small bot I run on my Mastodon instance, &lt;a href="https://git.linuxserver.pro/chris/mastogreet"&gt;&lt;code&gt;mastogreet&lt;/code&gt;&lt;/a&gt;. It watches the admin &lt;span class="caps"&gt;API&lt;/span&gt; for new local registrations and posts a welcome status mentioning each new account a configurable delay after they signed up. The bot itself is a few hundred lines of Python; what makes it interesting here is how it gets to the&amp;nbsp;host.&lt;/p&gt;
&lt;h2 id="why-ansible-and-not-just-scp-plus-systemctl"&gt;Why Ansible (and Not&amp;nbsp;Just &lt;code&gt;scp&lt;/code&gt; Plus &lt;code&gt;systemctl&lt;/code&gt;)&lt;/h2&gt;
&lt;p&gt;For a single homelab box, a&amp;nbsp;hand-edited &lt;code&gt;.container&lt;/code&gt; file is fine. I have plenty of those. For anything that needs to survive a reinstall, run on more than one host, or be reviewed by someone other than me, that approach falls over fairly&amp;nbsp;quickly:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;No idempotency.&lt;/strong&gt; Did I update the unit file? Did&amp;nbsp;I &lt;code&gt;daemon-reload&lt;/code&gt;? Did I remember&amp;nbsp;to &lt;code&gt;systemctl restart&lt;/code&gt;? A shell script is one missed step from&amp;nbsp;divergence.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Secrets get mishandled.&lt;/strong&gt; A scp&amp;#8217;d unit file with a password&amp;nbsp;in &lt;code&gt;Environment=&lt;/code&gt; is a leak waiting to happen, and the alternative&amp;nbsp;(typing &lt;code&gt;podman secret create&lt;/code&gt; by hand on each host) doesn&amp;#8217;t exist in version control&amp;nbsp;anywhere.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No drift detection.&lt;/strong&gt; When I come back six months later, did somebody hand-edit the unit on the host? With Ansible, the next playbook run rewrites&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Multi-host scaling is a copy-paste exercise.&lt;/strong&gt; Inventory groups solve that for&amp;nbsp;free.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;There are other tools in this space (cdist, salt, even cloud-init for first-boot scenarios), and they all work. I reach for Ansible because the agentless &lt;span class="caps"&gt;SSH&lt;/span&gt; model fits how I run small fleets, and because&amp;nbsp;the &lt;code&gt;containers.podman&lt;/code&gt; collection has gotten genuinely good. If you&amp;#8217;re already running Ansible elsewhere, this is a low-friction extension. If you&amp;#8217;re not, the rest of this article reads as a case study and the patterns translate to whatever configuration tool you&amp;nbsp;prefer.&lt;/p&gt;
&lt;h2 id="the-bot-in-one-paragraph"&gt;The Bot in One&amp;nbsp;Paragraph&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;mastogreet&lt;/code&gt; polls Mastodon&amp;#8217;s admin &lt;span class="caps"&gt;API&lt;/span&gt; for new local accounts, filters out bots, suspended/unconfirmed users, and anyone older than the backfill cutoff (set to &amp;#8220;now&amp;#8221; on first run so the bot doesn&amp;#8217;t greet pre-existing users), then waits a configurable delay (default 15 minutes) before posting a welcome status&amp;nbsp;that &lt;code&gt;@&lt;/code&gt;-mentions the new user. State lives in a tiny SQLite database so the bot doesn&amp;#8217;t double-greet across restarts. It picks a localized message bundle based on the new user&amp;#8217;s signup locale, falling back to a configured default. There&amp;#8217;s&amp;nbsp;a &lt;code&gt;dry_run&lt;/code&gt; mode for testing, and graceful shutdown&amp;nbsp;on &lt;code&gt;SIGTERM&lt;/code&gt; so systemd can stop it&amp;nbsp;cleanly.&lt;/p&gt;
&lt;p&gt;The Python is unremarkable. The interesting part is that everything an operator might want to change (instance &lt;span class="caps"&gt;URL&lt;/span&gt;, access token, welcome delay, message bundles, registry to pull the image from) is either an environment variable or a config-file value, which means it maps cleanly onto Ansible&amp;nbsp;variables.&lt;/p&gt;
&lt;h2 id="the-container-briefly"&gt;The Container,&amp;nbsp;Briefly&lt;/h2&gt;
&lt;p&gt;The container side is deliberately boring. The&amp;nbsp;full &lt;code&gt;Containerfile&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;python:3.12-slim&lt;/span&gt;

&lt;span class="k"&gt;ARG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;UID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10001&lt;/span&gt;
&lt;span class="k"&gt;ARG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;GID&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;10001&lt;/span&gt;

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;groupadd&lt;span class="w"&gt; &lt;/span&gt;--system&lt;span class="w"&gt; &lt;/span&gt;--gid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;GID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;mastogreet&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;useradd&lt;span class="w"&gt;  &lt;/span&gt;--system&lt;span class="w"&gt; &lt;/span&gt;--uid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;UID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--gid&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;GID&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--home-dir&lt;span class="w"&gt; &lt;/span&gt;/app&lt;span class="w"&gt; &lt;/span&gt;--no-create-home&lt;span class="w"&gt; &lt;/span&gt;mastogreet

&lt;span class="k"&gt;WORKDIR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/app&lt;/span&gt;

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;requirements.txt&lt;span class="w"&gt; &lt;/span&gt;.
&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;pip&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;--no-cache-dir&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;requirements.txt

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;mastowelcome.py&lt;span class="w"&gt; &lt;/span&gt;.

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;mastogreet&lt;span class="w"&gt; &lt;/span&gt;-g&lt;span class="w"&gt; &lt;/span&gt;mastogreet&lt;span class="w"&gt; &lt;/span&gt;/data
&lt;span class="k"&gt;VOLUME&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/data&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="k"&gt;USER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet&lt;/span&gt;

&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;PYTHONUNBUFFERED&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_CONFIG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/data/config.yaml&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;MASTOGREET_DB&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;/data/state.db
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A non-root system user with a fixed &lt;span class="caps"&gt;UID&lt;/span&gt;/&lt;span class="caps"&gt;GID&lt;/span&gt; (10001), config and state&amp;nbsp;under &lt;code&gt;/data&lt;/code&gt;,&amp;nbsp;and &lt;code&gt;PYTHONUNBUFFERED=1&lt;/code&gt; so the logs flow through to journald in real time instead of buffering. The fixed &lt;span class="caps"&gt;UID&lt;/span&gt; matters: when Ansible creates the host-side state directory, it needs to chown to that &lt;span class="caps"&gt;UID&lt;/span&gt; so the container can actually write to its bind&amp;nbsp;mount.&lt;/p&gt;
&lt;p&gt;Build is a one-time multi-arch step (my deploy host is aarch64, my workstation is&amp;nbsp;amd64):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;TAG&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;date&lt;span class="w"&gt; &lt;/span&gt;+%Y%m%d&lt;span class="k"&gt;)&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;--platform&lt;span class="w"&gt; &lt;/span&gt;linux/amd64,linux/arm64&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--manifest&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastogreet:&lt;span class="nv"&gt;$TAG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;.
podman&lt;span class="w"&gt; &lt;/span&gt;manifest&lt;span class="w"&gt; &lt;/span&gt;push&lt;span class="w"&gt; &lt;/span&gt;--all&lt;span class="w"&gt; &lt;/span&gt;git.linuxserver.pro/chris/mastogreet:&lt;span class="nv"&gt;$TAG&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After that, the host never builds anything; it pulls. Which means a registry credential becomes part of the deployment&amp;nbsp;story.&lt;/p&gt;
&lt;h2 id="the-ansible-layout"&gt;The Ansible&amp;nbsp;Layout&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;ansible/&lt;/code&gt; directory is intentionally&amp;nbsp;small:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;ansible&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;group_vars&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;span class="err"&gt;│&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="err"&gt;└──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yml&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="c1"&gt;# variables, secrets vault-encrypted&lt;/span&gt;
&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;inventory&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;example&lt;/span&gt;
&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;playbook&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yml&lt;/span&gt;&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="c1"&gt;# one-liner that applies the role&lt;/span&gt;
&lt;span class="err"&gt;└──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;roles&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;└──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;defaults&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yml&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# safe defaults&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;handlers&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yml&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="c1"&gt;# daemon-reload + restart&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;├──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;tasks&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;main&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yml&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="c1"&gt;# the actual deployment logic&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;└──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;templates&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="err"&gt;└──&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yaml&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;j2&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;# bot config rendered from vars&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The playbook is the obvious&amp;nbsp;one-liner:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Deploy mastogreet&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;hosts&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;become&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;roles&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The interesting work happens in the role. Let&amp;#8217;s walk through&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="step-1-fail-fast-on-missing-configuration"&gt;Step 1: Fail Fast on Missing&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;The first task is an assertion. It costs almost nothing and saves a lot of confusion&amp;nbsp;later:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Assert required variables are set&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.assert&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;that&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet_image is defined and mastogreet_image | length &amp;gt; 0&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet_instance_url is defined and mastogreet_instance_url | length &amp;gt; 0&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet_access_token is defined and mastogreet_access_token | length &amp;gt; 0&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet_languages is defined and (mastogreet_languages | length) &amp;gt; 0&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet_default_language in mastogreet_languages&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;fail_msg&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;&amp;gt;-&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;mastogreet_image, mastogreet_instance_url, mastogreet_access_token must&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;be set, and mastogreet_languages must include at least one bundle whose&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="no"&gt;key matches mastogreet_default_language.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A&amp;nbsp;misconfigured &lt;code&gt;mastogreet_default_language&lt;/code&gt; (one that doesn&amp;#8217;t have a corresponding bundle) is the kind of thing the bot itself would only notice when the first new user signed up, somewhere between minutes and hours after deployment. Catching it at task zero is cheap; debugging it&amp;nbsp;from &lt;code&gt;journalctl&lt;/code&gt; at 11pm is&amp;nbsp;not.&lt;/p&gt;
&lt;p&gt;This pattern (assertion task at the top of the role) shows up in nearly every production role I write. It&amp;#8217;s the role&amp;#8217;s contract with the&amp;nbsp;operator.&lt;/p&gt;
&lt;h2 id="step-2-host-side-state-directory"&gt;Step 2: Host-Side State&amp;nbsp;Directory&lt;/h2&gt;
&lt;p&gt;The bot writes&amp;nbsp;to &lt;code&gt;/data&lt;/code&gt; inside the container, which is bind-mounted&amp;nbsp;from &lt;code&gt;/opt/mastogreet/&lt;/code&gt; on the host. That directory needs to exist with the right ownership before the container&amp;nbsp;starts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Ensure state directory exists with container UID ownership&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.file&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;path&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_state_dir&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;directory&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_container_uid&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;group&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_container_gid&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;0750&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;mastogreet_container_uid&lt;/code&gt; defaults&amp;nbsp;to &lt;code&gt;10001&lt;/code&gt;, matching&amp;nbsp;the &lt;code&gt;Containerfile&lt;/code&gt;&lt;span class="quo"&gt;&amp;#8216;&lt;/span&gt;s &lt;code&gt;useradd&lt;/code&gt;.&amp;nbsp;The &lt;code&gt;0750&lt;/code&gt; mode keeps anything other than that &lt;span class="caps"&gt;UID&lt;/span&gt; and root out of the directory, which matters&amp;nbsp;because &lt;code&gt;config.yaml&lt;/code&gt; lands here&amp;nbsp;too.&lt;/p&gt;
&lt;p&gt;This is where Ansible earns its keep over a bare Quadlet drop: the file ownership has to be in lock-step with the in-container user, and &amp;#8220;I&amp;#8217;ll just remember to chown the directory&amp;#8221; doesn&amp;#8217;t survive contact with a host&amp;nbsp;rebuild.&lt;/p&gt;
&lt;h2 id="step-3-templated-configuration"&gt;Step 3: Templated&amp;nbsp;Configuration&lt;/h2&gt;
&lt;p&gt;The non-secret bot configuration lives&amp;nbsp;in &lt;code&gt;config.yaml&lt;/code&gt;, templated from the role&amp;#8217;s&amp;nbsp;variables:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Template config.yaml&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.template&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;src&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;config.yaml.j2&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;dest&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_state_dir&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}/config.yaml&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;owner&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_container_uid&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;group&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_container_gid&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;mode&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;0640&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;notify&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Restart mastogreet&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;config.yaml.j2&lt;/code&gt; template is mostly a one-to-one mapping from role variables to &lt;span class="caps"&gt;YAML&lt;/span&gt; keys&amp;nbsp;(&lt;code&gt;poll_interval_seconds&lt;/code&gt;, &lt;code&gt;welcome_delay_minutes&lt;/code&gt;, &lt;code&gt;visibility&lt;/code&gt;, the language bundles), with one nice touch: the language bundles are passed as a single nested data structure and rendered&amp;nbsp;with &lt;code&gt;to_nice_yaml&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="x"&gt;default_language: &lt;/span&gt;&lt;span class="cp"&gt;{{&lt;/span&gt; &lt;span class="nv"&gt;mastogreet_default_language&lt;/span&gt; &lt;span class="cp"&gt;}}&lt;/span&gt;
&lt;span class="x"&gt;languages:&lt;/span&gt;
&lt;span class="cp"&gt;{{&lt;/span&gt; &lt;span class="nv"&gt;mastogreet_languages&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nf"&gt;to_nice_yaml&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="nv"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="o"&gt;|&lt;/span&gt; &lt;span class="nf"&gt;indent&lt;/span&gt;&lt;span class="o"&gt;(&lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&lt;span class="o"&gt;,&lt;/span&gt; &lt;span class="kp"&gt;true&lt;/span&gt;&lt;span class="o"&gt;)&lt;/span&gt; &lt;span class="cp"&gt;}}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This means an operator can add a third or fourth language by&amp;nbsp;editing &lt;code&gt;mastogreet_languages&lt;/code&gt; in &lt;code&gt;group_vars/all.yml&lt;/code&gt;, no template changes required. The nested structure&amp;nbsp;(per-language &lt;code&gt;message_template&lt;/code&gt;, &lt;code&gt;tips&lt;/code&gt;, &lt;code&gt;recommendations&lt;/code&gt;) round-trips cleanly through Ansible&amp;#8217;s &lt;span class="caps"&gt;YAML&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;notify: Restart mastogreet&lt;/code&gt; matters: a config-only change (say, tweaking the welcome message) re-templates the file and triggers the handler, but it doesn&amp;#8217;t recreate the container. The bot picks up the new config on its next&amp;nbsp;start.&lt;/p&gt;
&lt;h2 id="step-4-optional-registry-login"&gt;Step 4: Optional Registry&amp;nbsp;Login&lt;/h2&gt;
&lt;p&gt;The bot image lives on a private Forgejo registry, which means the host needs credentials before it can&amp;nbsp;pull:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Log in to container registry&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;containers.podman.podman_login&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;registry&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_registry&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;username&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_registry_username&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;password&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_registry_password&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;when&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet_registry is defined&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet_registry_username is defined&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet_registry_password is defined&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;no_log&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Three things worth pointing&amp;nbsp;out:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Conditional execution.&lt;/strong&gt; If you&amp;#8217;re pulling from a public registry, the variables stay undefined and the task is skipped. The role works in both&amp;nbsp;cases.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;no_log: true&lt;/code&gt;&lt;/strong&gt; prevents Ansible from echoing the password into stdout or the log file on&amp;nbsp;a &lt;code&gt;-vv&lt;/code&gt; run. This should be reflexive for any task touching&amp;nbsp;credentials.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Module, not shell.&lt;/strong&gt;&amp;nbsp;Calling &lt;code&gt;containers.podman.podman_login&lt;/code&gt; rather&amp;nbsp;than &lt;code&gt;command: podman login ...&lt;/code&gt; means idempotency, structured error reporting, and no risk of credentials landing&amp;nbsp;in &lt;code&gt;bash&lt;/code&gt; history because of a&amp;nbsp;misconfigured &lt;code&gt;become&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="step-5-the-podman-secret"&gt;Step 5: The Podman&amp;nbsp;Secret&lt;/h2&gt;
&lt;p&gt;Here&amp;#8217;s where the corrected understanding of &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/#secrets-management"&gt;Podman&amp;#8217;s secret store&lt;/a&gt; matters. The&amp;nbsp;default &lt;code&gt;file&lt;/code&gt; driver doesn&amp;#8217;t encrypt at rest; it base64-encodes secrets and relies on filesystem permissions. That&amp;#8217;s a meaningful improvement over putting the access token&amp;nbsp;in &lt;code&gt;Environment=&lt;/code&gt; inside the unit file (where it would show up&amp;nbsp;in &lt;code&gt;systemctl cat&lt;/code&gt;, in the journal, and&amp;nbsp;in &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/environ&lt;/code&gt;), but it isn&amp;#8217;t a vault. For a single-host bot deployment behind tight filesystem permissions, it&amp;#8217;s the right level of protection. For larger fleets, swap&amp;nbsp;in &lt;code&gt;pass&lt;/code&gt; or &lt;code&gt;shell&lt;/code&gt; drivers and integrate with HashiCorp Vault or &lt;a href="https://openbao.org/"&gt;OpenBAO&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The role&amp;nbsp;uses &lt;code&gt;containers.podman.podman_secret&lt;/code&gt; to manage the secret&amp;nbsp;idempotently:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Create podman secret for access token&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;containers.podman.podman_secret&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_secret_name&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;present&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;data&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_access_token&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;force&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;no_log&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;notify&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Restart mastogreet&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;force: true&lt;/code&gt; means the secret is removed and recreated on every run with the current variable value. That sounds wasteful,&amp;nbsp;but &lt;code&gt;containers.podman.podman_secret&lt;/code&gt; does not expose a &amp;#8220;compare current value to desired value and update if different&amp;#8221; mode in a way that doesn&amp;#8217;t leak the existing value in diff output. Recreating unconditionally is the cheap, deterministic option: when the playbook variable changes, the secret reflects it on the next&amp;nbsp;run.&lt;/p&gt;
&lt;p&gt;The trade-off is honest:&amp;nbsp;with &lt;code&gt;force: true&lt;/code&gt; the secret task can&amp;nbsp;report &lt;code&gt;changed&lt;/code&gt; on every run, which then notifies the restart handler. I cover the practical implication of that under &amp;#8220;What You Get&amp;#8221; below; for the rest of the role, just keep in mind that the secret value flowing through Ansible variables is the source of truth, and the secret store is regenerated to&amp;nbsp;match.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;notify&lt;/code&gt; chain matters because Podman&amp;nbsp;injects &lt;code&gt;type=env&lt;/code&gt; secrets when the container process is created. Updating the secret store on its own does not mutate the environment of an already-running process.&amp;nbsp;A &lt;code&gt;systemctl restart&lt;/code&gt; of the Quadlet-generated service stops the old container and starts a new one, and that fresh container creation is when Podman reads the current secret value. So the rotation flow is: change the variable, run the playbook, the secret task rewrites the store, the handler chain restarts the service, the new container starts with the new value in its&amp;nbsp;environment.&lt;/p&gt;
&lt;h2 id="step-6-generating-the-quadlet"&gt;Step 6: Generating the&amp;nbsp;Quadlet&lt;/h2&gt;
&lt;p&gt;This is the task this whole article is really&amp;nbsp;about:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Generate Quadlet unit for mastogreet&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;containers.podman.podman_container&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_image&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;quadlet&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;quadlet_dir&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_quadlet_dir&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;quadlet_filename&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_quadlet_name&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_state_dir&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}:/data:Z&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;secrets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_secret_name&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}},type=env,target=MASTOGREET_ACCESS_TOKEN&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;MASTOGREET_INSTANCE_URL&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;{{&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;mastogreet_instance_url&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;}}&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;quadlet_options&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;AutoUpdate=registry&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;[Service]&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;Restart=on-failure&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;RestartSec=30s&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p p-Indicator"&gt;|&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;[Install]&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="no"&gt;WantedBy=multi-user.target default.target&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;notify&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Reload systemd&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Restart mastogreet&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The key parameter&amp;nbsp;is &lt;code&gt;state: quadlet&lt;/code&gt;. With it, the module writes&amp;nbsp;a &lt;code&gt;.container&lt;/code&gt; Quadlet file&amp;nbsp;to &lt;code&gt;quadlet_dir&lt;/code&gt; instead of starting a container directly. Everything Podman natively understands (image, volumes, env, secrets) maps to first-class module parameters.&amp;nbsp;The &lt;code&gt;quadlet_options&lt;/code&gt; list is the escape hatch for everything else: anything you&amp;#8217;d write into the Quadlet by hand goes here, either as a&amp;nbsp;single &lt;code&gt;Key=value&lt;/code&gt; line or as a&amp;nbsp;multi-line &lt;code&gt;[Section]&lt;/code&gt; block.&lt;/p&gt;
&lt;p&gt;A few details worth&amp;nbsp;highlighting:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;AutoUpdate=registry&lt;/code&gt;&lt;/strong&gt; opts the container&amp;nbsp;into &lt;code&gt;podman auto-update&lt;/code&gt;. With the systemd timer enabled&amp;nbsp;(&lt;code&gt;systemctl enable --now podman-auto-update.timer&lt;/code&gt;), the host checks whether the configured image reference resolves to a different remote digest than the local copy and restarts the systemd unit if it does. The registry policy requires a fully qualified image reference, which is&amp;nbsp;why &lt;code&gt;mastogreet_image&lt;/code&gt; always carries the registry hostname. This is most useful for mutable tags such&amp;nbsp;as &lt;code&gt;latest&lt;/code&gt; or a moving channel tag; for immutable date tags&amp;nbsp;like &lt;code&gt;:20260427&lt;/code&gt;, assuming you never move those tags, the digest does not change&amp;nbsp;and &lt;code&gt;AutoUpdate=registry&lt;/code&gt; is effectively a no-op. I leave the option in the role anyway so the same playbook works for both styles, and bump the variable through Ansible when I&amp;#8217;m&amp;nbsp;pinning.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No&amp;nbsp;explicit &lt;code&gt;Pull=&lt;/code&gt; policy&lt;/strong&gt; is set. With Quadlet&amp;#8217;s defaults, the image is pulled when missing and otherwise reused from local storage. Updates happen either&amp;nbsp;through &lt;code&gt;AutoUpdate=registry&lt;/code&gt; (for mutable tags) or through Ansible (for pinned tags, where&amp;nbsp;changing &lt;code&gt;mastogreet_image&lt;/code&gt; triggers a re-render of the unit and a restart; if that image reference is not already present locally, Quadlet/Podman pulls it during startup). If you wanted to force &amp;#8220;always check at restart&amp;#8221;, you could&amp;nbsp;add &lt;code&gt;Pull=newer&lt;/code&gt; via &lt;code&gt;quadlet_options&lt;/code&gt; or the&amp;nbsp;module&amp;#8217;s &lt;code&gt;pull&lt;/code&gt; parameter; I avoid it here because for an immutable date tag it adds a round-trip to the registry on every restart for no&amp;nbsp;gain.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;Restart=on-failure&lt;/code&gt; plus &lt;code&gt;RestartSec=30s&lt;/code&gt;&lt;/strong&gt; is more conservative than&amp;nbsp;the &lt;code&gt;Restart=always&lt;/code&gt; you see in the &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Forgejo Quadlet article&lt;/a&gt;. &lt;code&gt;on-failure&lt;/code&gt; means systemd won&amp;#8217;t restart the bot if it exits cleanly with code 0 (which it does&amp;nbsp;on &lt;code&gt;SIGTERM&lt;/code&gt;), only if it crashes. The 30-second back-off prevents tight crash loops if the Mastodon &lt;span class="caps"&gt;API&lt;/span&gt; is&amp;nbsp;flapping.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;WantedBy=multi-user.target default.target&lt;/code&gt;&lt;/strong&gt; ensures the service starts at boot in either target. Redundant by design; one of the two would usually be&amp;nbsp;enough.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The &lt;code&gt;Z&lt;/code&gt; flag on the volume&lt;/strong&gt;: same as the &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Forgejo example&lt;/a&gt;, this&amp;nbsp;relabels &lt;code&gt;/opt/mastogreet/&lt;/code&gt; with &lt;code&gt;container_file_t&lt;/code&gt; so SELinux lets the container read and write.&amp;nbsp;Capital &lt;code&gt;Z&lt;/code&gt; (private label) rather than&amp;nbsp;lowercase &lt;code&gt;z&lt;/code&gt; (shared) because no other container needs access to this&amp;nbsp;directory.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The&amp;nbsp;resulting &lt;code&gt;/etc/containers/systemd/mastogreet.container&lt;/code&gt; looks like a Quadlet you&amp;#8217;d write by hand. That&amp;#8217;s the point: the module isn&amp;#8217;t a layer of indirection, it&amp;#8217;s a generator. If you want to see what it produces, run the playbook&amp;nbsp;in &lt;code&gt;--check --diff&lt;/code&gt; mode and Ansible will show you the proposed file content, or&amp;nbsp;run &lt;code&gt;/usr/libexec/podman/quadlet --dryrun&lt;/code&gt; after deployment to see the systemd unit Quadlet itself derives from&amp;nbsp;your &lt;code&gt;.container&lt;/code&gt; file.&lt;/p&gt;
&lt;h2 id="step-7-the-handler-chain"&gt;Step 7: The Handler&amp;nbsp;Chain&lt;/h2&gt;
&lt;p&gt;Quadlet is a generator. Changing&amp;nbsp;the &lt;code&gt;.container&lt;/code&gt; file does not, by itself, do anything; systemd needs to be told to re-read its unit files, and the service needs to be restarted to pick up the&amp;nbsp;change:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Reload systemd&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.systemd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;daemon_reload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;

&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Restart mastogreet&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.systemd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet.service&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;restarted&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Two things that matter&amp;nbsp;here:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Order matters.&lt;/strong&gt; Ansible runs notified handlers in the order they&amp;#8217;re &lt;em&gt;defined&lt;/em&gt; in the handlers file, not the order they were notified. So even though&amp;nbsp;the &lt;code&gt;Generate Quadlet unit&lt;/code&gt; task&amp;nbsp;notifies &lt;code&gt;Reload systemd&lt;/code&gt; and &lt;code&gt;Restart mastogreet&lt;/code&gt; in that order, Ansible&amp;#8217;s guarantee is that the handlers file&amp;#8217;s order is what executes.&amp;nbsp;Defining &lt;code&gt;Reload systemd&lt;/code&gt; first means&amp;nbsp;a &lt;code&gt;daemon-reload&lt;/code&gt; always happens before any restart, regardless of which task triggered the notification. Without that ordering, you could end up restarting against a stale unit definition and wonder why your config change didn&amp;#8217;t take&amp;nbsp;effect.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Flush handlers before the start task.&lt;/strong&gt; The role ends&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Flush handlers so the unit is reloaded before we start it&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.meta&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;flush_handlers&lt;/span&gt;

&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Ensure mastogreet service is started&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ansible.builtin.systemd&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;mastogreet.service&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;started&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;By default, handlers run at the end of the play. That&amp;#8217;s fine for restarts, but&amp;nbsp;the &lt;code&gt;Ensure ... is started&lt;/code&gt; task is the first-deployment safety net (in case nothing notified the restart handler because nothing changed). For that to work on a brand-new host, the daemon-reload has to have happened first; otherwise systemd doesn&amp;#8217;t know&amp;nbsp;what &lt;code&gt;mastogreet.service&lt;/code&gt; is. &lt;code&gt;meta: flush_handlers&lt;/code&gt; forces them to run immediately. It&amp;#8217;s a small ergonomic detail that&amp;#8217;s easy to miss until your first-run deployment fails on a unit-not-found&amp;nbsp;error.&lt;/p&gt;
&lt;h2 id="what-you-get"&gt;What You&amp;nbsp;Get&lt;/h2&gt;
&lt;p&gt;A first run on a fresh host&amp;nbsp;produces:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;PLAY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;Deploy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*********************************************************&lt;/span&gt;

&lt;span class="n"&gt;TASK&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Assert&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;required&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;variables&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;are&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;***************************&lt;/span&gt;
&lt;span class="nl"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;TASK&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Ensure&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;directory&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;exists&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;container&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;UID&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;ownership&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;**&lt;/span&gt;
&lt;span class="nl"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;TASK&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Template&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;config&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;yaml&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;****************************************&lt;/span&gt;
&lt;span class="nl"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;TASK&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;container&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;registry&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;********************************&lt;/span&gt;
&lt;span class="nl"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;TASK&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Create&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;podman&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;secret&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;access&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;token&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;***********************&lt;/span&gt;
&lt;span class="nl"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;TASK&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Generate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Quadlet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;unit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;************************&lt;/span&gt;
&lt;span class="nl"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;RUNNING&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;HANDLER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Reload&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;systemd&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;***********************************&lt;/span&gt;
&lt;span class="nl"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;RUNNING&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;HANDLER&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Restart&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*******************************&lt;/span&gt;
&lt;span class="nl"&gt;changed&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;

&lt;span class="n"&gt;TASK&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Ensure&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;service&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;is&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;started&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;************************&lt;/span&gt;
&lt;span class="nl"&gt;ok&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On the host, the bot is a normal systemd&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;systemctl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;
&lt;span class="err"&gt;●&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;service&lt;/span&gt;
&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;Loaded&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;loaded&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;etc&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;containers&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;systemd&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;container&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;generated&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="n"&gt;Active&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;active&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;running&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="n"&gt;Main&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;PID&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;12342&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;conmon&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;

&lt;span class="o"&gt;$&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;journalctl&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;u&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;n&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;
&lt;span class="n"&gt;Apr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;19&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12345&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Authenticated&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;as&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="n"&gt;larvitz&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;on&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;burningboard&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;&lt;span class="n"&gt;net&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;
&lt;span class="n"&gt;Apr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;19&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12345&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Backfill&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cutoff&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2026&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;04&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="n"&gt;T19&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;11&lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;00&lt;/span&gt;
&lt;span class="n"&gt;Apr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;27&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;19&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;14&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="mi"&gt;12&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;argon&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mastogreet&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;12345&lt;/span&gt;&lt;span class="p"&gt;]:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;Scanned&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;accounts&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;since&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;cutoff&lt;/span&gt;&lt;span class="o"&gt;.&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A&amp;nbsp;second &lt;code&gt;ansible-playbook&lt;/code&gt; run on an unchanged host is mostly quiet. The exception is the secret task:&amp;nbsp;with &lt;code&gt;force: true&lt;/code&gt;, it can&amp;nbsp;report &lt;code&gt;changed&lt;/code&gt; and notify the restart handler even when the access-token variable hasn&amp;#8217;t moved, because the module rebuilds the secret unconditionally rather than peeking at the existing value to compare. That is the trade-off I accept here to keep secret rotation deterministic without exposing secret material in diffs. If you find the spurious restarts disruptive, the alternative is to&amp;nbsp;drop &lt;code&gt;force: true&lt;/code&gt; and accept that secret rotation requires either a separate &amp;#8220;rotate&amp;#8221; playbook or a&amp;nbsp;manual &lt;code&gt;podman secret rm&lt;/code&gt; before the next&amp;nbsp;run.&lt;/p&gt;
&lt;p&gt;Everything else in the role is properly idempotent: file, template, podman_login, and the Quadlet generation only&amp;nbsp;report &lt;code&gt;changed&lt;/code&gt; when the underlying state actually&amp;nbsp;drifts.&lt;/p&gt;
&lt;h2 id="updates-and-secret-rotation"&gt;Updates and Secret&amp;nbsp;Rotation&lt;/h2&gt;
&lt;p&gt;Image updates work two&amp;nbsp;ways:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Pin a tag, bump it manually.&lt;/strong&gt;&amp;nbsp;Set &lt;code&gt;mastogreet_image: registry.example.com/chris/mastogreet:20260427&lt;/code&gt; in &lt;code&gt;group_vars/all.yml&lt;/code&gt;, push a new tag, update the variable, re-run the playbook.&amp;nbsp;The &lt;code&gt;Generate Quadlet unit&lt;/code&gt; task detects the changed image, rewrites the unit, the handlers reload systemd and restart the service. Predictable and&amp;nbsp;reviewable.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Use &lt;code&gt;:latest&lt;/code&gt; plus auto-update.&lt;/strong&gt;&amp;nbsp;With &lt;code&gt;mastogreet_image: ...:latest&lt;/code&gt; and &lt;code&gt;AutoUpdate=registry&lt;/code&gt; in the Quadlet, the&amp;nbsp;host&amp;#8217;s &lt;code&gt;podman-auto-update.timer&lt;/code&gt; notices&amp;nbsp;when &lt;code&gt;latest&lt;/code&gt; resolves to a new digest and restarts the unit so a fresh container starts on that digest. No playbook run needed for a routine image refresh. Convenient, but you give up the &amp;#8220;nothing changes on the host without an Ansible run&amp;#8221; property, which I value enough that I usually pin tags in production and only&amp;nbsp;use &lt;code&gt;:latest&lt;/code&gt; for development&amp;nbsp;boxes.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Secret rotation is a one-line variable change followed by a playbook&amp;nbsp;run:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Update the access token in the vaulted group_vars file&lt;/span&gt;
ansible-vault&lt;span class="w"&gt; &lt;/span&gt;edit&lt;span class="w"&gt; &lt;/span&gt;ansible/group_vars/all.yml

&lt;span class="c1"&gt;# Re-run; the secret task is unconditional, so the new value lands,&lt;/span&gt;
&lt;span class="c1"&gt;# and the handler-driven restart recreates the container so Podman&lt;/span&gt;
&lt;span class="c1"&gt;# injects the new env value.&lt;/span&gt;
ansible-playbook&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;ansible/inventory&lt;span class="w"&gt; &lt;/span&gt;ansible/playbook.yml&lt;span class="w"&gt; &lt;/span&gt;--ask-vault-pass
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The bot is briefly stopped during the restart (a few seconds), and on resume it picks up where it left off. The&amp;nbsp;SQLite &lt;code&gt;greeted&lt;/code&gt; table tracks who has already been welcomed across&amp;nbsp;restarts.&lt;/p&gt;
&lt;h2 id="when-this-pattern-pays-off-and-when-its-overkill"&gt;When This Pattern Pays Off (and When It&amp;#8217;s&amp;nbsp;Overkill)&lt;/h2&gt;
&lt;p&gt;I&amp;#8217;d reach&amp;nbsp;for &lt;code&gt;containers.podman&lt;/code&gt; plus Quadlet generation any&amp;nbsp;time:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;More than one host runs the same&amp;nbsp;workload.&lt;/li&gt;
&lt;li&gt;The deployment configuration should be in version control (it&amp;nbsp;should).&lt;/li&gt;
&lt;li&gt;Secrets exist (they&amp;nbsp;do).&lt;/li&gt;
&lt;li&gt;A mix of Ansible-managed and hand-written Quadlets is fine; the collection generates files that look like any other&amp;nbsp;Quadlet.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I&amp;#8217;d skip it&amp;nbsp;for:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;One-off experiments on my workstation. A shell script or a&amp;nbsp;hand-edited &lt;code&gt;.container&lt;/code&gt; file is&amp;nbsp;faster.&lt;/li&gt;
&lt;li&gt;Workloads that are genuinely orchestration-shaped (multi-host scheduling, rolling updates, service mesh). Use Kubernetes for that, not Ansible plus&amp;nbsp;Quadlets.&lt;/li&gt;
&lt;li&gt;Cases where the entire team already lives in a different config-management tool. The patterns translate; the syntax&amp;nbsp;doesn&amp;#8217;t.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The middle ground (a handful of single-host Linux boxes running &lt;span class="caps"&gt;OCI&lt;/span&gt; containers) is exactly where&amp;nbsp;the &lt;code&gt;containers.podman&lt;/code&gt; collection shines. It gives you the systemd-native Quadlet model from the &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman in Production deep-dive&lt;/a&gt;, with declarative Ansible state on top, and almost none of the impedance mismatch you&amp;#8217;d expect from gluing those two&amp;nbsp;together.&lt;/p&gt;
&lt;p&gt;For me, mastogreet is now one of about a dozen services managed this way on my fleet of small &lt;span class="caps"&gt;RHEL&lt;/span&gt; hosts. The role is a hundred lines of &lt;span class="caps"&gt;YAML&lt;/span&gt;, the playbook is six lines, and the next host that needs this bot is a new entry&amp;nbsp;in &lt;code&gt;inventory&lt;/code&gt; away.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://galaxy.ansible.com/ui/repo/published/containers/podman/"&gt;&lt;code&gt;containers.podman&lt;/code&gt; Ansible Collection&lt;/a&gt; - the collection that does the heavy&amp;nbsp;lifting&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.ansible.com/ansible/latest/collections/containers/podman/podman_container_module.html"&gt;&lt;code&gt;podman_container&lt;/code&gt; module documentation&lt;/a&gt; -&amp;nbsp;including &lt;code&gt;state: quadlet&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.ansible.com/ansible/latest/collections/containers/podman/podman_secret_module.html"&gt;&lt;code&gt;podman_secret&lt;/code&gt; module&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.ansible.com/ansible/latest/collections/containers/podman/podman_login_module.html"&gt;&lt;code&gt;podman_login&lt;/code&gt; module&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html"&gt;Podman Quadlet Documentation&lt;/a&gt; - the underlying file&amp;nbsp;format&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Production-Grade Container Deployment with Podman Quadlets&lt;/a&gt; - the original Quadlet&amp;nbsp;walkthrough&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/"&gt;Podman in Production: Quadlets, Secrets, Auto-Updates, and Docker Compatibility&lt;/a&gt; - the broader Podman story this article&amp;nbsp;extends&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/keycloak-26-on-podman-with-quadlets-identity-management-the-systemd-way/"&gt;Keycloak 26 on Podman with Quadlets&lt;/a&gt; - another worked Quadlet&amp;nbsp;example&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="ansible"/><category term="podman"/><category term="quadlet"/><category term="containers"/><category term="systemd"/><category term="mastodon"/><category term="automation"/><category term="rhel"/></entry><entry><title>Replacing Lenovo’s WWAN Unlock Blob with a 100-Line Bash Script</title><link href="https://hofstede.it/replacing-lenovos-wwan-unlock-blob-with-a-100-line-bash-script/" rel="alternate"/><published>2026-04-11T00:00:00+02:00</published><updated>2026-04-11T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-04-11:/replacing-lenovos-wwan-unlock-blob-with-a-100-line-bash-script/</id><summary type="html">&lt;p&gt;My ThinkPad T14s shipped with an Intel &lt;span class="caps"&gt;XMM7560&lt;/span&gt; &lt;span class="caps"&gt;LTE&lt;/span&gt; modem that would not register on the network until Lenovo&amp;#8217;s proprietary &lt;span class="caps"&gt;FCC&lt;/span&gt;-unlock helper ran. I replaced it with a roughly 100-line bash script from a ModemManager merge request, and along the way learned that the &amp;#8220;unlock&amp;#8221; is just a small challenge-response handshake that is easy to explain in plain&amp;nbsp;shell.&lt;/p&gt;</summary><content type="html">&lt;p&gt;My Lenovo ThinkPad T14s Gen 4 (&lt;span class="caps"&gt;AMD&lt;/span&gt; Ryzen) shipped with an Intel &lt;span class="caps"&gt;XMM7560&lt;/span&gt; &lt;span class="caps"&gt;LTE&lt;/span&gt; Advanced Pro modem soldered to the mainboard. Useful little thing: real &lt;span class="caps"&gt;LTE&lt;/span&gt; on the go, no tethering dance, no MiFi puck in the bag. The catch: out of the box, the modem refuses to register on the network. ModemManager dutifully detects it, the &lt;span class="caps"&gt;SIM&lt;/span&gt; is recognised, but on my&amp;nbsp;machine &lt;code&gt;AT+CFUN=1&lt;/code&gt; would come&amp;nbsp;back &lt;code&gt;OK&lt;/code&gt; while the radio quietly stayed&amp;nbsp;dark.&lt;/p&gt;
&lt;p&gt;The reason is something called &lt;em&gt;&lt;span class="caps"&gt;FCC&lt;/span&gt; lock&lt;/em&gt;, and the official fix from Lenovo is a package of proprietary helpers and shared libraries. I replaced Lenovo&amp;#8217;s proprietary helper with a bash script that performs the same handshake in clear, auditable shell. Here is the why, the how, and the&amp;nbsp;script.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-fcc-lock-actually-is"&gt;What &lt;span class="caps"&gt;FCC&lt;/span&gt; Lock Actually&amp;nbsp;Is&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-modemmanager-fcc-unlock-mechanism"&gt;The ModemManager &lt;span class="caps"&gt;FCC&lt;/span&gt; Unlock&amp;nbsp;Mechanism&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#lenovos-solution-a-proprietary-helper-package"&gt;Lenovo&amp;#8217;s Solution: A Proprietary Helper&amp;nbsp;Package&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-alternative-a-shell-script-that-does-the-same-thing"&gt;The Alternative: A Shell Script That Does the Same&amp;nbsp;Thing&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#how-the-unlock-actually-works"&gt;How the Unlock Actually&amp;nbsp;Works&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#installing-it-yourself"&gt;Installing It&amp;nbsp;Yourself&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-disclaimer-about-jurisdiction"&gt;A Disclaimer About&amp;nbsp;Jurisdiction&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#wrapping-up"&gt;Wrapping&amp;nbsp;Up&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="what-fcc-lock-actually-is"&gt;What &lt;span class="caps"&gt;FCC&lt;/span&gt; Lock Actually&amp;nbsp;Is&lt;/h2&gt;
&lt;p&gt;If you have never run into this before, the term &amp;#8220;&lt;span class="caps"&gt;FCC&lt;/span&gt; lock&amp;#8221; sounds like &lt;span class="caps"&gt;DRM&lt;/span&gt;. It is not - or at least, not exactly. It is a regulatory compliance&amp;nbsp;mechanism.&lt;/p&gt;
&lt;p&gt;When a laptop ships with an embedded &lt;span class="caps"&gt;WWAN&lt;/span&gt; modem, the &lt;em&gt;combination&lt;/em&gt; of the laptop chassis, the antenna layout, and the modem radio is what gets certified by the regulator. In the &lt;span class="caps"&gt;US&lt;/span&gt; that regulator is the &lt;span class="caps"&gt;FCC&lt;/span&gt;, and the relevant rule is the well-known 47 &lt;span class="caps"&gt;CFR&lt;/span&gt; Part 15 / Part 22-27 framework: a radio module is certified for a specific host device, with a specific antenna gain, in a specific physical configuration. If you yank the modem out of one laptop and stick it into another, the certification no longer&amp;nbsp;applies.&lt;/p&gt;
&lt;p&gt;To enforce this, modem vendors ship the radio in a state where the &lt;span class="caps"&gt;RF&lt;/span&gt; subsystem is &lt;em&gt;administratively disabled&lt;/em&gt; until the host system performs a vendor-defined unlock handshake. The handshake is meant to be a challenge-response proof that the modem is sitting in a host the &lt;span class="caps"&gt;OEM&lt;/span&gt; has already certified together with the modem. Until it succeeds, attempts to fully enable the modem fail - upstream documentation talks about the enable step erroring out, and on my &lt;span class="caps"&gt;XMM7560&lt;/span&gt; specifically the symptom was&amp;nbsp;that &lt;code&gt;AT+CFUN=1&lt;/code&gt; came&amp;nbsp;back &lt;code&gt;OK&lt;/code&gt; while the radio never actually came up on the&amp;nbsp;network.&lt;/p&gt;
&lt;p&gt;In practice, it behaves like an &lt;span class="caps"&gt;OEM&lt;/span&gt; lock justified in regulatory terms. The same &lt;span class="caps"&gt;XMM7560&lt;/span&gt; family has shown up across multiple &lt;span class="caps"&gt;OEM&lt;/span&gt; laptops, with different vendors using different unlock material. The radio is identical across hosts; what changes is the &lt;span class="caps"&gt;OEM&lt;/span&gt;-controlled signature that gates its&amp;nbsp;use.&lt;/p&gt;
&lt;p&gt;On Windows this is invisible because the Lenovo / Dell / &lt;span class="caps"&gt;HP&lt;/span&gt; driver stack performs the handshake at boot. On Linux the responsibility falls to ModemManager, which provides the unlock mechanism and even ships some upstream helper scripts itself - but, since version 1.18.4, leaves activation to either the user or a vendor package rather than running anything by&amp;nbsp;default.&lt;/p&gt;
&lt;h2 id="the-modemmanager-fcc-unlock-mechanism"&gt;The ModemManager &lt;span class="caps"&gt;FCC&lt;/span&gt; Unlock&amp;nbsp;Mechanism&lt;/h2&gt;
&lt;p&gt;ModemManager&amp;#8217;s design here is sensible, and it is worth understanding properly because it is more nuanced than a single search path. There are &lt;em&gt;three&lt;/em&gt; directories involved, each with a distinct&amp;nbsp;role:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;${libdir}/ModemManager/fcc-unlock.d/&lt;/code&gt;&lt;/strong&gt;&amp;nbsp;(e.g. &lt;code&gt;/usr/lib64/ModemManager/fcc-unlock.d/&lt;/code&gt; on Fedora) - this is where vendor-shipped, third-party unlock tools live. If Lenovo packages their helper for your distribution, this is where it&amp;nbsp;goes.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;/usr/share/ModemManager/fcc-unlock.available.d/&lt;/code&gt;&lt;/strong&gt; - this is where ModemManager itself ships a small library of unlock scripts upstream. They are present on the system but not&amp;nbsp;enabled.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;/etc/ModemManager/fcc-unlock.d/&lt;/code&gt;&lt;/strong&gt; - this is where &lt;em&gt;you&lt;/em&gt;, the system administrator, opt in to one of the available scripts (or to your own) by symlinking it under the appropriate &lt;span class="caps"&gt;VID&lt;/span&gt;:&lt;span class="caps"&gt;PID&lt;/span&gt;&amp;nbsp;name.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;In all three locations the file is named after the modem&amp;#8217;s &lt;span class="caps"&gt;USB&lt;/span&gt; or &lt;span class="caps"&gt;PCI&lt;/span&gt; &lt;span class="caps"&gt;VID&lt;/span&gt;:&lt;span class="caps"&gt;PID&lt;/span&gt;. For my &lt;span class="caps"&gt;XMM7560&lt;/span&gt; the relevant name&amp;nbsp;is &lt;code&gt;8086:7560&lt;/code&gt; - Intel&amp;#8217;s vendor &lt;span class="caps"&gt;ID&lt;/span&gt;, the modem&amp;#8217;s product &lt;span class="caps"&gt;ID&lt;/span&gt;. ModemManager invokes the helper with the modem&amp;#8217;s DBus path and the names of its control ports as arguments, and waits for it to exit successfully. If the helper returns 0, ModemManager continues bringing the modem up; if not, the modem stays&amp;nbsp;disabled.&lt;/p&gt;
&lt;p&gt;Two things are worth noting about this layout. First, the split&amp;nbsp;between &lt;code&gt;/etc&lt;/code&gt; and the&amp;nbsp;upstream &lt;code&gt;fcc-unlock.available.d&lt;/code&gt; directory exists precisely so that activation is an explicit decision, not something that happens by default - that change landed in ModemManager 1.18.4. Second, the helper can be &lt;em&gt;any&lt;/em&gt; executable. A binary, a Python script, a shell script - ModemManager doesn&amp;#8217;t care. That is the seam I&amp;nbsp;used.&lt;/p&gt;
&lt;h2 id="lenovos-solution-a-proprietary-helper-package"&gt;Lenovo&amp;#8217;s Solution: A Proprietary Helper&amp;nbsp;Package&lt;/h2&gt;
&lt;p&gt;Lenovo&amp;#8217;s answer to this is &lt;a href="https://github.com/lenovo/lenovo-wwan-unlock"&gt;&lt;code&gt;lenovo-wwan-unlock&lt;/code&gt;&lt;/a&gt;, a GitHub-distributed package of proprietary helpers and shared libraries with setup scripts, SELinux policy and release packaging for several modules and systems. The &lt;span class="caps"&gt;README&lt;/span&gt; itself describes the project as &amp;#8220;&lt;span class="caps"&gt;FCC&lt;/span&gt; and &lt;span class="caps"&gt;DPR&lt;/span&gt; unlock for Lenovo PCs&amp;#8221;, which is worth pausing on: it is not just one unlock primitive. On supported systems Lenovo&amp;#8217;s tooling also covers &lt;span class="caps"&gt;DPR&lt;/span&gt; (Dynamic Power Reduction) / &lt;span class="caps"&gt;SAR&lt;/span&gt;-related behaviour, the mechanism by which the laptop tells the modem to back off transmit power based on proximity sensors. So a vendor helper in this space can do more than flip a single lock&amp;nbsp;bit.&lt;/p&gt;
&lt;p&gt;You install the package, it drops the right files into the right places, and it works. It is&amp;nbsp;also:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Closed-source, with no way to audit what the helpers actually do to your&amp;nbsp;modem&lt;/li&gt;
&lt;li&gt;Linked against specific runtime libraries, so on slightly newer or older distributions it can break in surprising&amp;nbsp;ways&lt;/li&gt;
&lt;li&gt;The only path the vendor offers, on a free operating system, to use a piece of hardware you already paid&amp;nbsp;for&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The whole point of running Fedora on a ThinkPad is that I get to look at, modify, and trust every component of the user-space stack. Even if the proprietary helper is benign, accepting it as the permanent answer to &amp;#8220;how do I use my modem&amp;#8221; felt like the wrong default. So I went looking for an&amp;nbsp;alternative.&lt;/p&gt;
&lt;h2 id="the-alternative-a-shell-script-that-does-the-same-thing"&gt;The Alternative: A Shell Script That Does the Same&amp;nbsp;Thing&lt;/h2&gt;
&lt;p&gt;The alternative turned out to be sitting in a &lt;a href="https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/1141#21974e3af4aa392e9cd9a19cfadb2e992ed26538"&gt;ModemManager merge request&lt;/a&gt;, posted by Floris Stoica-Marcu. It is a single bash script, about 100 lines, &lt;span class="caps"&gt;CC0&lt;/span&gt;-licensed, that performs the entire &lt;span class="caps"&gt;XMM7560&lt;/span&gt; unlock handshake using&amp;nbsp;only &lt;code&gt;bash&lt;/code&gt;, &lt;code&gt;grep&lt;/code&gt;, &lt;code&gt;awk&lt;/code&gt;, &lt;code&gt;printf&lt;/code&gt;, &lt;code&gt;xxd&lt;/code&gt; and &lt;code&gt;sha256sum&lt;/code&gt;. No compiled code, no hidden state, no vendor &lt;span class="caps"&gt;SDK&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;I want to be very clear about credit: I did not write this script. Floris did. What I did was take their script, drop it&amp;nbsp;into &lt;code&gt;/opt/unlocker/8086&lt;/code&gt;, and symlink it from ModemManager&amp;#8217;s helper&amp;nbsp;directory:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~ ❯ ls -lah /usr/lib64/ModemManager/fcc-unlock.d/
Permissions Size User Date Modified Name
drwxr-xr-x@    - root 11 Apr 12:14  .
drwxr-xr-x@    - root 11 Apr 12:14  ..
lrwxrwxrwx@    - root 11 Apr 12:14  8086:7560 -&amp;gt; /opt/unlocker/8086
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is the entire installation.&amp;nbsp;Restart &lt;code&gt;ModemManager.service&lt;/code&gt;, plug in the &lt;span class="caps"&gt;SIM&lt;/span&gt;, and watch the&amp;nbsp;journal:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Apr 11 12:15:13 neochristop ModemManager[9628]: [modem0] state changed (enabling -&amp;gt; enabled)
Apr 11 12:15:15 neochristop ModemManager[9628]: [modem0] 3GPP registration state changed (registering -&amp;gt; home)
Apr 11 12:15:15 neochristop ModemManager[9628]: [modem0] state changed (enabled -&amp;gt; registered)
Apr 11 12:15:16 neochristop ModemManager[9628]: [modem0] state changed (connecting -&amp;gt; connected)
Apr 11 12:15:16 neochristop ModemManager[9628]: [modem0] simple connect state (10/10): all done
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;And the interface itself, with a real public IPv6 from my&amp;nbsp;carrier:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;~ ❯ ifconfig wwan0
wwan0: flags=209&amp;lt;UP,POINTOPOINT,RUNNING,NOARP&amp;gt;  mtu 1500
        inet 10.127.157.102  netmask 255.255.255.0  destination 10.127.157.102
        inet6 fe80::b2ab:8cc1:6305:c055  prefixlen 64  scopeid 0x20&amp;lt;link&amp;gt;
        inet6 2a02:3037:48d:1b55:f07e:922f:8154:e65  prefixlen 64  scopeid 0x0&amp;lt;global&amp;gt;
        RX packets 199131  bytes 251330013 (239.6 MiB)
        TX packets 52690  bytes 4544146 (4.3 MiB)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That is 240 MiB of traffic over &lt;span class="caps"&gt;LTE&lt;/span&gt; through an entirely auditable user-space&amp;nbsp;path.&lt;/p&gt;
&lt;h2 id="how-the-unlock-actually-works"&gt;How the Unlock Actually&amp;nbsp;Works&lt;/h2&gt;
&lt;p&gt;Here is the part that matters for anyone who wants to understand what they are running. The Intel &lt;span class="caps"&gt;XMM7560&lt;/span&gt; unlock is a challenge/response handshake using a small set of vendor &lt;span class="caps"&gt;AT&lt;/span&gt; commands. Walking through the script in&amp;nbsp;order:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;1. Find the &lt;span class="caps"&gt;AT&lt;/span&gt; control port.&lt;/strong&gt; ModemManager passes the modem&amp;#8217;s control port names as arguments. The modem exposes both &lt;span class="caps"&gt;MBIM&lt;/span&gt; and &lt;span class="caps"&gt;AT&lt;/span&gt; ports; the &lt;span class="caps"&gt;AT&lt;/span&gt; port is what we need. The script picks it by&amp;nbsp;reading &lt;code&gt;/sys/class/wwan/&amp;lt;port&amp;gt;/type&lt;/code&gt; (Linux 5.14+) or by name pattern (older&amp;nbsp;kernels):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;PORT&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;do&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-q&lt;span class="w"&gt; &lt;/span&gt;AT&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/sys/class/wwan/&lt;/span&gt;&lt;span class="nv"&gt;$PORT&lt;/span&gt;&lt;span class="s2"&gt;/type&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2&lt;/span&gt;&amp;gt;/dev/null&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nv"&gt;AT_PORT&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nv"&gt;$PORT&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;break&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="k"&gt;done&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The chosen port is something&amp;nbsp;like &lt;code&gt;wwan0at0&lt;/code&gt;, which is then opened&amp;nbsp;as &lt;code&gt;/dev/wwan0at0&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;2. Talk &lt;span class="caps"&gt;AT&lt;/span&gt; over a file descriptor.&lt;/strong&gt; Bash can open a character device on a numbered file descriptor&amp;nbsp;with &lt;code&gt;exec&lt;/code&gt;, write a command, and read the reply. There is&amp;nbsp;no &lt;code&gt;socat&lt;/code&gt;,&amp;nbsp;no &lt;code&gt;chat&lt;/code&gt;, no helper - just bash redirection. The script wraps that in a&amp;nbsp;small &lt;code&gt;at_command&lt;/code&gt; function:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;at_command&lt;span class="o"&gt;()&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;99&lt;/span&gt;&amp;lt;&amp;gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$DEVICE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$1&lt;/span&gt;&lt;span class="s2"&gt;\r&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;99&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;answer&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;99&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;read&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;answer&lt;span class="w"&gt; &lt;/span&gt;&amp;lt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;&lt;span class="m"&gt;99&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$answer&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;99&lt;/span&gt;&amp;gt;&lt;span class="p"&gt;&amp;amp;&lt;/span&gt;-
&lt;span class="o"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The&amp;nbsp;double &lt;code&gt;read&lt;/code&gt; is there because the modem echoes the command on the first line and replies on the second. Crude, but it works on this&amp;nbsp;modem.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;3. Request a challenge.&lt;/strong&gt; The &lt;span class="caps"&gt;XMM7560&lt;/span&gt; implements a vendor&amp;nbsp;command, &lt;code&gt;AT+GTFCCLOCKGEN&lt;/code&gt;, that returns a fresh 32-bit pseudo-random nonce. This is the &amp;#8220;challenge&amp;#8221; half of the&amp;nbsp;handshake:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;RAW_CHALLENGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;at_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;at+gtfcclockgen&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;CHALLENGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$RAW_CHALLENGE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;0x[0-9a-fA-F]\+&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;awk&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{print $1}&amp;#39;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;4. Compute the response.&lt;/strong&gt; This is the only &amp;#8220;magic&amp;#8221; step, and it is not really magic at all. The response is a &lt;span class="caps"&gt;SHA&lt;/span&gt;-256 hash of the challenge concatenated with a fixed per-vendor secret, with some byte-order shuffling on each end. The Lenovo per-vendor secret in this script is the&amp;nbsp;constant &lt;code&gt;bb23be7f&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nv"&gt;VENDOR_ID_HASH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;bb23be7f&amp;quot;&lt;/span&gt;

&lt;span class="nv"&gt;HEX_CHALLENGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;%08x&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$CHALLENGE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;REVERSE_HEX_CHALLENGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;reverseWithLittleEndian&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;HEX_CHALLENGE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;COMBINED_CHALLENGE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;REVERSE_HEX_CHALLENGE&lt;/span&gt;&lt;span class="si"&gt;}${&lt;/span&gt;&lt;span class="nv"&gt;VENDOR_ID_HASH&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;RESPONSE_HASH&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;%s&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$COMBINED_CHALLENGE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;xxd&lt;span class="w"&gt; &lt;/span&gt;-r&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;sha256sum&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;cut&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39; &amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;TRUNCATED_RESPONSE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;%.8s&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;RESPONSE_HASH&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;REVERSED_RESPONSE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;reverseWithLittleEndian&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="nv"&gt;$TRUNCATED_RESPONSE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;span class="nv"&gt;RESPONSE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nb"&gt;printf&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;%d&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;0x&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;REVERSED_RESPONSE&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In plain&amp;nbsp;English:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Take the challenge as a 32-bit value, write it as 8 hex chars in little-endian byte&amp;nbsp;order.&lt;/li&gt;
&lt;li&gt;Append the Lenovo vendor secret&amp;nbsp;(&lt;code&gt;bb23be7f&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Decode the resulting hex string into raw bytes&amp;nbsp;(&lt;code&gt;xxd -r -p&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;Compute &lt;span class="caps"&gt;SHA&lt;/span&gt;-256 over those&amp;nbsp;bytes.&lt;/li&gt;
&lt;li&gt;Take the first 4 bytes of the digest, swap to little-endian, and convert to a decimal&amp;nbsp;integer.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;That decimal integer is the&amp;nbsp;response.&lt;/p&gt;
&lt;p&gt;This is the entire secret embedded in Lenovo&amp;#8217;s helper: a four-byte constant and the byte-order conventions around it. There is no hardware-bound key, no &lt;span class="caps"&gt;TPM&lt;/span&gt;-sealed secret, no per-laptop derivation - just a baked-in constant that someone reverse-engineered out of the Lenovo helper years ago. The &lt;span class="caps"&gt;XMM7560&lt;/span&gt; hardware does not actually verify &lt;em&gt;that&lt;/em&gt; host certified it; it verifies that whoever is talking to it knows the&amp;nbsp;constant.&lt;/p&gt;
&lt;p&gt;That is why I am comfortable running this script. I am not defeating any hardware-bound secret here; I am reproducing the same host-side response the proprietary helper would generate, in shell instead of machine&amp;nbsp;code.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;5. Send the response and unlock the radio.&lt;/strong&gt; Three more &lt;span class="caps"&gt;AT&lt;/span&gt;&amp;nbsp;commands:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;at_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;at+gtfcclockver=&lt;/span&gt;&lt;span class="nv"&gt;$RESPONSE&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="c1"&gt;# verify the response&lt;/span&gt;
at_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;at+gtfcclockmodeunlock&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="c1"&gt;# actually flip the lock bit&lt;/span&gt;
at_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;at+cfun=1&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;                       &lt;/span&gt;&lt;span class="c1"&gt;# bring the radio fully online&lt;/span&gt;
at_command&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;at+gtfcclockstate&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="c1"&gt;# read the current lock state&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If the&amp;nbsp;final &lt;code&gt;at+gtfcclockstate&lt;/code&gt; reports &lt;code&gt;OK&lt;/code&gt; or &lt;code&gt;1&lt;/code&gt;, the modem is unlocked and the script exits 0. ModemManager picks up from there, runs its normal enable sequence, and the modem registers on the network. If the response is wrong, the modem responds&amp;nbsp;with &lt;code&gt;ERROR&lt;/code&gt; and the script retries up to nine times with a 0.5 second&amp;nbsp;backoff.&lt;/p&gt;
&lt;p&gt;That is the entire handshake. About 100 lines of bash, no compiled dependencies, every step&amp;nbsp;inspectable.&lt;/p&gt;
&lt;h2 id="installing-it-yourself"&gt;Installing It&amp;nbsp;Yourself&lt;/h2&gt;
&lt;p&gt;If you have the same modem - check&amp;nbsp;with &lt;code&gt;lspci | grep XMM&lt;/code&gt; for &lt;code&gt;XMM7560 LTE Advanced Pro&lt;/code&gt; - the steps to swap the proprietary helper for this script are short. One caveat first about paths. On Fedora, Lenovo&amp;#8217;s package and some third-party tooling may&amp;nbsp;use &lt;code&gt;/usr/lib64/ModemManager/fcc-unlock.d/&lt;/code&gt;, but the upstream user-facing path for manually enabled scripts&amp;nbsp;is &lt;code&gt;/etc/ModemManager/fcc-unlock.d/&lt;/code&gt;. The example below uses the upstream user&amp;nbsp;path.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;/opt/unlocker
sudo&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0755&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8086&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/opt/unlocker/8086
sudo&lt;span class="w"&gt; &lt;/span&gt;ln&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;/opt/unlocker/8086&lt;span class="w"&gt; &lt;/span&gt;/etc/ModemManager/fcc-unlock.d/8086:7560
sudo&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;ModemManager.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then&amp;nbsp;watch &lt;code&gt;journalctl -u ModemManager -f&lt;/code&gt; while you bring the connection up in NetworkManager. You should see the modem walk&amp;nbsp;through &lt;code&gt;enabling -&amp;gt; enabled -&amp;gt; registered -&amp;gt; connecting -&amp;gt; connected&lt;/code&gt; and end up with an interface&amp;nbsp;like &lt;code&gt;wwan0&lt;/code&gt; carrying real&amp;nbsp;traffic.&lt;/p&gt;
&lt;p&gt;If you want to see exactly what the script does on your machine,&amp;nbsp;set &lt;code&gt;FCC_UNLOCK_DEBUG_LOG=1&lt;/code&gt; in the environment ModemManager runs in, and the script will append every step (including the &lt;span class="caps"&gt;AT&lt;/span&gt; exchanges)&amp;nbsp;to &lt;code&gt;/var/log/mm-xmm7560-fcc.log&lt;/code&gt;. That is a great way to convince yourself the script is doing nothing but the handshake described&amp;nbsp;above.&lt;/p&gt;
&lt;p&gt;If you have a &lt;em&gt;different&lt;/em&gt; modem - a Sierra Wireless &lt;span class="caps"&gt;EM7565&lt;/span&gt;, a Quectel &lt;span class="caps"&gt;EM160&lt;/span&gt;, a different Intel chip - the same overall pattern (drop a helper&amp;nbsp;into &lt;code&gt;fcc-unlock.d&lt;/code&gt; named after the &lt;span class="caps"&gt;VID&lt;/span&gt;:&lt;span class="caps"&gt;PID&lt;/span&gt;) applies, but the &lt;span class="caps"&gt;AT&lt;/span&gt; command set and the secret are different. Don&amp;#8217;t blindly use this script on hardware it wasn&amp;#8217;t written&amp;nbsp;for.&lt;/p&gt;
&lt;h2 id="a-disclaimer-about-jurisdiction"&gt;A Disclaimer About&amp;nbsp;Jurisdiction&lt;/h2&gt;
&lt;p&gt;I have to write this part carefully, because the legal picture here is genuinely fuzzy, varies by country, and I am not a&amp;nbsp;lawyer.&lt;/p&gt;
&lt;p&gt;The &lt;span class="caps"&gt;FCC&lt;/span&gt; unlock mechanism exists because regulators in many jurisdictions require radio equipment to be certified as part of a specific host device. In the &lt;span class="caps"&gt;US&lt;/span&gt; that is the &lt;span class="caps"&gt;FCC&lt;/span&gt;&amp;#8217;s modular and limited modular approval rules. In the &lt;span class="caps"&gt;EU&lt;/span&gt; the &lt;span class="caps"&gt;RED&lt;/span&gt; directive (2014/53/&lt;span class="caps"&gt;EU&lt;/span&gt;) imposes broadly similar obligations on the manufacturer. Other countries have their own equivalents. The &lt;span class="caps"&gt;OEM&lt;/span&gt; - in this case Lenovo - is the entity that holds the certification, and the unlock procedure exists so that the &lt;span class="caps"&gt;OEM&lt;/span&gt; can attest to the modem that &amp;#8220;yes, I am still the host you were certified&amp;nbsp;with.&amp;#8221;&lt;/p&gt;
&lt;p&gt;When you replace the &lt;span class="caps"&gt;OEM&lt;/span&gt;&amp;#8217;s unlock helper with your own, you are bypassing that attestation. The modem does not become illegal hardware, and the radio characteristics do not change one bit - the script flips exactly the same lock bit the proprietary helper would have - but you may, depending on where you live and how strictly the rule is read, be operating a radio in a configuration the regulator did not formally bless. In some jurisdictions that is a non-issue for an end user modifying their own equipment for personal use. In others it might not&amp;nbsp;be.&lt;/p&gt;
&lt;p&gt;To be specific about the risks I am personally aware&amp;nbsp;of:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;United States (&lt;span class="caps"&gt;FCC&lt;/span&gt;):&lt;/strong&gt; &lt;span class="caps"&gt;US&lt;/span&gt; equipment authorization rules are not especially sympathetic to modified certified equipment, even though the direct compliance burden usually falls on manufacturers and grantees rather than on end&amp;nbsp;users.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;European Union (&lt;span class="caps"&gt;RED&lt;/span&gt;):&lt;/strong&gt; the user is generally not the addressee of the directive, but a modification that departs from the certified configuration creates an obvious compliance grey area around the original &lt;span class="caps"&gt;CE&lt;/span&gt;-marked&amp;nbsp;configuration.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;United Kingdom, Switzerland, Canada, Australia:&lt;/strong&gt; broadly similar to the &lt;span class="caps"&gt;EU&lt;/span&gt;&amp;nbsp;model.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Anywhere else:&lt;/strong&gt; check your own regulator before you assume you are&amp;nbsp;fine.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;I am running this on my own laptop, on a &lt;span class="caps"&gt;SIM&lt;/span&gt; I personally own, on bands and with output power that the modem itself enforces in firmware. I consider the risk to me, in my jurisdiction, acceptable. I cannot make that call for you. &lt;strong&gt;If you are not comfortable with the regulatory ambiguity, keep using Lenovo&amp;#8217;s helper package, or do not use the modem at&amp;nbsp;all.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;I will also point out the obvious: this changes nothing about the radio&amp;#8217;s actual emissions. The modem firmware still respects the band tables, the antenna gain limits, and the transmit power caps that Intel certified. You cannot &amp;#8220;unlock more bands&amp;#8221; or &amp;#8220;transmit at higher power&amp;#8221; through this handshake. You are flipping the bit that says &amp;#8220;I am allowed to talk to the network at all,&amp;#8221; nothing&amp;nbsp;else.&lt;/p&gt;
&lt;h2 id="wrapping-up"&gt;Wrapping&amp;nbsp;Up&lt;/h2&gt;
&lt;p&gt;This was not a heroic hack. I did not reverse-engineer the modem, and I did not write the script. Floris Stoica-Marcu did the real work years ago in a ModemManager merge request. What I did was notice that Lenovo&amp;#8217;s proprietary helper was not the only option, install the auditable alternative, and write down what it actually does. That still feels worth documenting, because free software does not stop mattering just because the proprietary piece is&amp;nbsp;small.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://gitlab.freedesktop.org/mobile-broadband/ModemManager/-/merge_requests/1141#21974e3af4aa392e9cd9a19cfadb2e992ed26538"&gt;ModemManager merge request !1141 - the bash unlocker&lt;/a&gt; - the script and the discussion around&amp;nbsp;it&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/lenovo/lenovo-wwan-unlock/"&gt;lenovo-wwan-unlock&lt;/a&gt; - Lenovo&amp;#8217;s official proprietary helper package, for&amp;nbsp;comparison&lt;/li&gt;
&lt;li&gt;&lt;a href="https://modemmanager.org/docs/modemmanager/fcc-unlock/"&gt;ModemManager &lt;span class="caps"&gt;FCC&lt;/span&gt; Unlock Tools documentation&lt;/a&gt; - upstream description of the helper directory&amp;nbsp;mechanism&lt;/li&gt;
&lt;li&gt;&lt;a href="https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A32014L0053"&gt;Directive 2014/53/&lt;span class="caps"&gt;EU&lt;/span&gt; (&lt;span class="caps"&gt;RED&lt;/span&gt;)&lt;/a&gt; - &lt;span class="caps"&gt;EU&lt;/span&gt; radio equipment&amp;nbsp;directive&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="linux"/><category term="fedora"/><category term="hardware"/><category term="modemmanager"/><category term="lte"/><category term="wwan"/><category term="foss"/><category term="thinkpad"/></entry><entry><title>Podman in Production: Quadlets, Secrets, Auto-Updates, and Docker Compatibility</title><link href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/" rel="alternate"/><published>2026-04-05T00:00:00+02:00</published><updated>2026-04-05T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-04-05:/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/</id><summary type="html">&lt;p&gt;An opinionated production-ops guide to Podman on Linux servers - why I prefer it over Docker, how Quadlets replace Compose files, and practical patterns from real deployments including secrets management, auto-updates, and Docker&amp;nbsp;compatibility.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Podman Logo" src="https://hofstede.it/images/2026-04-05-podman-deep-dive.png" title="Podman Logo"&gt; &lt;/p&gt;
&lt;div class="alert alert-info" role="alert"&gt;
&lt;p&gt;&lt;strong&gt;Update (2026-04-27):&lt;/strong&gt; An earlier version of this article implied that Podman&amp;#8217;s secret store provides encrypted-at-rest secret management. It does not. With the&amp;nbsp;default &lt;code&gt;file&lt;/code&gt; driver, secrets are stored base64-encoded&amp;nbsp;in &lt;code&gt;containers/storage/secrets/filedriver/secretsdata.json&lt;/code&gt;, protected by filesystem permissions rather than encryption. The &lt;a href="#secrets-management"&gt;Secrets Management&lt;/a&gt; section has been corrected and expanded to discuss what the secret store actually buys you, and when an external system like Vault or OpenBAO is the right answer. Thanks to Tomáš Odehnal (&lt;a href="https://odehnal.online/@tomas"&gt;@tomas@odehnal.online&lt;/a&gt;) for the&amp;nbsp;heads-up.&lt;/p&gt;
&lt;/div&gt;
&lt;p&gt;I&amp;#8217;ve been running Podman in production for years now. Every container on my infrastructure - Forgejo, Traefik, PostgreSQL, Keycloak, &lt;span class="caps"&gt;CI&lt;/span&gt; runners - is managed by Podman on &lt;span class="caps"&gt;RHEL&lt;/span&gt;. Not a single Docker daemon in sight. Regular readers know that my first instinct for isolation is &lt;a href="https://hofstede.it/freebsd-foundationals-jails-from-chroot-on-steroids-to-full-virtual-networks/"&gt;FreeBSD Jails&lt;/a&gt; - but when I&amp;#8217;m on Linux and dealing with &lt;span class="caps"&gt;OCI&lt;/span&gt; containers, Podman is the tool I reach for. It follows the Unix model more closely than Docker, using smaller composable pieces instead of a single central&amp;nbsp;daemon.&lt;/p&gt;
&lt;p&gt;This isn&amp;#8217;t a &amp;#8220;getting started&amp;#8221; tutorial. This is an opinionated production-ops perspective on Linux hosts - not a universal answer for every developer workstation or platform. It&amp;#8217;s the article I wish someone had written when I was migrating away from Docker: the architectural reasons I prefer Podman, the practical patterns that make it work in production, and the real gotchas you&amp;#8217;ll hit along the way. If you&amp;#8217;ve read my earlier &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Quadlets guide&lt;/a&gt; or the &lt;a href="https://hofstede.it/keycloak-26-on-podman-with-quadlets-identity-management-the-systemd-way/"&gt;Keycloak deployment article&lt;/a&gt;, this ties those threads together into a bigger&amp;nbsp;picture.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-podman-is-better-than-docker"&gt;Why Podman Is Better Than Docker&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#no-daemon-no-single-point-of-failure"&gt;No Daemon, No Single Point of&amp;nbsp;Failure&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rootless-by-design-not-by-afterthought"&gt;Rootless by Design, Not by&amp;nbsp;Afterthought&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#selinux-is-a-feature-not-an-obstacle"&gt;SELinux Is a Feature, Not an&amp;nbsp;Obstacle&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-oci-guarantee"&gt;The &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;Guarantee&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#quadlets-containers-as-systemd-services"&gt;Quadlets: Containers as systemd Services&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-basics"&gt;The&amp;nbsp;Basics&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#why-quadlets-beat-compose"&gt;Why Quadlets Beat&amp;nbsp;Compose&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#real-world-deployment-a-complete-stack"&gt;Real-World Deployment: A Complete Stack&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#network-topology"&gt;Network&amp;nbsp;Topology&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#database-container"&gt;Database&amp;nbsp;Container&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#application-container"&gt;Application&amp;nbsp;Container&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#reverse-proxy"&gt;Reverse&amp;nbsp;Proxy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#ci-runner"&gt;&lt;span class="caps"&gt;CI&lt;/span&gt;&amp;nbsp;Runner&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#secrets-management"&gt;Secrets Management&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#what-the-default-secret-store-actually-does"&gt;What the Default Secret Store Actually&amp;nbsp;Does&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#creating-secrets"&gt;Creating&amp;nbsp;Secrets&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#using-secrets-in-quadlets"&gt;Using Secrets in&amp;nbsp;Quadlets&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#rotating-secrets"&gt;Rotating&amp;nbsp;Secrets&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#automatic-updates"&gt;Automatic Updates&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#two-update-modes"&gt;Two Update&amp;nbsp;Modes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#a-word-of-caution"&gt;A Word of&amp;nbsp;Caution&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#docker-compatibility"&gt;Docker Compatibility&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-docker-socket"&gt;The Docker&amp;nbsp;Socket&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#docker-compose-compatibility"&gt;Docker Compose&amp;nbsp;Compatibility&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-podman-cli-alias"&gt;The podman &lt;span class="caps"&gt;CLI&lt;/span&gt;&amp;nbsp;Alias&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#practical-tips"&gt;Practical Tips&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#networking-patterns"&gt;Networking&amp;nbsp;Patterns&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#volume-and-permission-patterns"&gt;Volume and Permission&amp;nbsp;Patterns&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#security-hardening"&gt;Security&amp;nbsp;Hardening&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#debugging"&gt;Debugging&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#registry-authentication"&gt;Registry&amp;nbsp;Authentication&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#automation-with-ansible"&gt;Automation with&amp;nbsp;Ansible&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#when-podman-isnt-the-right-choice"&gt;When Podman Isn&amp;#8217;t the Right&amp;nbsp;Choice&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-bigger-picture"&gt;The Bigger&amp;nbsp;Picture&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="why-podman-is-better-than-docker"&gt;Why Podman Is Better Than&amp;nbsp;Docker&lt;/h2&gt;
&lt;p&gt;I&amp;#8217;m not going to pretend this is a balanced comparison. On Linux servers, I think Podman is the cleaner architecture, and the reasons are structural rather than&amp;nbsp;cosmetic.&lt;/p&gt;
&lt;h3 id="no-daemon-no-single-point-of-failure"&gt;No Daemon, No Single Point of&amp;nbsp;Failure&lt;/h3&gt;
&lt;p&gt;Docker&amp;#8217;s architecture has a trade-off I no longer think is worth accepting on Linux hosts: a privileged central daemon.&amp;nbsp;Every &lt;code&gt;docker run&lt;/code&gt;,&amp;nbsp;every &lt;code&gt;docker build&lt;/code&gt;, every container lifecycle event goes&amp;nbsp;through &lt;code&gt;dockerd&lt;/code&gt; - a process running as root with effectively unrestricted access to your&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;Docker does&amp;nbsp;offer &lt;code&gt;--live-restore&lt;/code&gt; to keep containers running during daemon restarts, but it comes with limitations around networking and interactive sessions, and it&amp;#8217;s not the default. The fundamental coupling remains: every container&amp;#8217;s control path runs through a single privileged process. That&amp;#8217;s operational risk and attack surface that adds&amp;nbsp;up.&lt;/p&gt;
&lt;p&gt;Podman uses a fork/exec model. Each container is a direct child process of whatever started it - your shell, systemd, a &lt;span class="caps"&gt;CI&lt;/span&gt; runner. There is no central daemon. Containers are independent processes managed by the kernel, the way Unix was designed to&amp;nbsp;work.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Docker:
  User → dockerd (root daemon) → containerd → runc → container
                ↑
        centralized control, shared fate by default

Podman:
  User → conmon → runc → container
  systemd → conmon → runc → container
         (no daemon, direct process tree)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;I can update Podman on my server without touching running containers. Each container&amp;#8217;s lifecycle is&amp;nbsp;independent.&lt;/p&gt;
&lt;h3 id="rootless-by-design-not-by-afterthought"&gt;Rootless by Design, Not by&amp;nbsp;Afterthought&lt;/h3&gt;
&lt;p&gt;Docker bolted on rootless support years after the fact, and it shows. Running Docker rootless still requires a&amp;nbsp;separate &lt;code&gt;dockerd-rootless&lt;/code&gt; process, has compatibility limitations with storage drivers, and adds limitations and extra setup around things like privileged&amp;nbsp;ports.&lt;/p&gt;
&lt;p&gt;Podman was designed rootless from the start. User namespaces, subordinate &lt;span class="caps"&gt;UID&lt;/span&gt;/&lt;span class="caps"&gt;GID&lt;/span&gt; mapping, and unprivileged container execution are core architecture, not a compatibility layer. A regular user can run containers without any elevated&amp;nbsp;privileges:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# As a regular user, no sudo needed&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;-it&lt;span class="w"&gt; &lt;/span&gt;docker.io/library/alpine:latest&lt;span class="w"&gt; &lt;/span&gt;sh
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The security implications are significant. A container escape in a rootless Podman setup lands you in an unprivileged user namespace. A compromise involving Docker&amp;#8217;s privileged daemon has a much higher potential blast radius on the host than a compromise contained inside a rootless user namespace. That&amp;#8217;s not a theoretical distinction - it&amp;#8217;s a meaningful difference in your threat&amp;nbsp;model.&lt;/p&gt;
&lt;h3 id="selinux-is-a-feature-not-an-obstacle"&gt;SELinux Is a Feature, Not an&amp;nbsp;Obstacle&lt;/h3&gt;
&lt;p&gt;Docker&amp;#8217;s relationship with SELinux has historically been adversarial. Countless Docker tutorials start&amp;nbsp;with &lt;code&gt;setenforce 0&lt;/code&gt; because the daemon and SELinux don&amp;#8217;t always cooperate. Podman integrates with SELinux as a first-class security&amp;nbsp;layer.&lt;/p&gt;
&lt;p&gt;Every Podman container automatically gets an SELinux label&amp;nbsp;(&lt;code&gt;container_t&lt;/code&gt;). Volume mounts&amp;nbsp;use &lt;code&gt;:z&lt;/code&gt; and &lt;code&gt;:Z&lt;/code&gt; flags to handle context relabeling.&amp;nbsp;The &lt;code&gt;container_runtime_t&lt;/code&gt; type exists specifically for containers that need elevated access patterns (like mounting the Podman socket). None of this requires disabling SELinux or writing custom policy&amp;nbsp;modules.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re running &lt;span class="caps"&gt;RHEL&lt;/span&gt; or Fedora with SELinux enforcing (as you should be - see my &lt;a href="https://hofstede.it/selinux-a-practical-guide-for-fedora-and-rhel/"&gt;SELinux guide&lt;/a&gt;), Podman just works with it. Docker sometimes fights&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="the-oci-guarantee"&gt;The &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;Guarantee&lt;/h3&gt;
&lt;p&gt;Podman is fully &lt;span class="caps"&gt;OCI&lt;/span&gt;-compliant. For most real-world use cases, Docker images, registries, and Dockerfiles work with Podman without modification. You&amp;nbsp;can &lt;code&gt;podman pull&lt;/code&gt; from Docker&amp;nbsp;Hub, &lt;code&gt;podman build&lt;/code&gt; with a Dockerfile,&amp;nbsp;and &lt;code&gt;podman push&lt;/code&gt; to any registry. The images are identical because they follow the same &lt;span class="caps"&gt;OCI&lt;/span&gt;&amp;nbsp;specification.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# These do the same thing&lt;/span&gt;
docker&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;nginx:latest
podman&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;docker.io/library/nginx:latest

&lt;span class="c1"&gt;# Buildah (Podman&amp;#39;s build companion) reads Dockerfiles natively&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;myapp&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;Dockerfile&lt;span class="w"&gt; &lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The migration path from Docker to Podman is a find-and-replace&amp;nbsp;of &lt;code&gt;docker&lt;/code&gt; with &lt;code&gt;podman&lt;/code&gt; in most cases. Your images, your registries, your &lt;span class="caps"&gt;CI&lt;/span&gt; pipelines - they all&amp;nbsp;work.&lt;/p&gt;
&lt;h2 id="quadlets-containers-as-systemd-services"&gt;Quadlets: Containers as systemd&amp;nbsp;Services&lt;/h2&gt;
&lt;p&gt;If you&amp;#8217;re still&amp;nbsp;writing &lt;code&gt;podman run&lt;/code&gt; commands in shell scripts or using Compose files, you&amp;#8217;re missing the most compelling feature in Podman&amp;#8217;s ecosystem:&amp;nbsp;Quadlets.&lt;/p&gt;
&lt;p&gt;Quadlets are systemd unit files that describe containers, networks, volumes, and images. Drop them&amp;nbsp;in &lt;code&gt;/etc/containers/systemd/&lt;/code&gt; (system-wide)&amp;nbsp;or &lt;code&gt;~/.config/containers/systemd/&lt;/code&gt; (rootless), and systemd manages them like any other service. No daemon, no orchestrator, no &lt;span class="caps"&gt;YAML&lt;/span&gt; parser sitting between you and your&amp;nbsp;containers.&lt;/p&gt;
&lt;h3 id="the-basics"&gt;The&amp;nbsp;Basics&lt;/h3&gt;
&lt;p&gt;A Quadlet file looks like a systemd unit file with&amp;nbsp;a &lt;code&gt;[Container]&lt;/code&gt; section:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;my-app&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/library/nginx:latest&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;8080:80&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/srv/www:/usr/share/nginx/html:z&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;multi-user.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Save this&amp;nbsp;as &lt;code&gt;/etc/containers/systemd/my-app.container&lt;/code&gt;,&amp;nbsp;run &lt;code&gt;systemctl daemon-reload&lt;/code&gt;, and you have a container managed by&amp;nbsp;systemd:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;my-app.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;my-app.service
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;my-app.service&lt;span class="w"&gt; &lt;/span&gt;-f
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;No compose file. No daemon. Just systemd doing what it already does well: managing service&amp;nbsp;lifecycles.&lt;/p&gt;
&lt;h3 id="why-quadlets-beat-compose"&gt;Why Quadlets Beat&amp;nbsp;Compose&lt;/h3&gt;
&lt;p&gt;Docker Compose requires a separate binary, parses &lt;span class="caps"&gt;YAML&lt;/span&gt;, and maintains its own state about which containers belong to which project. It&amp;#8217;s a parallel service manager running alongside (or on top of) your actual service&amp;nbsp;manager.&lt;/p&gt;
&lt;p&gt;Quadlets eliminate that entire&amp;nbsp;layer:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Concern&lt;/th&gt;
&lt;th&gt;Docker Compose&lt;/th&gt;
&lt;th&gt;Podman Quadlets&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Service lifecycle&lt;/td&gt;
&lt;td&gt;&lt;code&gt;docker compose up/down&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;systemctl start/stop&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Boot startup&lt;/td&gt;
&lt;td&gt;Requires daemon + compose plugin&lt;/td&gt;
&lt;td&gt;Native&amp;nbsp;systemd &lt;code&gt;WantedBy=&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Dependencies&lt;/td&gt;
&lt;td&gt;&lt;code&gt;depends_on:&lt;/code&gt; (limited)&lt;/td&gt;
&lt;td&gt;systemd &lt;code&gt;After=&lt;/code&gt;, &lt;code&gt;Requires=&lt;/code&gt; (battle-tested)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Logs&lt;/td&gt;
&lt;td&gt;&lt;code&gt;docker compose logs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;journalctl -u&lt;/code&gt; (integrated with system logging)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Resource limits&lt;/td&gt;
&lt;td&gt;Compose &lt;span class="caps"&gt;YAML&lt;/span&gt; &lt;code&gt;deploy:&lt;/code&gt; section&lt;/td&gt;
&lt;td&gt;systemd cgroup directives (kernel-enforced)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Update mechanism&lt;/td&gt;
&lt;td&gt;Manual &lt;code&gt;docker compose pull &amp;amp;&amp;amp; up&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;code&gt;podman auto-update&lt;/code&gt; with systemd timer&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Restart policy&lt;/td&gt;
&lt;td&gt;Compose-level&lt;/td&gt;
&lt;td&gt;systemd&amp;#8217;s proven restart logic&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The dependency management alone is worth the switch. systemd&amp;#8217;s dependency graph has been solving service ordering for over a decade.&amp;nbsp;Compose&amp;#8217;s &lt;code&gt;depends_on&lt;/code&gt; is limited to basic ordering, and it can&amp;#8217;t express dependency chains involving non-container services.&amp;nbsp;systemd&amp;#8217;s &lt;code&gt;After=&lt;/code&gt; and &lt;code&gt;Requires=&lt;/code&gt; handle ordering and hard dependencies natively, and when you need actual readiness (not just &amp;#8220;the process started&amp;#8221;), Quadlets support health check&amp;nbsp;primitives:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;HealthCmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;pg_isready -U myuser&lt;/span&gt;
&lt;span class="na"&gt;HealthInterval&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;10s&lt;/span&gt;
&lt;span class="na"&gt;HealthTimeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5s&lt;/span&gt;
&lt;span class="na"&gt;HealthRetries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5&lt;/span&gt;
&lt;span class="na"&gt;HealthStartPeriod&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;30s&lt;/span&gt;
&lt;span class="na"&gt;HealthOnFailure&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;stop&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;HealthStartPeriod&lt;/code&gt; gives the application time to initialize before health checks&amp;nbsp;count. &lt;code&gt;HealthOnFailure=stop&lt;/code&gt; tells Podman to stop the container if it fails health checks - and&amp;nbsp;systemd&amp;#8217;s &lt;code&gt;Restart=always&lt;/code&gt; brings it back. This gets you much closer to real readiness and recovery behavior than simple startup ordering&amp;nbsp;alone.&lt;/p&gt;
&lt;h2 id="real-world-deployment-a-complete-stack"&gt;Real-World Deployment: A Complete&amp;nbsp;Stack&lt;/h2&gt;
&lt;p&gt;Let me show you what a real Podman deployment looks like. These are based on my production Quadlets, sanitized for opsec but architecturally&amp;nbsp;identical.&lt;/p&gt;
&lt;h3 id="network-topology"&gt;Network&amp;nbsp;Topology&lt;/h3&gt;
&lt;p&gt;First, the network isolation. Every multi-container deployment gets a dedicated backend&amp;nbsp;network:&lt;/p&gt;
&lt;p&gt;&lt;code&gt;/etc/containers/systemd/forgejo-backend.network&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;Subnet&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;172.16.0.0/24&lt;/span&gt;
&lt;span class="na"&gt;Gateway&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;172.16.0.1&lt;/span&gt;
&lt;span class="na"&gt;IPRange&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;172.16.0.0/28&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;/28&lt;/code&gt; in &lt;code&gt;IPRange&lt;/code&gt; limits &lt;span class="caps"&gt;DHCP&lt;/span&gt;-assigned addresses to 14 hosts. The database and application container need two. This prevents the network from becoming a dumping ground for unrelated containers and makes the architecture&amp;nbsp;self-documenting.&lt;/p&gt;
&lt;p&gt;The frontend network (where Traefik lives) is separate. Application containers join both; databases join only the backend. The database has no route to the internet,&amp;nbsp;ever.&lt;/p&gt;
&lt;h3 id="database-container"&gt;Database&amp;nbsp;Container&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;/etc/containers/systemd/forgejo-db.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-db&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry.redhat.io/rhel10/postgresql-16:latest&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-backend.network&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;POSTGRESQL_USER=forgejo&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;POSTGRESQL_DATABASE=forgejo&lt;/span&gt;

&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo_db_password,type=env,target=POSTGRESQL_PASSWORD&lt;/span&gt;

&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/forgejo/postgres:/var/lib/pgsql/data:z&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;default.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note what&amp;#8217;s &lt;em&gt;not&lt;/em&gt; here: no frontend network, no published ports, no Traefik labels. This container talks to exactly one network and serves exactly one purpose. The password comes from Podman&amp;#8217;s secret store (more on that below), not from the unit&amp;nbsp;file.&lt;/p&gt;
&lt;h3 id="application-container"&gt;Application&amp;nbsp;Container&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;/etc/containers/systemd/forgejo-server.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-server&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;codeberg.org/forgejo/forgejo:14&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;

&lt;span class="c1"&gt;# Internal network for database connection&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-backend.network&lt;/span&gt;
&lt;span class="c1"&gt;# External network with Traefik&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;frontend.network&lt;/span&gt;

&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;USER_UID=1000&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;USER_GID=1000&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__database__DB_TYPE=postgres&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__database__HOST=forgejo-db:5432&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__database__NAME=forgejo&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__database__USER=forgejo&lt;/span&gt;

&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo_db_password,type=env,target=FORGEJO__database__PASSWD&lt;/span&gt;

&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/forgejo/forgejo:/data:z&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/timezone:/etc/timezone:ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/localtime:/etc/localtime:ro&lt;/span&gt;

&lt;span class="c1"&gt;# Traefik routing labels&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.enable=true&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.docker.network=frontend&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.rule=Host(`git.example.com`)&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.entrypoints=https&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.service=forgejo-http&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.tls.certresolver=traefiktls&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.middlewares=secure-headers@file&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.services.forgejo-http.loadbalancer.server.port=3000&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.tcp.routers.forgejo-ssh.rule=HostSNI(`*`)&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.tcp.routers.forgejo-ssh.entrypoints=ssh&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.tcp.routers.forgejo-ssh.service=forgejo-ssh&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.tcp.services.forgejo-ssh.loadbalancer.server.port=22&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;default.target&lt;/span&gt;

&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-db.service&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.service&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The dual network attachment is the key pattern: backend for the database, frontend for Traefik. Podman&amp;#8217;s built-in &lt;span class="caps"&gt;DNS&lt;/span&gt;&amp;nbsp;resolves &lt;code&gt;forgejo-db&lt;/code&gt; on the backend network, so the app finds its database by hostname without managing &lt;span class="caps"&gt;IP&lt;/span&gt;&amp;nbsp;addresses.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;After=&lt;/code&gt; directives ensure proper startup ordering - systemd won&amp;#8217;t start Forgejo until the database and Traefik services have started. Note&amp;nbsp;that &lt;code&gt;After=&lt;/code&gt; is ordering, not readiness: it guarantees the services launch in sequence, but not that the database is actually accepting connections yet. For that, combine with the health check primitives shown&amp;nbsp;earlier.&lt;/p&gt;
&lt;h3 id="reverse-proxy"&gt;Reverse&amp;nbsp;Proxy&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;/etc/containers/systemd/traefik.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/traefik:latest&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;

&lt;span class="na"&gt;AddCapability&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;CAP_NET_BIND_SERVICE&lt;/span&gt;

&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;frontend.network&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;80:80&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;443:443&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;2222:2222&lt;/span&gt;

&lt;span class="na"&gt;NoNewPrivileges&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;SecurityLabelType&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;container_runtime_t&lt;/span&gt;

&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/localtime:/etc/localtime:ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/run/podman/podman.sock:/var/run/docker.sock:ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/traefik/traefik.yml:/etc/traefik/traefik.yml:z,ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/traefik/config.yml:/etc/traefik/config.yml:z,ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/traefik/letsencrypt:/letsencrypt:z&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/var/log/traefik:/var/log/traefik:z&lt;/span&gt;

&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.enable=true&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.rule=Host(`traefik.example.com`)&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.entrypoints=https&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.service=api@internal&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.tls=true&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.tls.certresolver=traefiktls&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.middlewares=dashboard-auth,secure-headers@file&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.middlewares.dashboard-auth.basicauth.users=admin:$$2y$$05$$...&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;default.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Notice &lt;code&gt;SecurityLabelType=container_runtime_t&lt;/code&gt;. Traefik needs to read the Podman socket to discover containers, and the&amp;nbsp;default &lt;code&gt;container_t&lt;/code&gt; SELinux type doesn&amp;#8217;t permit that.&amp;nbsp;The &lt;code&gt;container_runtime_t&lt;/code&gt; type grants the necessary access without disabling SELinux. This is precisely the kind of fine-grained security control that Docker makes difficult and Podman makes&amp;nbsp;natural.&lt;/p&gt;
&lt;h3 id="ci-runner"&gt;&lt;span class="caps"&gt;CI&lt;/span&gt;&amp;nbsp;Runner&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;/etc/containers/systemd/forgejo-runner.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Forgejo Runner&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-server.service network-online.target&lt;/span&gt;

&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-runner&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;code.forgejo.org/forgejo/runner:12&lt;/span&gt;

&lt;span class="na"&gt;User&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;root&lt;/span&gt;

&lt;span class="na"&gt;NoNewPrivileges&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;Exec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-runner daemon&lt;/span&gt;

&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-backend.network&lt;/span&gt;

&lt;span class="na"&gt;SecurityLabelType&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;container_runtime_t&lt;/span&gt;

&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/forgejo/runner:/data:z&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/forgejo/runner/config.yml:/data/config.yml:ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/run/podman/podman.sock:/var/run/docker.sock:z&lt;/span&gt;

&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;CONFIG_FILE=/data/config.yml&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;DOCKER_HOST=unix:///var/run/docker.sock&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;default.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;CI&lt;/span&gt; runner is a container that spawns other containers - it needs the Podman socket. The socket is mounted&amp;nbsp;as &lt;code&gt;/var/run/docker.sock&lt;/code&gt; because the runner expects Docker&amp;#8217;s socket path. Podman&amp;#8217;s Docker-compatible &lt;span class="caps"&gt;API&lt;/span&gt; handles the rest transparently. The runner doesn&amp;#8217;t know (or care) that it&amp;#8217;s talking to Podman instead of&amp;nbsp;Docker.&lt;/p&gt;
&lt;h2 id="secrets-management"&gt;Secrets&amp;nbsp;Management&lt;/h2&gt;
&lt;p&gt;Hardcoding passwords in unit files is a non-starter. Environment variables in Quadlet files are visible in process listings, in systemd&amp;#8217;s journal, and in the unit files themselves on disk. Podman&amp;#8217;s secret store helps with that, but it&amp;#8217;s worth being precise about what it does and doesn&amp;#8217;t give&amp;nbsp;you.&lt;/p&gt;
&lt;h3 id="what-the-default-secret-store-actually-does"&gt;What the Default Secret Store Actually&amp;nbsp;Does&lt;/h3&gt;
&lt;p&gt;It&amp;#8217;s tempting to read &amp;#8220;secret store&amp;#8221; as &amp;#8220;encrypted vault.&amp;#8221; It isn&amp;#8217;t. With Podman&amp;#8217;s&amp;nbsp;default &lt;code&gt;file&lt;/code&gt; driver, secrets are stored as base64-encoded values&amp;nbsp;in &lt;code&gt;containers/storage/secrets/filedriver/secretsdata.json&lt;/code&gt;, protected only by filesystem permissions on that file. Anyone with read access (root, or the user account running rootless Podman) can recover every secret with a&amp;nbsp;single &lt;code&gt;base64 -d&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;What you do get is still useful: secrets stay out of unit files on disk, out of systemd&amp;#8217;s journal, out&amp;nbsp;of &lt;code&gt;ps&lt;/code&gt; and &lt;code&gt;/proc/&amp;lt;pid&amp;gt;/environ&lt;/code&gt;, and out of any image layers you build. They are injected into the container at creation time and never persisted into images you might push to a registry. For single-host deployments where filesystem ACLs and host hardening are already part of the threat model, that&amp;#8217;s a reasonable trade-off, and it&amp;#8217;s the model the rest of this section&amp;nbsp;assumes.&lt;/p&gt;
&lt;p&gt;If you need actual encryption at rest with a separate trust boundary, you need an external system like &lt;a href="https://www.vaultproject.io/"&gt;HashiCorp Vault&lt;/a&gt; or &lt;a href="https://openbao.org/"&gt;OpenBAO&lt;/a&gt;. Both come with their own operational footprint: unsealing, token lifecycles, agent sidecars, audit log retention. Podman can integrate with them via&amp;nbsp;the &lt;code&gt;pass&lt;/code&gt; driver (&lt;span class="caps"&gt;GPG&lt;/span&gt;-encrypted via &lt;a href="https://www.passwordstore.org/"&gt;&lt;code&gt;pass&lt;/code&gt;&lt;/a&gt;) or&amp;nbsp;the &lt;code&gt;shell&lt;/code&gt; driver (arbitrary command, which is the usual escape hatch for Vault Agent and similar setups). Weigh the added complexity against the threat you&amp;#8217;re actually defending against; for a small fleet behind tight filesystem permissions, the default driver is often the right&amp;nbsp;answer.&lt;/p&gt;
&lt;h3 id="creating-secrets"&gt;Creating&amp;nbsp;Secrets&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Generate a strong password and store it&lt;/span&gt;
pwgen&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;forgejo_db_password&lt;span class="w"&gt; &lt;/span&gt;-

&lt;span class="c1"&gt;# Or from an existing file&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;my_api_key&lt;span class="w"&gt; &lt;/span&gt;/path/to/keyfile

&lt;span class="c1"&gt;# List stored secrets&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;ls
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;tr -d '\n'&lt;/code&gt; matters. &lt;code&gt;pwgen&lt;/code&gt; appends a trailing newline,&amp;nbsp;and &lt;code&gt;podman secret create&lt;/code&gt; stores bytes verbatim. Some applications strip trailing whitespace from passwords; others don&amp;#8217;t. A PostgreSQL init script might strip it while a &lt;span class="caps"&gt;JDBC&lt;/span&gt; driver sends it as-is, resulting in a password mismatch between two containers reading the exact same secret. Always strip the&amp;nbsp;newline.&lt;/p&gt;
&lt;h3 id="using-secrets-in-quadlets"&gt;Using Secrets in&amp;nbsp;Quadlets&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="c1"&gt;# Inject as environment variable&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo_db_password,type=env,target=FORGEJO__database__PASSWD&lt;/span&gt;

&lt;span class="c1"&gt;# Or mount as file&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;my_tls_cert,type=mount,target=/etc/ssl/certs/app.crt&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;type=env&lt;/code&gt; variant injects the secret as an environment variable at container creation time.&amp;nbsp;The &lt;code&gt;type=mount&lt;/code&gt; variant mounts it as a file inside the container, which is preferable for &lt;span class="caps"&gt;TLS&lt;/span&gt; certificates or configuration files that applications expect on&amp;nbsp;disk.&lt;/p&gt;
&lt;h3 id="rotating-secrets"&gt;Rotating&amp;nbsp;Secrets&lt;/h3&gt;
&lt;p&gt;Because &lt;code&gt;type=env&lt;/code&gt; secrets are injected at container creation time, a normal restart of the existing container won&amp;#8217;t refresh a secret that was injected when the container was created. You need to destroy the old container and create a fresh&amp;nbsp;one:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Remove the old secret&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;forgejo_db_password

&lt;span class="c1"&gt;# Create new secret&lt;/span&gt;
pwgen&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;forgejo_db_password&lt;span class="w"&gt; &lt;/span&gt;-

&lt;span class="c1"&gt;# Stop and start (not restart) to force container recreation&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt; &lt;/span&gt;forgejo-db.service&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;forgejo-db.service&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The stop/start cycle destroys the old container and creates a new one from the Quadlet definition, picking up the new secret value. Volumes persist across this cycle, so no data is&amp;nbsp;lost.&lt;/p&gt;
&lt;h2 id="automatic-updates"&gt;Automatic&amp;nbsp;Updates&lt;/h2&gt;
&lt;p&gt;One of Podman&amp;#8217;s most underrated features. Enable the systemd&amp;nbsp;timer:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;podman-auto-update.timer
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With &lt;code&gt;AutoUpdate=registry&lt;/code&gt; in your Quadlet files, Podman checks daily for new image versions. When a newer image is available at the same tag, Podman pulls it and recreates the container, preserving volumes and&amp;nbsp;configuration.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# See what would be updated without doing it&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;auto-update&lt;span class="w"&gt; &lt;/span&gt;--dry-run

&lt;span class="c1"&gt;# Force an update check now&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;auto-update
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="two-update-modes"&gt;Two Update&amp;nbsp;Modes&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;AutoUpdate=registry&lt;/code&gt;&lt;/strong&gt;: Checks the remote registry for a newer image at the same tag. Use this for images you pull from Docker Hub, Quay, or Red Hat&amp;#8217;s&amp;nbsp;registry.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;AutoUpdate=local&lt;/code&gt;&lt;/strong&gt;: Checks if the local image has been rebuilt. Use this for images you build locally&amp;nbsp;with &lt;code&gt;podman build&lt;/code&gt;.&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="a-word-of-caution"&gt;A Word of&amp;nbsp;Caution&lt;/h3&gt;
&lt;p&gt;Auto-updates are excellent for reverse proxies, utility containers, and applications where rolling forward is safe. For databases and stateful services, be more deliberate: pin to a specific version tag and bump manually after reviewing the changelog. A PostgreSQL major version bump via auto-update at 3 &lt;span class="caps"&gt;AM&lt;/span&gt; is not how you want to discover that a migration is&amp;nbsp;required.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Safe to auto-update (stateless, easily rolled back)&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/traefik:latest&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;

&lt;span class="c1"&gt;# Pin and update manually (stateful, needs migration planning)&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry.redhat.io/rhel10/postgresql-16:1-30&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Even with a pinned version&amp;nbsp;like &lt;code&gt;16:1-30&lt;/code&gt;, &lt;code&gt;AutoUpdate=registry&lt;/code&gt; still checks for image rebuilds at that exact tag (security patches, base image updates). The tag acts as a ceiling for how far the update can&amp;nbsp;go.&lt;/p&gt;
&lt;h2 id="docker-compatibility"&gt;Docker&amp;nbsp;Compatibility&lt;/h2&gt;
&lt;p&gt;Podman goes out of its way to be a drop-in replacement for Docker. This isn&amp;#8217;t just marketing - the compatibility is real, practical, and covers the two areas people worry about&amp;nbsp;most.&lt;/p&gt;
&lt;h3 id="the-docker-socket"&gt;The Docker&amp;nbsp;Socket&lt;/h3&gt;
&lt;p&gt;Podman provides a Docker-compatible &lt;span class="caps"&gt;API&lt;/span&gt; socket via a systemd-managed&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Enable the socket (root)&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;podman.socket

&lt;span class="c1"&gt;# Or rootless&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;podman.socket
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This&amp;nbsp;creates &lt;code&gt;/run/podman/podman.sock&lt;/code&gt; (root)&amp;nbsp;or &lt;code&gt;/run/user/$UID/podman/podman.sock&lt;/code&gt; (rootless), exposing a &lt;span class="caps"&gt;REST&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt; that speaks Docker&amp;#8217;s protocol. Any tool expecting a Docker socket - Traefik, Portainer, &lt;span class="caps"&gt;CI&lt;/span&gt; runners, monitoring agents - can connect to this socket and work without&amp;nbsp;modification.&lt;/p&gt;
&lt;p&gt;In Quadlet files, mount it where the application expects Docker&amp;#8217;s&amp;nbsp;socket:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/run/podman/podman.sock:/var/run/docker.sock:ro&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The application&amp;nbsp;sees &lt;code&gt;/var/run/docker.sock&lt;/code&gt;, sends Docker &lt;span class="caps"&gt;API&lt;/span&gt; calls, and Podman handles them. The application never knows the difference. This is how the Traefik and &lt;span class="caps"&gt;CI&lt;/span&gt; runner examples above work - Traefik&amp;#8217;s Docker provider discovers containers through this socket, reading their labels for routing configuration, without any Podman-specific&amp;nbsp;configuration.&lt;/p&gt;
&lt;h3 id="docker-compose-compatibility"&gt;Docker Compose&amp;nbsp;Compatibility&lt;/h3&gt;
&lt;p&gt;Podman&amp;nbsp;provides &lt;code&gt;podman compose&lt;/code&gt; as a thin wrapper around an external Compose provider, wiring that provider up to the local Podman socket. The provider can&amp;nbsp;be &lt;code&gt;docker-compose&lt;/code&gt; (the standalone Python binary), the&amp;nbsp;Go-based &lt;code&gt;docker compose&lt;/code&gt; plugin,&amp;nbsp;or &lt;code&gt;podman-compose&lt;/code&gt;. Install a supported&amp;nbsp;provider:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# podman-compose is the most common choice on Fedora/RHEL&lt;/span&gt;
dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;podman-compose
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Your&amp;nbsp;existing &lt;code&gt;docker-compose.yml&lt;/code&gt; files work&amp;nbsp;as-is:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;compose&lt;span class="w"&gt; &lt;/span&gt;up&lt;span class="w"&gt; &lt;/span&gt;-d
podman&lt;span class="w"&gt; &lt;/span&gt;compose&lt;span class="w"&gt; &lt;/span&gt;logs&lt;span class="w"&gt; &lt;/span&gt;-f
podman&lt;span class="w"&gt; &lt;/span&gt;compose&lt;span class="w"&gt; &lt;/span&gt;down
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The result is standard Podman containers - you can inspect them&amp;nbsp;with &lt;code&gt;podman ps&lt;/code&gt;, check their logs&amp;nbsp;with &lt;code&gt;podman logs&lt;/code&gt;, and manage them with all the usual Podman&amp;nbsp;tooling.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;That said, I don&amp;#8217;t recommend using Compose on Podman in production.&lt;/strong&gt; It works, and it&amp;#8217;s useful for quick local development or migrating an existing Docker Compose setup. But you&amp;#8217;re layering a Docker abstraction on top of Podman, bypassing the systemd integration that makes Podman compelling in the first place. Compose manages its own container lifecycle, its own dependency ordering, its own restart logic - duplicating what systemd already does&amp;nbsp;better.&lt;/p&gt;
&lt;p&gt;The migration path looks like&amp;nbsp;this:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Start&amp;nbsp;with &lt;code&gt;podman compose&lt;/code&gt; to verify your existing setup works on&amp;nbsp;Podman&lt;/li&gt;
&lt;li&gt;Convert each service to a Quadlet&amp;nbsp;file&lt;/li&gt;
&lt;li&gt;Delete the Compose&amp;nbsp;file&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;For step&amp;nbsp;2, &lt;code&gt;podman generate systemd --new --files --name my-container&lt;/code&gt; can produce systemd unit files from running containers, but this command is deprecated in Podman 5.x in favor of writing Quadlet files directly. In practice, manually converting Compose services to Quadlets is straightforward - the mapping from &lt;span class="caps"&gt;YAML&lt;/span&gt; keys to Quadlet directives is mostly one-to-one, and you end up with cleaner, more maintainable&amp;nbsp;configuration.&lt;/p&gt;
&lt;h3 id="the-podman-cli-alias"&gt;The &lt;code&gt;podman&lt;/code&gt; &lt;span class="caps"&gt;CLI&lt;/span&gt;&amp;nbsp;Alias&lt;/h3&gt;
&lt;p&gt;Many tools and scripts&amp;nbsp;hardcode &lt;code&gt;docker&lt;/code&gt; as the command. A simple alias handles&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# In /etc/profile.d/docker-compat.sh or your shell rc&lt;/span&gt;
&lt;span class="nb"&gt;alias&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;docker&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;podman
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;On Fedora and &lt;span class="caps"&gt;RHEL&lt;/span&gt;,&amp;nbsp;the &lt;code&gt;podman-docker&lt;/code&gt; package does this system-wide and also creates a Docker-compatible socket&amp;nbsp;symlink:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;podman-docker
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This installs&amp;nbsp;a &lt;code&gt;docker&lt;/code&gt; wrapper that&amp;nbsp;calls &lt;code&gt;podman&lt;/code&gt;,&amp;nbsp;provides &lt;code&gt;/var/run/docker.sock&lt;/code&gt; as a symlink to the Podman socket, and makes Docker-expecting tools work without configuration changes. It even suppresses the &amp;#8220;Emulate Docker &lt;span class="caps"&gt;CLI&lt;/span&gt; using podman&amp;#8221; message&amp;nbsp;that &lt;code&gt;podman&lt;/code&gt; normally&amp;nbsp;prints.&lt;/p&gt;
&lt;h2 id="practical-tips"&gt;Practical&amp;nbsp;Tips&lt;/h2&gt;
&lt;h3 id="networking-patterns"&gt;Networking&amp;nbsp;Patterns&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Dedicated backend networks for every stack.&lt;/strong&gt; Don&amp;#8217;t share a single network across unrelated services. Each application stack (app + database) gets its own isolated backend. This is defense-in-depth: if one application is compromised, the attacker can&amp;#8217;t reach another stack&amp;#8217;s&amp;nbsp;database.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Each stack gets its own backend&lt;/span&gt;
/etc/containers/systemd/forgejo-backend.network
/etc/containers/systemd/keycloak-backend.network
/etc/containers/systemd/nextcloud-backend.network
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;One shared frontend network for the reverse proxy.&lt;/strong&gt; Traefik (or whatever you use) joins this network, and each application container joins it as a second network. This is the only network with a route to the&amp;nbsp;outside.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;DNS&lt;/span&gt; resolution is automatic.&lt;/strong&gt; Containers on the same Podman network resolve each other by container name. No need to manage &lt;span class="caps"&gt;IP&lt;/span&gt; addresses&amp;nbsp;or &lt;code&gt;/etc/hosts&lt;/code&gt; entries. &lt;code&gt;forgejo-db:5432&lt;/code&gt; just works from any container&amp;nbsp;on &lt;code&gt;forgejo-backend.network&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="volume-and-permission-patterns"&gt;Volume and Permission&amp;nbsp;Patterns&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Always&amp;nbsp;use &lt;code&gt;:z&lt;/code&gt; on SELinux systems.&lt;/strong&gt;&amp;nbsp;The &lt;code&gt;:z&lt;/code&gt; flag relabels the host directory&amp;nbsp;with &lt;code&gt;container_file_t&lt;/code&gt;, allowing container access. Without it, SELinux blocks the mount and you get cryptic permission&amp;nbsp;errors.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Shared volume (multiple containers can access)&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/app/data:/data:z&lt;/span&gt;

&lt;span class="c1"&gt;# Private volume (only this container)&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/app/secrets:/secrets:Z&lt;/span&gt;

&lt;span class="c1"&gt;# Read-only system files don&amp;#39;t need relabeling&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/localtime:/etc/localtime:ro&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Set filesystem ACLs for container UIDs.&lt;/strong&gt; Containers often run as non-root UIDs internally. The &lt;span class="caps"&gt;RHEL&lt;/span&gt; PostgreSQL image uses &lt;span class="caps"&gt;UID&lt;/span&gt; 26, Forgejo uses &lt;span class="caps"&gt;UID&lt;/span&gt; 1000. Without filesystem-level access, the container can&amp;#8217;t write to its mounted&amp;nbsp;volumes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/opt/forgejo/postgres
setfacl&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;u:26:rwx&lt;span class="w"&gt; &lt;/span&gt;/opt/forgejo/postgres
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is separate from SELinux labeling&amp;nbsp;- &lt;code&gt;:z&lt;/code&gt; handles the &lt;span class="caps"&gt;MAC&lt;/span&gt;&amp;nbsp;context, &lt;code&gt;setfacl&lt;/code&gt; handles the &lt;span class="caps"&gt;DAC&lt;/span&gt; permissions. You need&amp;nbsp;both.&lt;/p&gt;
&lt;h3 id="security-hardening"&gt;Security&amp;nbsp;Hardening&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;NoNewPrivileges=true&lt;/code&gt;&lt;/strong&gt; on every container that doesn&amp;#8217;t explicitly need privilege escalation. This prevents setuid binaries inside the container from gaining elevated&amp;nbsp;privileges:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;NoNewPrivileges&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Drop capabilities by default.&lt;/strong&gt; Podman already drops most capabilities, but you can explicitly add only what&amp;#8217;s&amp;nbsp;needed:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Only grant what the container actually needs&lt;/span&gt;
&lt;span class="na"&gt;AddCapability&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;CAP_NET_BIND_SERVICE&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Use &lt;code&gt;SecurityLabelType&lt;/code&gt; sparingly.&lt;/strong&gt; Most containers run fine with the&amp;nbsp;default &lt;code&gt;container_t&lt;/code&gt; type. Only&amp;nbsp;use &lt;code&gt;container_runtime_t&lt;/code&gt; for containers that need to manage other containers (&lt;span class="caps"&gt;CI&lt;/span&gt; runners, monitoring tools that access the Podman&amp;nbsp;socket).&lt;/p&gt;
&lt;h3 id="debugging"&gt;Debugging&lt;/h3&gt;
&lt;p&gt;When things go wrong, systemd and Podman give you more integrated diagnostic tooling on Linux&amp;nbsp;hosts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Service status with recent log lines&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service

&lt;span class="c1"&gt;# Full journal with filtering&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service&lt;span class="w"&gt; &lt;/span&gt;--since&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;10 minutes ago&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Container events&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;events&lt;span class="w"&gt; &lt;/span&gt;--filter&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;container&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;forgejo-server

&lt;span class="c1"&gt;# Inspect networking&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;forgejo-server&lt;span class="w"&gt; &lt;/span&gt;--format&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;{{.NetworkSettings.Networks}}&amp;#39;&lt;/span&gt;

&lt;span class="c1"&gt;# Check systemd dependency tree&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;list-dependencies&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service

&lt;span class="c1"&gt;# See what Quadlet generated&lt;/span&gt;
/usr/libexec/podman/quadlet&lt;span class="w"&gt; &lt;/span&gt;--dryrun
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The last command is especially useful. Quadlet is a generator that produces systemd unit files from&amp;nbsp;your &lt;code&gt;.container&lt;/code&gt; files. If a Quadlet isn&amp;#8217;t behaving as&amp;nbsp;expected, &lt;code&gt;--dryrun&lt;/code&gt; shows you the actual systemd unit that was generated, so you can see exactly what systemd is working&amp;nbsp;with.&lt;/p&gt;
&lt;h3 id="registry-authentication"&gt;Registry&amp;nbsp;Authentication&lt;/h3&gt;
&lt;p&gt;For private registries (Red&amp;nbsp;Hat&amp;#8217;s &lt;code&gt;registry.redhat.io&lt;/code&gt;, corporate registries), authenticate once and Podman stores the&amp;nbsp;credentials:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Interactive login&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;login&lt;span class="w"&gt; &lt;/span&gt;registry.redhat.io

&lt;span class="c1"&gt;# Credentials are stored in&lt;/span&gt;
&lt;span class="c1"&gt;# /run/containers/0/auth.json (root)&lt;/span&gt;
&lt;span class="c1"&gt;# $XDG_RUNTIME_DIR/containers/auth.json (rootless)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For automated environments, you can provide credentials via a &lt;span class="caps"&gt;JSON&lt;/span&gt;&amp;nbsp;file:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;login&lt;span class="w"&gt; &lt;/span&gt;--authfile&lt;span class="w"&gt; &lt;/span&gt;/etc/containers/auth.json&lt;span class="w"&gt; &lt;/span&gt;registry.redhat.io
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="automation-with-ansible"&gt;Automation with&amp;nbsp;Ansible&lt;/h3&gt;
&lt;p&gt;Once you&amp;#8217;re managing more than a handful of hosts, writing Quadlet files by hand stops scaling. The &lt;a href="https://galaxy.ansible.com/ui/repo/published/containers/podman/"&gt;&lt;code&gt;containers.podman&lt;/code&gt;&lt;/a&gt; Ansible Collection can manage every aspect of Podman - containers, pods, networks, volumes, secrets, registry logins - and recent versions can generate Quadlet files directly. Instead of templating unit files yourself, you declare the desired state in a playbook and the collection handles the&amp;nbsp;rest:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;Deploy Forgejo database&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;containers.podman.podman_container&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;name&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;forgejo-db&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;image&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;registry.redhat.io/rhel10/postgresql-16:latest&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;state&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;quadlet&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;quadlet_dir&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/etc/containers/systemd&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;network&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;forgejo-backend.network&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;secrets&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;forgejo_db_password,type=env,target=POSTGRESQL_PASSWORD&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;volumes&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="p p-Indicator"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/opt/forgejo/postgres:/var/lib/pgsql/data:z&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;env&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;POSTGRESQL_USER&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;forgejo&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;POSTGRESQL_DATABASE&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;forgejo&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;state: quadlet&lt;/code&gt; parameter is the key - it tells the module to generate a Quadlet file&amp;nbsp;in &lt;code&gt;quadlet_dir&lt;/code&gt; rather than starting a container directly. This gives you Ansible&amp;#8217;s idempotency and inventory management on top of Podman&amp;#8217;s systemd integration. If you&amp;#8217;re running Podman across a fleet of &lt;span class="caps"&gt;RHEL&lt;/span&gt; hosts, this collection is the missing piece between &amp;#8220;it works on one server&amp;#8221; and &amp;#8220;it works&amp;nbsp;everywhere.&amp;#8221;&lt;/p&gt;
&lt;h2 id="when-podman-isnt-the-right-choice"&gt;When Podman Isn&amp;#8217;t the Right&amp;nbsp;Choice&lt;/h2&gt;
&lt;p&gt;I said this article was opinionated, not delusional. There are cases where Docker or Kubernetes makes more&amp;nbsp;sense:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Docker Desktop for local development on macOS/Windows.&lt;/strong&gt; Podman&amp;nbsp;has &lt;code&gt;podman machine&lt;/code&gt; for non-Linux platforms, and it works, but Docker Desktop&amp;#8217;s integration with macOS and Windows is more polished. If your developers are on Macs and just need to run containers locally, Docker Desktop is a fine choice. The production host should still be&amp;nbsp;Podman.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Multi-node orchestration.&lt;/strong&gt; If you need containers spanning multiple hosts with service discovery, rolling updates, and horizontal scaling, that&amp;#8217;s Kubernetes territory. Podman is a single-host container runtime. It does that job exceptionally well, but it doesn&amp;#8217;t pretend to be an&amp;nbsp;orchestrator.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Ecosystem lock-in.&lt;/strong&gt; Some &lt;span class="caps"&gt;CI&lt;/span&gt;/&lt;span class="caps"&gt;CD&lt;/span&gt; platforms, development tools, and monitoring solutions have deep Docker-specific integrations that go beyond the &lt;span class="caps"&gt;API&lt;/span&gt; compatibility layer. If your entire toolchain assumes Docker and the compatibility layer doesn&amp;#8217;t fully cover your use case, forcing Podman may create more friction than it&amp;nbsp;solves.&lt;/p&gt;
&lt;p&gt;For everything else - single-host deployments, homelab infrastructure, edge computing, production services that don&amp;#8217;t need orchestration - Podman with Quadlets is the better&amp;nbsp;tool.&lt;/p&gt;
&lt;h2 id="the-bigger-picture"&gt;The Bigger&amp;nbsp;Picture&lt;/h2&gt;
&lt;p&gt;Containers are processes. systemd manages processes. Quadlets connect the two. Your containers start at boot, restart on failure, log to journald, respect cgroup limits, and integrate with SELinux - all through mechanisms that have been battle-tested in production Linux systems for over a&amp;nbsp;decade.&lt;/p&gt;
&lt;p&gt;Docker popularized containers and deserves credit for that. But the Linux ecosystem has matured - user namespaces, cgroups v2, and systemd&amp;#8217;s service management have made the central daemon architecture unnecessary for most Linux server workloads. Podman builds on that maturity rather than working around&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re starting new container infrastructure on Linux, start with Podman. If you&amp;#8217;re running Docker in production, the compatibility layer makes migration surprisingly low-friction, and the architectural improvements make it&amp;nbsp;worthwhile.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.podman.io/en/latest/"&gt;Podman&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html"&gt;Podman Quadlet&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.podman.io/en/latest/markdown/podman-auto-update.1.html"&gt;Podman Auto-Update&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.podman.io/en/latest/markdown/podman-secret.1.html"&gt;Podman Secrets&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.redhat.com/en/blog/quadlet-podman"&gt;Red Hat: From Docker Compose to Podman&amp;nbsp;Quadlets&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://galaxy.ansible.com/ui/repo/published/containers/podman/"&gt;containers.podman Ansible Collection&lt;/a&gt; - manage Podman and generate Quadlets via&amp;nbsp;Ansible&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Production-Grade Container Deployment with Podman Quadlets&lt;/a&gt; - my earlier Quadlet&amp;nbsp;guide&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/keycloak-26-on-podman-with-quadlets-identity-management-the-systemd-way/"&gt;Keycloak 26 on Podman with Quadlets&lt;/a&gt; - practical Quadlet deployment&amp;nbsp;example&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/selinux-a-practical-guide-for-fedora-and-rhel/"&gt;SELinux: A Practical Guide&lt;/a&gt; - understanding SELinux in container&amp;nbsp;contexts&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="podman"/><category term="containers"/><category term="quadlet"/><category term="systemd"/><category term="linux"/><category term="rhel"/><category term="docker"/><category term="secrets"/><category term="traefik"/></entry><entry><title>Keycloak 26 on Podman with Quadlets: Identity Management the systemd Way</title><link href="https://hofstede.it/keycloak-26-on-podman-with-quadlets-identity-management-the-systemd-way/" rel="alternate"/><published>2026-03-31T00:00:00+02:00</published><updated>2026-03-31T00:00:00+02:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-03-31:/keycloak-26-on-podman-with-quadlets-identity-management-the-systemd-way/</id><summary type="html">&lt;p&gt;Deploying Keycloak 26 as an identity provider using Podman Quadlets with network segmentation, secret management, and systemd&amp;nbsp;integration.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Logo" src="https://hofstede.it/images/2026-03-31-keycloak-26-podman-quadlets.png" title="Keycloak on Podman: Header image"&gt;&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Update, 2026-04-27: an earlier version of this article described Podman&amp;#8217;s secret store as keeping credentials encrypted. With the&amp;nbsp;default &lt;code&gt;file&lt;/code&gt; driver it does not: secrets are base64-encoded with filesystem permissions as the only protection. The wording in Step 1 has been corrected. See &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/#secrets-management"&gt;Podman in Production: Secrets Management&lt;/a&gt; for what the default driver actually buys&amp;nbsp;you.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Running your own identity provider is one of those things that sounds straightforward until you&amp;#8217;re three hours into debugging &lt;span class="caps"&gt;OIDC&lt;/span&gt; token flows at 2 &lt;span class="caps"&gt;AM&lt;/span&gt;. Keycloak has become the de facto open-source solution for identity and access management, but deploying it properly - with a real database backend, network isolation, and no credentials in plaintext - still takes some deliberate&amp;nbsp;architecture.&lt;/p&gt;
&lt;p&gt;This guide walks through deploying the &lt;a href="https://access.redhat.com/products/red-hat-build-of-keycloak"&gt;Red Hat Build of Keycloak&lt;/a&gt; (&lt;span class="caps"&gt;RHBK&lt;/span&gt;) 26 on Podman using Quadlets: systemd-native unit files that give you declarative container management without a daemon or a compose file. If you&amp;#8217;ve read my earlier &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Podman Quadlets guide&lt;/a&gt;, this follows the same architectural pattern - isolated backend network for the database, dual-attached application container, secrets managed through Podman&amp;#8217;s secret&amp;nbsp;store.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;A note on support&lt;/strong&gt;: Red Hat&amp;#8217;s official documentation for the &lt;span class="caps"&gt;RHBK&lt;/span&gt; container image targets OpenShift as the supported deployment platform. Running &lt;span class="caps"&gt;RHBK&lt;/span&gt; on plain Podman is technically sound and works well, but it is not a Red Hat-supported configuration. If you need a fully supported deployment path, OpenShift is where Red Hat stands behind it. This guide is for those of us who want the &lt;span class="caps"&gt;RHBK&lt;/span&gt; image&amp;#8217;s quality, security errata, and build lifecycle on a single-host Podman setup - and are comfortable with that&amp;nbsp;trade-off.&lt;/p&gt;
&lt;h2 id="architecture-overview"&gt;Architecture&amp;nbsp;Overview&lt;/h2&gt;
&lt;p&gt;The deployment consists of three&amp;nbsp;components:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;PostgreSQL 16&lt;/strong&gt; - Keycloak&amp;#8217;s database backend, isolated on a private&amp;nbsp;network&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Keycloak 26&lt;/strong&gt; - The identity provider, connected to both backend and frontend&amp;nbsp;networks&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A dedicated backend network&lt;/strong&gt; - Ensuring the database is never reachable from the&amp;nbsp;outside&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                    Internet
                       |
               Reverse Proxy (443)
                       |
          +------------+------------+
          |     frontend network    |
          +------------+------------+
                       |
                Keycloak Container
                   (Port 8080)
                       |
          +------------+------------+
          | keycloak-backend.network|
          |   (172.16.0.0/24)       |
          +------------+------------+
                       |
              PostgreSQL Container
                  (Port 5432)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Keycloak sits on both networks: it talks to PostgreSQL over the isolated backend, and your reverse proxy reaches it on the frontend. The database has no route to the outside&amp;nbsp;world.&lt;/p&gt;
&lt;p&gt;This guide assumes you already have a reverse proxy (Traefik, Caddy, nginx) handling &lt;span class="caps"&gt;TLS&lt;/span&gt; termination on your frontend network. If you need that piece, my &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;earlier Quadlet article&lt;/a&gt; covers Traefik in&amp;nbsp;detail.&lt;/p&gt;
&lt;h2 id="prerequisites"&gt;Prerequisites&lt;/h2&gt;
&lt;p&gt;You&amp;#8217;ll&amp;nbsp;need:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; 10, Fedora 43+, or any system with Podman 5.x+&lt;/strong&gt; and Quadlet&amp;nbsp;support&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;An active Red Hat subscription&lt;/strong&gt; for pulling&amp;nbsp;from &lt;code&gt;registry.redhat.io&lt;/code&gt; (this guide uses the Red Hat Build of&amp;nbsp;Keycloak)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A reverse proxy&lt;/strong&gt; already handling &lt;span class="caps"&gt;TLS&lt;/span&gt; on your frontend&amp;nbsp;network&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Root access&lt;/strong&gt; for system-wide Quadlet deployment (or adapt for rootless&amp;nbsp;with &lt;code&gt;systemctl --user&lt;/code&gt;)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;All Quadlet files go&amp;nbsp;into &lt;code&gt;/etc/containers/systemd/&lt;/code&gt; for system-wide&amp;nbsp;deployments.&lt;/p&gt;
&lt;h2 id="step-1-create-the-secrets"&gt;Step 1: Create the&amp;nbsp;Secrets&lt;/h2&gt;
&lt;p&gt;Never put passwords in unit files. Podman&amp;#8217;s secret store keeps credentials out of unit files on disk, out&amp;nbsp;of &lt;code&gt;ps&lt;/code&gt; and the systemd journal, and out of any image layers you might&amp;nbsp;build:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Generate and store the database password&lt;/span&gt;
pwgen&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;keycloak_db_password&lt;span class="w"&gt; &lt;/span&gt;-

&lt;span class="c1"&gt;# Generate and store the Keycloak admin password&lt;/span&gt;
pwgen&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;keycloak_admin_password&lt;span class="w"&gt; &lt;/span&gt;-
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;tr -d '\n'&lt;/code&gt; is&amp;nbsp;important. &lt;code&gt;pwgen&lt;/code&gt; appends a trailing newline,&amp;nbsp;and &lt;code&gt;podman secret create&lt;/code&gt; stores the raw bytes verbatim. PostgreSQL&amp;#8217;s init script strips the newline when setting the password, but Keycloak&amp;#8217;s &lt;span class="caps"&gt;JDBC&lt;/span&gt; driver sends it as-is - resulting in a password mismatch that produces a confusing &amp;#8220;authentication failed&amp;#8221; error despite both containers reading the exact same&amp;nbsp;secret.&lt;/p&gt;
&lt;p&gt;Write down the admin password somewhere safe - you&amp;#8217;ll need it for first login. After initial setup, you can create additional admin accounts through the Keycloak &lt;span class="caps"&gt;UI&lt;/span&gt; and remove the bootstrap&amp;nbsp;admin.&lt;/p&gt;
&lt;h2 id="step-2-backend-network"&gt;Step 2: Backend&amp;nbsp;Network&lt;/h2&gt;
&lt;p&gt;Create &lt;code&gt;/etc/containers/systemd/keycloak-backend.network&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;NetworkName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak-backend&lt;/span&gt;
&lt;span class="na"&gt;Subnet&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;172.16.0.0/24&lt;/span&gt;
&lt;span class="na"&gt;Gateway&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;172.16.0.1&lt;/span&gt;
&lt;span class="na"&gt;IPRange&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;172.16.0.0/28&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A dedicated subnet with an explicit &lt;span class="caps"&gt;IP&lt;/span&gt; range.&amp;nbsp;The &lt;code&gt;/28&lt;/code&gt; range&amp;nbsp;in &lt;code&gt;IPRange&lt;/code&gt; limits &lt;span class="caps"&gt;DHCP&lt;/span&gt; assignments to 14 addresses - more than enough for a database and application container, and it prevents the network from being used as a dumping ground for unrelated&amp;nbsp;containers.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Why not just use the default bridge?&lt;/strong&gt; Network segmentation is the point. The database should only be reachable by containers that explicitly join this network. Defense-in-depth starts at the network&amp;nbsp;layer.&lt;/p&gt;
&lt;h2 id="step-3-database-container"&gt;Step 3: Database&amp;nbsp;Container&lt;/h2&gt;
&lt;p&gt;Create &lt;code&gt;/etc/containers/systemd/keycloak-db.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Keycloak PostgreSQL database&lt;/span&gt;

&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak-db&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry.redhat.io/rhel10/postgresql-16:latest&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;

&lt;span class="c1"&gt;# Isolated on backend network only - no frontend access&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak-backend.network&lt;/span&gt;

&lt;span class="c1"&gt;# PostgreSQL configuration&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;POSTGRESQL_USER=keycloak&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;POSTGRESQL_DATABASE=keycloak&lt;/span&gt;

&lt;span class="c1"&gt;# Password injected at runtime from Podman secret store&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak_db_password,type=env,target=POSTGRESQL_PASSWORD&lt;/span&gt;

&lt;span class="c1"&gt;# Persistent storage with SELinux relabeling&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/keycloak/postgres:/var/lib/pgsql/data:z&lt;/span&gt;

&lt;span class="c1"&gt;# Health check - verify PostgreSQL is accepting connections&lt;/span&gt;
&lt;span class="na"&gt;HealthCmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/libexec/check-container&lt;/span&gt;
&lt;span class="na"&gt;HealthInterval&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;30s&lt;/span&gt;
&lt;span class="na"&gt;HealthTimeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5s&lt;/span&gt;
&lt;span class="na"&gt;HealthRetries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;3&lt;/span&gt;
&lt;span class="na"&gt;HealthStartPeriod&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;30s&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;multi-user.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Key&amp;nbsp;points:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;registry.redhat.io/rhel10/postgresql-16&lt;/code&gt;&lt;/strong&gt;: The &lt;span class="caps"&gt;RHEL&lt;/span&gt;-based PostgreSQL image follows Red Hat&amp;#8217;s support lifecycle and receives security errata. If you don&amp;#8217;t have a Red Hat&amp;nbsp;subscription, &lt;code&gt;docker.io/postgres:16-alpine&lt;/code&gt; is a solid alternative - just adjust the environment variables&amp;nbsp;(&lt;code&gt;POSTGRES_USER&lt;/code&gt;, &lt;code&gt;POSTGRES_DB&lt;/code&gt;, &lt;code&gt;POSTGRES_PASSWORD&lt;/code&gt;), the internal data path&amp;nbsp;(&lt;code&gt;/var/lib/postgresql/data&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;/var/lib/pgsql/data&lt;/code&gt;), and the health check&amp;nbsp;(&lt;code&gt;pg_isready -U keycloak&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;/usr/libexec/check-container&lt;/code&gt;).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Single network attachment&lt;/strong&gt;: The database lives exclusively on the backend network. It has no route to the frontend and cannot be reached from outside the backend&amp;nbsp;subnet.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;:z&lt;/code&gt; volume flag&lt;/strong&gt;: Tells Podman to relabel the SELinux context for shared container access. Don&amp;#8217;t skip this on SELinux-enforcing&amp;nbsp;systems.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Health check&lt;/strong&gt;: The &lt;span class="caps"&gt;RHEL&lt;/span&gt; PostgreSQL image ships&amp;nbsp;with &lt;code&gt;/usr/libexec/check-container&lt;/code&gt;. For the upstream image,&amp;nbsp;use &lt;code&gt;pg_isready -U keycloak&lt;/code&gt; instead.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Create the data directory and set permissions for the container&amp;#8217;s PostgreSQL user (&lt;span class="caps"&gt;UID&lt;/span&gt;&amp;nbsp;26):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/opt/keycloak/postgres
setfacl&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;u:26:rwx&lt;span class="w"&gt; &lt;/span&gt;/opt/keycloak/postgres
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;RHEL&lt;/span&gt; PostgreSQL container runs as &lt;span class="caps"&gt;UID&lt;/span&gt; 26&amp;nbsp;(&lt;code&gt;postgres&lt;/code&gt;). Without this &lt;span class="caps"&gt;ACL&lt;/span&gt;, the container can&amp;#8217;t initialize the data directory and will fail with permission errors.&amp;nbsp;The &lt;code&gt;:z&lt;/code&gt; SELinux flag handles labeling, but filesystem-level ownership is a separate concern&amp;nbsp;- &lt;code&gt;setfacl&lt;/code&gt; grants access without changing the directory&amp;#8217;s&amp;nbsp;ownership.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Password rotation&lt;/strong&gt;: Red Hat&amp;#8217;s PostgreSQL container resets the database password to match&amp;nbsp;the &lt;code&gt;POSTGRESQL_PASSWORD&lt;/code&gt; environment variable on each startup, so rotating the password is straightforward: update the Podman secret and recreate the container. Since Podman&amp;nbsp;injects &lt;code&gt;type=env&lt;/code&gt; secrets at container creation time, a&amp;nbsp;simple &lt;code&gt;systemctl restart&lt;/code&gt; isn&amp;#8217;t enough - you need a fresh container creation to pick up the new secret&amp;nbsp;value:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;rm&lt;span class="w"&gt; &lt;/span&gt;keycloak_db_password
pwgen&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;tr&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;\n&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;keycloak_db_password&lt;span class="w"&gt; &lt;/span&gt;-
systemctl&lt;span class="w"&gt; &lt;/span&gt;stop&lt;span class="w"&gt; &lt;/span&gt;keycloak-db.service&lt;span class="w"&gt; &lt;/span&gt;keycloak-server.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;keycloak-db.service&lt;span class="w"&gt; &lt;/span&gt;keycloak-server.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="step-4-keycloak-application-container"&gt;Step 4: Keycloak Application&amp;nbsp;Container&lt;/h2&gt;
&lt;p&gt;Create &lt;code&gt;/etc/containers/systemd/keycloak-server.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;Description&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;Red Hat Build of Keycloak&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak-db.service&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.service&lt;/span&gt;
&lt;span class="na"&gt;Requires&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak-db.service&lt;/span&gt;

&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak-server&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry.redhat.io/rhbk/keycloak-rhel9:26.4-12&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;

&lt;span class="c1"&gt;# Dual network: backend for database, frontend for reverse proxy&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak-backend.network&lt;/span&gt;
&lt;span class="c1"&gt;# Must match the network your reverse proxy (e.g. Traefik) is on&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;frontend.network&lt;/span&gt;

&lt;span class="c1"&gt;# Database configuration&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_DB=postgres&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_DB_URL=jdbc:postgresql://keycloak-db:5432/keycloak&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_DB_USERNAME=keycloak&lt;/span&gt;

&lt;span class="c1"&gt;# Proxy configuration - Keycloak runs behind a reverse proxy&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_PROXY_HEADERS=xforwarded&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_HTTP_ENABLED=true&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_HOSTNAME_STRICT=false&lt;/span&gt;

&lt;span class="c1"&gt;# Hostname - set to your actual domain&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_HOSTNAME=keycloak.example.com&lt;/span&gt;

&lt;span class="c1"&gt;# Enable health and metrics endpoints&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_HEALTH_ENABLED=true&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_METRICS_ENABLED=true&lt;/span&gt;

&lt;span class="c1"&gt;# Bootstrap admin user (used for first login only)&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;KC_BOOTSTRAP_ADMIN_USERNAME=admin&lt;/span&gt;

&lt;span class="c1"&gt;# Secrets - database and admin passwords&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak_db_password,type=env,target=KC_DB_PASSWORD&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;keycloak_admin_password,type=env,target=KC_BOOTSTRAP_ADMIN_PASSWORD&lt;/span&gt;

&lt;span class="c1"&gt;# Persistent storage for themes and providers&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/keycloak/providers:/opt/keycloak/providers:z&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/keycloak/themes:/opt/keycloak/themes:z&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/localtime:/etc/localtime:ro&lt;/span&gt;

&lt;span class="c1"&gt;# Run Keycloak in production mode&lt;/span&gt;
&lt;span class="na"&gt;Exec&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;start&lt;/span&gt;

&lt;span class="c1"&gt;# Traefik labels for routing&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.enable=true&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.docker.network=frontend&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.rhbk.rule=Host(`keycloak.example.com`)&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.rhbk.entrypoints=https&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.rhbk.service=rhbk-svc&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.rhbk.tls.certresolver=traefiktls&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.rhbk.middlewares=secure-headers@file&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.services.rhbk-svc.loadbalancer.server.port=8080&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;multi-user.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;There&amp;#8217;s a lot happening here, so let&amp;#8217;s break it&amp;nbsp;down:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Network configuration&lt;/strong&gt;: The container joins&amp;nbsp;both &lt;code&gt;keycloak-backend.network&lt;/code&gt; (to reach PostgreSQL by&amp;nbsp;hostname &lt;code&gt;keycloak-db&lt;/code&gt;) and the frontend network where your reverse proxy lives.&amp;nbsp;Replace &lt;code&gt;frontend.network&lt;/code&gt; with whatever network your Traefik (or other reverse proxy) container is on - the name must match, or Traefik won&amp;#8217;t discover the container.&amp;nbsp;The &lt;code&gt;traefik.docker.network&lt;/code&gt; label must also&amp;nbsp;match.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Database connection&lt;/strong&gt;: Keycloak&amp;nbsp;resolves &lt;code&gt;keycloak-db&lt;/code&gt; via Podman&amp;#8217;s built-in &lt;span class="caps"&gt;DNS&lt;/span&gt; on the backend network. The &lt;span class="caps"&gt;JDBC&lt;/span&gt; &lt;span class="caps"&gt;URL&lt;/span&gt; points directly to the container name - no &lt;span class="caps"&gt;IP&lt;/span&gt; addresses to&amp;nbsp;manage.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Proxy settings&lt;/strong&gt;: &lt;code&gt;KC_PROXY_HEADERS=xforwarded&lt;/code&gt; tells Keycloak to&amp;nbsp;trust &lt;code&gt;X-Forwarded-*&lt;/code&gt; headers from your reverse proxy for correct &lt;span class="caps"&gt;URL&lt;/span&gt; generation, redirect URIs, and &lt;span class="caps"&gt;HTTPS&lt;/span&gt;&amp;nbsp;detection. &lt;code&gt;KC_HTTP_ENABLED=true&lt;/code&gt; allows plain &lt;span class="caps"&gt;HTTP&lt;/span&gt; on port 8080 since &lt;span class="caps"&gt;TLS&lt;/span&gt; terminates at the&amp;nbsp;proxy.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;KC_HOSTNAME_STRICT=false&lt;/code&gt;&lt;/strong&gt;: This is set&amp;nbsp;to &lt;code&gt;false&lt;/code&gt; here to simplify initial setup, but the Keycloak documentation&amp;nbsp;recommends &lt;code&gt;true&lt;/code&gt; for production.&amp;nbsp;With &lt;code&gt;hostname-strict&lt;/code&gt; disabled, Keycloak will respond on any hostname forwarded by the proxy, which can create security issues if your reverse proxy doesn&amp;#8217;t overwrite&amp;nbsp;the &lt;code&gt;Host&lt;/code&gt; header. Once you&amp;#8217;ve verified that your reverse proxy correctly&amp;nbsp;sets &lt;code&gt;Host&lt;/code&gt; and &lt;code&gt;X-Forwarded-*&lt;/code&gt; headers to your intended domain, switch this&amp;nbsp;to &lt;code&gt;true&lt;/code&gt;. If you&amp;#8217;re using Traefik with&amp;nbsp;the &lt;code&gt;Host()&lt;/code&gt; rule shown above, Traefik already filters by hostname, so&amp;nbsp;setting &lt;code&gt;KC_HOSTNAME_STRICT=true&lt;/code&gt; from the start is&amp;nbsp;safe.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Why &lt;code&gt;start&lt;/code&gt; without &lt;code&gt;--optimized&lt;/code&gt;?&lt;/strong&gt; Keycloak 26 has a build phase and a runtime phase. The upstream community image ships pre-built, but the &lt;span class="caps"&gt;RHBK&lt;/span&gt; image does not - it needs to run the build step on first startup.&amp;nbsp;Using &lt;code&gt;start&lt;/code&gt; (without &lt;code&gt;--optimized&lt;/code&gt;) lets Keycloak handle both phases automatically. This adds a few seconds to each startup, which is negligible for a service that starts once and runs continuously. If startup time matters to you (e.g., in a scaling scenario), you can build a custom image&amp;nbsp;with &lt;code&gt;kc.sh build&lt;/code&gt; baked in (see Step 5) and then switch&amp;nbsp;to &lt;code&gt;start --optimized&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Bootstrap admin&lt;/strong&gt;: &lt;code&gt;KC_BOOTSTRAP_ADMIN_USERNAME&lt;/code&gt; and &lt;code&gt;KC_BOOTSTRAP_ADMIN_PASSWORD&lt;/code&gt; create an admin account on first startup. This account is meant for initial setup only. After logging in, create a proper admin user in the Keycloak &lt;span class="caps"&gt;UI&lt;/span&gt; and consider removing the bootstrap&amp;nbsp;credentials.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;No health check in this Quadlet&lt;/strong&gt;: The &lt;span class="caps"&gt;RHBK&lt;/span&gt; image is based on &lt;span class="caps"&gt;UBI&lt;/span&gt; 9 Micro, which doesn&amp;#8217;t&amp;nbsp;include &lt;code&gt;curl&lt;/code&gt;, &lt;code&gt;wget&lt;/code&gt;, or any other &lt;span class="caps"&gt;HTTP&lt;/span&gt; client.&amp;nbsp;A &lt;code&gt;HealthCmd&lt;/code&gt; that relies&amp;nbsp;on &lt;code&gt;curl&lt;/code&gt; will always fail, marking the container as unhealthy - and Traefik ignores unhealthy containers by default. You can define a basic readiness check using Bash&amp;#8217;s&amp;nbsp;built-in &lt;code&gt;/dev/tcp&lt;/code&gt; to probe the management&amp;nbsp;port:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;HealthCmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;bash -c &amp;#39;echo &amp;gt; /dev/tcp/localhost/9000&amp;#39;&lt;/span&gt;
&lt;span class="na"&gt;HealthInterval&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;30s&lt;/span&gt;
&lt;span class="na"&gt;HealthTimeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5s&lt;/span&gt;
&lt;span class="na"&gt;HealthRetries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;3&lt;/span&gt;
&lt;span class="na"&gt;HealthStartPeriod&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;60s&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This confirms the management port is accepting connections, but it&amp;#8217;s a &lt;span class="caps"&gt;TCP&lt;/span&gt; check, not a full &lt;span class="caps"&gt;HTTP&lt;/span&gt; readiness probe. It won&amp;#8217;t catch cases where Keycloak&amp;#8217;s &lt;span class="caps"&gt;JVM&lt;/span&gt; is up but the application hasn&amp;#8217;t finished initializing. Use it as a basic liveness signal, not as a gate for traffic routing. We omit it from the main Quadlet above to keep things simple and avoid Traefik interaction issues, but add it if you want systemd-level health visibility&amp;nbsp;via &lt;code&gt;systemctl status&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Traefik labels&lt;/strong&gt;: The labels configure Traefik&amp;#8217;s dynamic&amp;nbsp;routing. &lt;code&gt;traefik.docker.network&lt;/code&gt; must point to the shared frontend network so Traefik routes traffic over the right interface. The service label explicitly names the load balancer backend and maps it to port 8080. If you&amp;#8217;re using a different reverse proxy, replace these labels with your proxy&amp;#8217;s configuration (see the reverse proxy section&amp;nbsp;below).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Systemd dependencies&lt;/strong&gt;: &lt;code&gt;Requires=keycloak-db.service&lt;/code&gt; ensures PostgreSQL is running before Keycloak attempts to connect - if the database service fails, Keycloak stops&amp;nbsp;too. &lt;code&gt;After=traefik.service&lt;/code&gt; ensures the reverse proxy is up before Keycloak starts, so Traefik can discover it&amp;nbsp;immediately.&lt;/p&gt;
&lt;p&gt;Create the data directories and set permissions. The &lt;span class="caps"&gt;RHBK&lt;/span&gt; container runs as &lt;span class="caps"&gt;UID&lt;/span&gt; 1000, so the bind-mounted directories need to be accessible to that user - just as we set up &lt;span class="caps"&gt;UID&lt;/span&gt; 26 for&amp;nbsp;PostgreSQL:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;/opt/keycloak/&lt;span class="o"&gt;{&lt;/span&gt;providers,themes&lt;span class="o"&gt;}&lt;/span&gt;
setfacl&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;u:1000:rwx&lt;span class="w"&gt; &lt;/span&gt;/opt/keycloak/providers&lt;span class="w"&gt; &lt;/span&gt;/opt/keycloak/themes
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="step-5-build-the-optimized-image-optional"&gt;Step 5: Build the Optimized Image&amp;nbsp;(Optional)&lt;/h2&gt;
&lt;p&gt;If you need custom providers, custom themes baked into the image, or non-default database drivers, build a custom&amp;nbsp;image:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;registry.redhat.io/rhbk/keycloak-rhel9:26.4-12&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;AS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;builder&lt;/span&gt;

&lt;span class="c"&gt;# Set build-time configuration&lt;/span&gt;
&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;KC_DB&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;postgres
&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;KC_HEALTH_ENABLED&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;
&lt;span class="k"&gt;ENV&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;KC_METRICS_ENABLED&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="nb"&gt;true&lt;/span&gt;

&lt;span class="c"&gt;# Add custom providers if needed&lt;/span&gt;
&lt;span class="c"&gt;# COPY my-provider.jar /opt/keycloak/providers/&lt;/span&gt;

&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;/opt/keycloak/bin/kc.sh&lt;span class="w"&gt; &lt;/span&gt;build

&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;registry.redhat.io/rhbk/keycloak-rhel9:26.4-12&lt;/span&gt;

&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--from&lt;span class="o"&gt;=&lt;/span&gt;builder&lt;span class="w"&gt; &lt;/span&gt;/opt/keycloak/&lt;span class="w"&gt; &lt;/span&gt;/opt/keycloak/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Build with&amp;nbsp;Podman:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;build&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;localhost/keycloak-custom:26.4&lt;span class="w"&gt; &lt;/span&gt;.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then update the Quadlet to&amp;nbsp;use &lt;code&gt;Image=localhost/keycloak-custom:26.4&lt;/code&gt;,&amp;nbsp;switch &lt;code&gt;AutoUpdate=registry&lt;/code&gt; to &lt;code&gt;AutoUpdate=local&lt;/code&gt; (Podman&amp;nbsp;uses &lt;code&gt;registry&lt;/code&gt; for remote refs&amp;nbsp;and &lt;code&gt;local&lt;/code&gt; for locally built images), and&amp;nbsp;change &lt;code&gt;Exec=start&lt;/code&gt; to &lt;code&gt;Exec=start --optimized&lt;/code&gt; since the build step is already baked into the&amp;nbsp;image.&lt;/p&gt;
&lt;p&gt;For most deployments, the default &lt;span class="caps"&gt;RHBK&lt;/span&gt; image&amp;nbsp;with &lt;code&gt;start&lt;/code&gt; is sufficient. Use a custom image&amp;nbsp;plus &lt;code&gt;start --optimized&lt;/code&gt; when you need faster restarts or want providers and themes baked in rather than&amp;nbsp;bind-mounted.&lt;/p&gt;
&lt;h2 id="step-6-deployment"&gt;Step 6:&amp;nbsp;Deployment&lt;/h2&gt;
&lt;p&gt;Reload systemd to pick up the new Quadlet&amp;nbsp;files:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;daemon-reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Start the&amp;nbsp;stack:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;keycloak-db.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;keycloak-server.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Verify everything is&amp;nbsp;running:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check container status&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;keycloak-db.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;keycloak-server.service

&lt;span class="c1"&gt;# Watch Keycloak logs for successful startup&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;keycloak-server.service&lt;span class="w"&gt; &lt;/span&gt;-f
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You should see Keycloak log its startup sequence, database migration (on first run), and&amp;nbsp;eventually:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Keycloak 26.4.10.redhat-00001 on JVM (powered by Quarkus 3.27.2.redhat-00001) started in 32.000s.
Listening on: http://0.0.0.0:8080. Management interface listening on http://0.0.0.0:9000.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="step-7-reverse-proxy-notes"&gt;Step 7: Reverse Proxy&amp;nbsp;Notes&lt;/h2&gt;
&lt;p&gt;The Quadlet above includes Traefik labels that handle routing configuration declaratively - no separate Traefik config files needed. If you&amp;#8217;re using Traefik, the Quadlet is&amp;nbsp;self-contained.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re using a different reverse proxy, remove the Traefik labels and configure routing manually. Keycloak listens on port 8080 for &lt;span class="caps"&gt;HTTP&lt;/span&gt;. Here are two common&amp;nbsp;alternatives:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Caddy&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;Caddyfile&lt;/code&gt; snippet):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;keycloak.example.com {
    reverse_proxy keycloak-server:8080
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;nginx&lt;/strong&gt; (server&amp;nbsp;block):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;server&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;listen&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;ssl&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;server_name&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;keycloak.example.com&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kn"&gt;location&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_pass&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;http://keycloak-server:8080&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;Host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;X-Real-IP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$remote_addr&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;X-Forwarded-For&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$proxy_add_x_forwarded_for&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;X-Forwarded-Proto&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$scheme&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_set_header&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;X-Forwarded-Host&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$host&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_buffer_size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;128k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_buffers&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;256k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kn"&gt;proxy_busy_buffers_size&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;256k&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The larger proxy buffers for nginx may be necessary - Keycloak&amp;#8217;s &lt;span class="caps"&gt;OIDC&lt;/span&gt; responses can include headers that exceed nginx&amp;#8217;s conservative defaults. Without sufficient buffer space, you&amp;#8217;ll&amp;nbsp;see &lt;code&gt;502 Bad Gateway&lt;/code&gt; errors during authentication flows. The sizes above are generous; adjust downward if your setup works without&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;Both examples above assume the reverse proxy is containerized on the same frontend network, so Podman&amp;#8217;s &lt;span class="caps"&gt;DNS&lt;/span&gt;&amp;nbsp;resolves &lt;code&gt;keycloak-server&lt;/code&gt;. If your reverse proxy runs directly on the host, you&amp;#8217;ll need to publish Keycloak&amp;#8217;s port&amp;nbsp;(&lt;code&gt;PublishPort=8080:8080&lt;/code&gt; in the Quadlet) and&amp;nbsp;use &lt;code&gt;proxy_pass http://127.0.0.1:8080&lt;/code&gt; instead.&lt;/p&gt;
&lt;h2 id="step-8-automatic-updates"&gt;Step 8: Automatic&amp;nbsp;Updates&lt;/h2&gt;
&lt;p&gt;Enable Podman&amp;#8217;s auto-update&amp;nbsp;timer:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;podman-auto-update.timer
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With &lt;code&gt;AutoUpdate=registry&lt;/code&gt; in both Quadlet files, Podman checks daily for new images and recreates containers when updates are available. Volumes persist across recreations, so your data is&amp;nbsp;safe.&lt;/p&gt;
&lt;p&gt;For Keycloak specifically, be deliberate about version updates. Pin to a specific tag&amp;nbsp;like &lt;code&gt;26.4-12&lt;/code&gt; (not &lt;code&gt;latest&lt;/code&gt;) in the Quadlet and bump the version manually after reviewing the changelog. Keycloak upgrades can include database migrations that you want to run intentionally, not at 3 &lt;span class="caps"&gt;AM&lt;/span&gt; via an&amp;nbsp;auto-update.&lt;/p&gt;
&lt;h2 id="post-deployment-first-login"&gt;Post-Deployment: First&amp;nbsp;Login&lt;/h2&gt;
&lt;p&gt;Navigate&amp;nbsp;to &lt;code&gt;https://keycloak.example.com&lt;/code&gt; and log in with the bootstrap admin credentials. From&amp;nbsp;here:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Create a new realm&lt;/strong&gt; for your applications (don&amp;#8217;t use&amp;nbsp;the &lt;code&gt;master&lt;/code&gt; realm for end&amp;nbsp;users)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Create a permanent admin user&lt;/strong&gt; within the master&amp;nbsp;realm&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Set up your first client&lt;/strong&gt; - this is the &lt;span class="caps"&gt;OIDC&lt;/span&gt;/&lt;span class="caps"&gt;SAML&lt;/span&gt; application&amp;nbsp;registration&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Configure identity providers&lt;/strong&gt; if you want social login or federation with existing &lt;span class="caps"&gt;LDAP&lt;/span&gt;/&lt;span class="caps"&gt;AD&lt;/span&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Once you have a proper admin account, clean up the bootstrap&amp;nbsp;user:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Delete the bootstrap admin&lt;/strong&gt; in the Keycloak &lt;span class="caps"&gt;UI&lt;/span&gt;: navigate to&amp;nbsp;the &lt;code&gt;master&lt;/code&gt; realm, go to Users, find&amp;nbsp;the &lt;code&gt;admin&lt;/code&gt; account, and delete&amp;nbsp;it&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Remove the environment variables&lt;/strong&gt; from the Quadlet: delete&amp;nbsp;the &lt;code&gt;KC_BOOTSTRAP_ADMIN_USERNAME&lt;/code&gt; line, and&amp;nbsp;the &lt;code&gt;Secret=keycloak_admin_password&lt;/code&gt; line&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Restart&lt;/strong&gt; to&amp;nbsp;apply:&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;daemon-reload
systemctl&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;keycloak-server.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Removing the environment variables alone doesn&amp;#8217;t delete the user from the database - it only stops Keycloak from trying to create it on boot. Both steps are&amp;nbsp;necessary.&lt;/p&gt;
&lt;h2 id="monitoring"&gt;Monitoring&lt;/h2&gt;
&lt;p&gt;Keycloak 26 exposes Prometheus-compatible metrics on the management&amp;nbsp;port:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# From the host (or any container on the same network)&lt;/span&gt;
curl&lt;span class="w"&gt; &lt;/span&gt;http://keycloak-server:9000/metrics
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Point your Prometheus instance at this endpoint for dashboards covering authentication rates, token issuance, active sessions, and &lt;span class="caps"&gt;JVM&lt;/span&gt;&amp;nbsp;health.&lt;/p&gt;
&lt;p&gt;Health endpoints are equally useful for&amp;nbsp;alerting:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Liveness - is the process alive?&lt;/span&gt;
curl&lt;span class="w"&gt; &lt;/span&gt;http://keycloak-server:9000/health/live

&lt;span class="c1"&gt;# Readiness - is it ready to handle requests?&lt;/span&gt;
curl&lt;span class="w"&gt; &lt;/span&gt;http://keycloak-server:9000/health/ready
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="operational-notes"&gt;Operational&amp;nbsp;Notes&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Backups&lt;/strong&gt;: The PostgreSQL data lives&amp;nbsp;at &lt;code&gt;/opt/keycloak/postgres&lt;/code&gt;.&amp;nbsp;Use &lt;code&gt;pg_dump&lt;/code&gt; inside the container for logical&amp;nbsp;backups:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;exec&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;keycloak-db&lt;span class="w"&gt; &lt;/span&gt;pg_dump&lt;span class="w"&gt; &lt;/span&gt;-U&lt;span class="w"&gt; &lt;/span&gt;keycloak&lt;span class="w"&gt; &lt;/span&gt;keycloak&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;keycloak-backup.sql
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Log management&lt;/strong&gt;: All container output flows to journald.&amp;nbsp;Standard &lt;code&gt;journalctl&lt;/code&gt; filtering&amp;nbsp;applies:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Keycloak errors only&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;keycloak-server.service&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;err

&lt;span class="c1"&gt;# Database logs since last boot&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;keycloak-db.service&lt;span class="w"&gt; &lt;/span&gt;-b
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Scaling considerations&lt;/strong&gt;: This single-host setup handles a surprising number of users. Keycloak&amp;#8217;s session cache is local by default. If you eventually need clustering, Keycloak supports Infinispan-based distributed caches - but that&amp;#8217;s a different&amp;nbsp;article.&lt;/p&gt;
&lt;h2 id="a-note-on-the-upstream-keycloak-image"&gt;A Note on the Upstream Keycloak&amp;nbsp;Image&lt;/h2&gt;
&lt;p&gt;The upstream community project publishes images&amp;nbsp;at &lt;code&gt;quay.io/keycloak/keycloak&lt;/code&gt;. If you want to use the upstream image&amp;nbsp;instead:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;quay.io/keycloak/keycloak:26.0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The configuration is identical - same environment variables, same startup flags. One difference: the upstream image ships pre-built, so you can&amp;nbsp;use &lt;code&gt;start --optimized&lt;/code&gt; directly without a custom build&amp;nbsp;step.&lt;/p&gt;
&lt;p&gt;The upstream images are community-maintained and don&amp;#8217;t carry Red Hat&amp;#8217;s security errata or support lifecycle. This guide uses &lt;span class="caps"&gt;RHBK&lt;/span&gt; for its build quality and predictable errata cadence, though as noted above, running &lt;span class="caps"&gt;RHBK&lt;/span&gt; on Podman (rather than OpenShift) is itself not a Red Hat-supported configuration. Choose whichever image fits your operational&amp;nbsp;model.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.keycloak.org/guides#server"&gt;Keycloak 26 Server&amp;nbsp;Guide&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.keycloak.org/server/containers"&gt;Keycloak Container Image&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html"&gt;Podman Quadlet&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://access.redhat.com/documentation/en-us/red_hat_build_of_keycloak/26.0"&gt;Red Hat Build of Keycloak&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Production-Grade Container Deployment with Podman Quadlets&lt;/a&gt; - my earlier deep-dive into Quadlet-based&amp;nbsp;deployments&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="linux"/><category term="podman"/><category term="containers"/><category term="quadlet"/><category term="systemd"/><category term="keycloak"/><category term="identity"/><category term="rhel"/></entry><entry><title>My Multi-Stage Backup Strategy: ZFS, Proxmox, and Paranoia</title><link href="https://hofstede.it/my-multi-stage-backup-strategy-zfs-proxmox-and-paranoia/" rel="alternate"/><published>2026-03-28T00:00:00+01:00</published><updated>2026-03-28T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-03-28:/my-multi-stage-backup-strategy-zfs-proxmox-and-paranoia/</id><summary type="html">&lt;p&gt;Backups are the thing everyone knows they should do and nobody does well enough. Here&amp;#8217;s my multi-stage strategy for keeping about a dozen servers safe: &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots with sanoid, off-site replication with syncoid to rsync.net, Proxmox Backup Server with an S3 backend for VMs, and a creative Podman trick for backing up &lt;span class="caps"&gt;RHEL&lt;/span&gt; hosts that don&amp;#8217;t have proxmox-backup-client. Plus a dead man&amp;#8217;s switch, because the only thing worse than no backups is backups that silently stopped working three months&amp;nbsp;ago.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Logo" src="https://hofstede.it/images/2026-03-28-multi-stage-backup-strategy.png" title="Multi-Stage Backup Strategy: Header image"&gt;&lt;/p&gt;
&lt;p&gt;Everybody has a backup strategy. Some people&amp;#8217;s backup strategy is &amp;#8220;I should really set up backups.&amp;#8221; Others involve a cronjob that worked once in 2019 and has been silently failing ever since. A select few involve a &lt;span class="caps"&gt;USB&lt;/span&gt; drive in a desk drawer labeled &amp;#8220;&lt;span class="caps"&gt;IMPORTANT&lt;/span&gt;&amp;#8221; in permanent&amp;nbsp;marker.&lt;/p&gt;
&lt;p&gt;Mine layers &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots, off-site replication, a Proxmox Backup Server that&amp;#8217;s really just a proxy in front of cheap object storage, a Podman container doing something that would make a packaging engineer cry, and a dead man&amp;#8217;s switch that pages me if any of it stops working. It&amp;#8217;s not elegant in every layer, but it covers roughly a dozen servers across FreeBSD and Linux, and it lets me sleep at&amp;nbsp;night.&lt;/p&gt;
&lt;p&gt;The classic &lt;strong&gt;3-2-1 backup rule&lt;/strong&gt; says: three copies of your data, on two different media types, with one copy off-site. What I&amp;#8217;m about to describe exceeds it in some areas and only meets it in others - but the principle is sound, and if you take nothing else from this article, take&amp;nbsp;3-2-1.&lt;/p&gt;
&lt;p&gt;Before diving into tools, it helps to name the threats each layer&amp;nbsp;addresses:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Accidental deletion or bad deploys&lt;/strong&gt; → local &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots (instant&amp;nbsp;rollback)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Disk or pool failure&lt;/strong&gt; → off-site replication to&amp;nbsp;rsync.net&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Total host loss&lt;/strong&gt; → remote backups via &lt;span class="caps"&gt;PBS&lt;/span&gt; + S3 and&amp;nbsp;syncoid&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Silent backup failure&lt;/strong&gt; → dead man&amp;#8217;s switch&amp;nbsp;monitoring&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Corrupt or incomplete backups&lt;/strong&gt; → quarterly restore&amp;nbsp;tests&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Every layer below maps to at least one of these. If you&amp;#8217;re designing your own strategy, start from your threat model and work outward - the tools are interchangeable, the thinking&amp;nbsp;isn&amp;#8217;t.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;div class="toc"&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#layer-1-zfs-snapshots-with-sanoid-freebsd"&gt;Layer 1: &lt;span class="caps"&gt;ZFS&lt;/span&gt; Snapshots with Sanoid&amp;nbsp;(FreeBSD)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#layer-2-off-site-replication-with-syncoid"&gt;Layer 2: Off-Site Replication with Syncoid&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#the-ssh-configuration"&gt;The &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;Configuration&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#layer-3-proxmox-backup-server-for-virtual-machines"&gt;Layer 3: Proxmox Backup Server for Virtual&amp;nbsp;Machines&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#layer-4-linux-servers"&gt;Layer 4: Linux Servers&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#option-a-proxmox-backup-client"&gt;Option A: Proxmox Backup&amp;nbsp;Client&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#option-b-plain-rsync"&gt;Option B: Plain&amp;nbsp;rsync&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#option-c-the-podman-trick-when-packages-dont-exist"&gt;Option C: The Podman Trick (When Packages Don&amp;#8217;t&amp;nbsp;Exist)&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-glue-dead-mans-switch-monitoring"&gt;The Glue: Dead Man&amp;#8217;s Switch&amp;nbsp;Monitoring&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#quarterly-restore-tests"&gt;Quarterly Restore Tests&lt;/a&gt;&lt;ul&gt;
&lt;li&gt;&lt;a href="#how-restores-actually-work"&gt;How Restores Actually&amp;nbsp;Work&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/li&gt;
&lt;li&gt;&lt;a href="#putting-it-all-together"&gt;Putting It All&amp;nbsp;Together&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#what-this-costs"&gt;What This&amp;nbsp;Costs&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#references"&gt;References&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;/div&gt;
&lt;h2 id="layer-1-zfs-snapshots-with-sanoid-freebsd"&gt;Layer 1: &lt;span class="caps"&gt;ZFS&lt;/span&gt; Snapshots with Sanoid&amp;nbsp;(FreeBSD)&lt;/h2&gt;
&lt;p&gt;All my FreeBSD servers run &lt;span class="caps"&gt;ZFS&lt;/span&gt;. If you&amp;#8217;re on FreeBSD and not using &lt;span class="caps"&gt;ZFS&lt;/span&gt;, I have questions. (If you want answers about &lt;span class="caps"&gt;ZFS&lt;/span&gt; first, I wrote &lt;a href="https://hofstede.it/freebsd-foundationals-zfs-the-last-filesystem-youll-ever-need/"&gt;an entire article about it&lt;/a&gt;.)&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots are the first line of defense. They&amp;#8217;re instant, they&amp;#8217;re effectively free at creation time (copy-on-write means a snapshot consumes zero additional space until data changes), and they&amp;#8217;ve saved me from &amp;#8220;oops&amp;#8221; moments more times than I care to admit. But snapshots alone aren&amp;#8217;t backups - they live on the same pool as your data. If the disk dies, the snapshots die with&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;Enter &lt;strong&gt;&lt;a href="https://github.com/jimsalterjrs/sanoid"&gt;sanoid&lt;/a&gt;&lt;/strong&gt;, a policy-driven snapshot management tool. On FreeBSD, it&amp;#8217;s&amp;nbsp;a &lt;code&gt;pkg install sanoid&lt;/code&gt; away. You define retention policies per dataset, point sanoid at them, and it handles creation and pruning&amp;nbsp;automatically.&lt;/p&gt;
&lt;p&gt;Here&amp;#8217;s a representative configuration from one of my&amp;nbsp;servers:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /usr/local/etc/sanoid/sanoid.conf&lt;/span&gt;

&lt;span class="k"&gt;[template_production]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autosnap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autoprune&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Retention Policy&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;hourly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;daily&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;14&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;weekly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;4&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;monthly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;yearly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;

&lt;span class="k"&gt;[template_database]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autosnap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autoprune&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;# Retention Policy&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;hourly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;daily&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;3&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;weekly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;monthly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;yearly&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;0&lt;/span&gt;

&lt;span class="c1"&gt;# Apply to datasets&lt;/span&gt;
&lt;span class="k"&gt;[zroot]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;use_template&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;production&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;recursive&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;process_children_only&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;no&lt;/span&gt;

&lt;span class="k"&gt;[zroot/postgres_data]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;use_template&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;database&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;recursive&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;yes&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;process_children_only&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;no&lt;/span&gt;

&lt;span class="c1"&gt;# Exclude datasets that don&amp;#39;t need snapshots&lt;/span&gt;
&lt;span class="k"&gt;[zroot/tmp]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autosnap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;no&lt;/span&gt;

&lt;span class="k"&gt;[zroot/bastille/logs]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autosnap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;no&lt;/span&gt;

&lt;span class="k"&gt;[zroot/bastille/jails/nginx/cache]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="na"&gt;autosnap&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;no&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The philosophy is straightforward: production data gets 14 daily snapshots and 4 weekly ones. That&amp;#8217;s about six weeks of rollback capability. Database datasets get a shorter window (3 days) because database snapshots without coordinated dumps are crash-consistent at best - useful, but not a replacement for proper application-level backups. Temporary directories and caches get nothing, because&amp;nbsp;snapshotting &lt;code&gt;/tmp&lt;/code&gt; is a waste of everyone&amp;#8217;s&amp;nbsp;time.&lt;/p&gt;
&lt;p&gt;Sanoid runs via&amp;nbsp;cron:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sanoid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="n"&gt;cron&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This pattern is identical across all ~10 of my FreeBSD servers. Same templates, same retention policies, same cron entry. The only thing that changes is which datasets exist and which exclusions apply. Consistency is a feature - when you&amp;#8217;re managing multiple hosts, you don&amp;#8217;t want to remember that &lt;em&gt;this&lt;/em&gt; server has different retention because you were feeling creative at 2 &lt;span class="caps"&gt;AM&lt;/span&gt; six months&amp;nbsp;ago.&lt;/p&gt;
&lt;h2 id="layer-2-off-site-replication-with-syncoid"&gt;Layer 2: Off-Site Replication with&amp;nbsp;Syncoid&lt;/h2&gt;
&lt;p&gt;Local snapshots protect against accidental deletion and bad updates. They do absolutely nothing against disk failure, ransomware, or the catastrophic scenario where the entire server goes away. For that, you need off-site&amp;nbsp;copies.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Syncoid&lt;/strong&gt; (part of the sanoid suite) handles this. It&amp;nbsp;uses &lt;code&gt;zfs send/recv&lt;/code&gt; over &lt;span class="caps"&gt;SSH&lt;/span&gt; to replicate datasets to a remote &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool. The first run sends a full copy; subsequent runs send only incremental differences. For a server with hundreds of gigabytes of data, the nightly incremental transfer is typically measured in&amp;nbsp;megabytes.&lt;/p&gt;
&lt;p&gt;I use &lt;a href="https://rsync.net"&gt;rsync.net&lt;/a&gt; as my off-site target. They offer &lt;span class="caps"&gt;ZFS&lt;/span&gt;-capable storage specifically designed for this use case - you get a remote &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool that&amp;nbsp;accepts &lt;code&gt;zfs recv&lt;/code&gt; over &lt;span class="caps"&gt;SSH&lt;/span&gt;. It&amp;#8217;s not the cheapest option - but &amp;#8220;cheap&amp;#8221; and &amp;#8220;I trust this with my only off-site copy of everything&amp;#8221; are different&amp;nbsp;categories.&lt;/p&gt;
&lt;p&gt;Here&amp;#8217;s the backup script that runs&amp;nbsp;nightly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/sh&lt;/span&gt;

&lt;span class="c1"&gt;# Configuration&lt;/span&gt;
&lt;span class="nv"&gt;SOURCE_POOL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;zroot&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;TARGET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;backup@ab1234c.rsync.net:data1/apollo/zroot&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;MONITOR_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://uptime.example.com/api/push/xxxxxxxxxx?status=up&amp;amp;msg=OK&amp;amp;ping=&amp;quot;&lt;/span&gt;

&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Starting off-site backup to rsync.net...&amp;quot;&lt;/span&gt;

/usr/local/bin/syncoid&lt;span class="w"&gt; &lt;/span&gt;--no-sync-snap&lt;span class="w"&gt; &lt;/span&gt;--no-privilege-elevation&lt;span class="w"&gt; &lt;/span&gt;--recursive&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--sendoptions&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;w&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--exclude&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;zroot/tmp&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--exclude&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;zroot/bastille/logs&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--exclude&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;zroot/bastille/jails/nginx/cache&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--recvoptions&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;u o canmount=off o mountpoint=none&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SOURCE_POOL&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;TARGET&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$?&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-eq&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;]&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;then&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;Backup successful. Triggering monitor...&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/usr/local/bin/curl&lt;span class="w"&gt; &lt;/span&gt;-fsS&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;MONITOR_URL&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;span class="k"&gt;else&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;echo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ERROR: Backup FAILED. Monitor NOT triggered.&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nb"&gt;exit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;span class="k"&gt;fi&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;A few things worth&amp;nbsp;noting:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;--sendoptions="w"&lt;/code&gt;&lt;/strong&gt; is critical.&amp;nbsp;The &lt;code&gt;w&lt;/code&gt; flag means &amp;#8220;raw send&amp;#8221; - encrypted datasets are sent in their encrypted form. Without&amp;nbsp;this, &lt;code&gt;zfs send&lt;/code&gt; would transmit the decrypted data stream, and the datasets would arrive unencrypted at rest on the remote end. With raw send, your encrypted datasets remain encrypted at rest on the backup target. rsync.net never needs your encryption keys, and if someone compromises the backup storage, they get&amp;nbsp;ciphertext.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;--no-privilege-elevation&lt;/code&gt;&lt;/strong&gt; means syncoid won&amp;#8217;t try to&amp;nbsp;use &lt;code&gt;sudo&lt;/code&gt; on the remote end. This works because the remote user&amp;nbsp;(&lt;code&gt;backup&lt;/code&gt; in this example) has been granted only the specific &lt;span class="caps"&gt;ZFS&lt;/span&gt; permissions needed to receive snapshots - nothing&amp;nbsp;more.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;--recvoptions="u o canmount=off o mountpoint=none"&lt;/code&gt;&lt;/strong&gt; tells the receiving end not to mount the datasets. This is a backup target, not a live system. You don&amp;#8217;t&amp;nbsp;want &lt;code&gt;zfs recv&lt;/code&gt; trying to&amp;nbsp;mount &lt;code&gt;zroot&lt;/code&gt; on the remote&amp;nbsp;machine.&lt;/p&gt;
&lt;p&gt;The same excludes from the sanoid config reappear here. Yes, the duplication is intentional - each layer should be independently correct, so if you change one without the other, neither silently breaks. There&amp;#8217;s no point in replicating temp files and cache directories across the&amp;nbsp;internet.&lt;/p&gt;
&lt;p&gt;The cron schedule runs this at 4 &lt;span class="caps"&gt;AM&lt;/span&gt;, after sanoid has already created the nightly snapshots at&amp;nbsp;midnight:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sanoid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="n"&gt;cron&lt;/span&gt;
&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;daily_backup&lt;/span&gt;&lt;span class="mf"&gt;.&lt;/span&gt;&lt;span class="n"&gt;sh&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;var&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;backup_last&lt;/span&gt;&lt;span class="mf"&gt;.&lt;/span&gt;&lt;span class="nb"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;2&lt;/span&gt;&lt;span class="o"&gt;&amp;gt;&lt;/span&gt;&lt;span class="err"&gt;&amp;amp;&lt;/span&gt;&lt;span class="mf"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Retention of old snapshots on the rsync.net side is handled by a separate sanoid cron job running &lt;em&gt;on&lt;/em&gt; the rsync.net host&amp;nbsp;itself:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="mf"&gt;0&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mf"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nb"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;local&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;bin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="n"&gt;sanoid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="n"&gt;verbose&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;--&lt;/span&gt;&lt;span class="n"&gt;prune&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="n"&gt;snapshots&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This keeps the remote snapshot history tidy without me having to manage it from every source&amp;nbsp;server.&lt;/p&gt;
&lt;h3 id="the-ssh-configuration"&gt;The &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;Configuration&lt;/h3&gt;
&lt;p&gt;The backup user connects via a dedicated &lt;span class="caps"&gt;SSH&lt;/span&gt; key with limited&amp;nbsp;permissions:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# ~/.ssh/config (on the source server)
Host rsync_backup
    HostName ab1234c.rsync.net
    User backup
    Port 22
    IdentityFile /root/.ssh/id_ed25519_backup
    Compression yes
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;backup&lt;/code&gt; user on rsync.net has only the &lt;span class="caps"&gt;ZFS&lt;/span&gt; permissions required for receiving snapshots. If someone compromises a source server and steals the &lt;span class="caps"&gt;SSH&lt;/span&gt; key, the worst they can do is send snapshots to the backup pool and list existing ones. They can&amp;#8217;t delete snapshots, destroy datasets, or access anything outside the designated backup path. This is the ransomware angle: even if an attacker has full root on a source server, they cannot destroy or encrypt the off-site&amp;nbsp;backups.&lt;/p&gt;
&lt;h2 id="layer-3-proxmox-backup-server-for-virtual-machines"&gt;Layer 3: Proxmox Backup Server for Virtual&amp;nbsp;Machines&lt;/h2&gt;
&lt;p&gt;Not everything is FreeBSD. I run two Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; nodes that serve as lab environments for testing things quickly, plus a handful of utility VMs. These need a different backup&amp;nbsp;strategy.&lt;/p&gt;
&lt;p&gt;My approach: a &lt;strong&gt;Proxmox Backup Server (&lt;span class="caps"&gt;PBS&lt;/span&gt;)&lt;/strong&gt; instance running on a minimal &lt;span class="caps"&gt;VM&lt;/span&gt; (1 vCPU, 1 &lt;span class="caps"&gt;GB&lt;/span&gt; &lt;span class="caps"&gt;RAM&lt;/span&gt;) at a cheap cloud provider, with &lt;strong&gt;Backblaze B2&lt;/strong&gt; as the storage&amp;nbsp;backend.&lt;/p&gt;
&lt;p&gt;The beauty of this setup is that &lt;span class="caps"&gt;PBS&lt;/span&gt; supports S3-compatible object storage as a datastore. The &lt;span class="caps"&gt;PBS&lt;/span&gt; &lt;span class="caps"&gt;VM&lt;/span&gt; itself is tiny and stores almost nothing locally - it&amp;#8217;s essentially a proxy that handles deduplication, encryption, and the Proxmox backup protocol, then shoves the actual data into B2. You get Proxmox&amp;#8217;s excellent incremental backup and deduplication without paying for a beefy &lt;span class="caps"&gt;VM&lt;/span&gt; with terabytes of local&amp;nbsp;storage.&lt;/p&gt;
&lt;p&gt;On the Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt; side, this is completely transparent. You add the &lt;span class="caps"&gt;PBS&lt;/span&gt; as a storage target in the Proxmox web &lt;span class="caps"&gt;UI&lt;/span&gt;, configure a backup schedule, and the VMs back up nightly through the standard Proxmox backup mechanism. The &lt;span class="caps"&gt;PVE&lt;/span&gt; nodes don&amp;#8217;t know or care that B2 is the actual storage behind it - they just see a &lt;span class="caps"&gt;PBS&lt;/span&gt;&amp;nbsp;endpoint.&lt;/p&gt;
&lt;p&gt;The economics work out nicely: a tiny &lt;span class="caps"&gt;VPS&lt;/span&gt; to run &lt;span class="caps"&gt;PBS&lt;/span&gt; costs a few euros per month, and B2 storage is $6/&lt;span class="caps"&gt;TB&lt;/span&gt;/month with free egress via the Cloudflare Bandwidth Alliance. For a lab environment with a few hundred gigabytes of &lt;span class="caps"&gt;VM&lt;/span&gt; data, the total cost is&amp;nbsp;negligible.&lt;/p&gt;
&lt;h2 id="layer-4-linux-servers"&gt;Layer 4: Linux&amp;nbsp;Servers&lt;/h2&gt;
&lt;p&gt;Linux servers are the messy middle child of this strategy. There&amp;#8217;s no single approach, because the servers themselves vary too&amp;nbsp;much.&lt;/p&gt;
&lt;h3 id="option-a-proxmox-backup-client"&gt;Option A: Proxmox Backup&amp;nbsp;Client&lt;/h3&gt;
&lt;p&gt;For Linux hosts where it&amp;#8217;s&amp;nbsp;available, &lt;code&gt;proxmox-backup-client&lt;/code&gt; is the cleanest option. It supports file-level and image-level backups to &lt;span class="caps"&gt;PBS&lt;/span&gt; with deduplication and encryption. Install the client, point it at your &lt;span class="caps"&gt;PBS&lt;/span&gt; instance, schedule a cron job, done. These backups land in the same &lt;span class="caps"&gt;PBS&lt;/span&gt; + B2 infrastructure described&amp;nbsp;above.&lt;/p&gt;
&lt;h3 id="option-b-plain-rsync"&gt;Option B: Plain&amp;nbsp;rsync&lt;/h3&gt;
&lt;p&gt;Not every Linux server needs the full &lt;span class="caps"&gt;PBS&lt;/span&gt; treatment. Some are simple workhorses where the important data is just files - configuration, application data, maybe some database exports. For these, rsync over &lt;span class="caps"&gt;SSH&lt;/span&gt; to rsync.net does the&amp;nbsp;job.&lt;/p&gt;
&lt;p&gt;The key is what happens &lt;em&gt;before&lt;/em&gt; rsync runs. You can&amp;#8217;t just rsync a live PostgreSQL data directory and expect a consistent backup. The same goes for MySQL, Redis &lt;span class="caps"&gt;AOF&lt;/span&gt; files, or anything else that keeps state in memory. So the backup script follows a pattern: dump first, then&amp;nbsp;sync.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-euo&lt;span class="w"&gt; &lt;/span&gt;pipefail

&lt;span class="nv"&gt;BACKUP_DIR&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/backups/pre-rsync&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;RSYNC_TARGET&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;backup@ab1234c.rsync.net:data1/hostname/&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/root/.ssh/id_ed25519_backup&amp;quot;&lt;/span&gt;
&lt;span class="nv"&gt;MONITOR_URL&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://uptime.example.com/api/push/xxxxxxxxxx?status=up&amp;amp;msg=OK&amp;amp;ping=&amp;quot;&lt;/span&gt;

mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BACKUP_DIR&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Dump databases before syncing&lt;/span&gt;
pg_dump&lt;span class="w"&gt; &lt;/span&gt;-U&lt;span class="w"&gt; &lt;/span&gt;postgres&lt;span class="w"&gt; &lt;/span&gt;myapp&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BACKUP_DIR&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/myapp.sql&amp;quot;&lt;/span&gt;
&lt;span class="c1"&gt;# Or for MySQL:&lt;/span&gt;
&lt;span class="c1"&gt;# mysqldump --single-transaction --all-databases &amp;gt; &amp;quot;${BACKUP_DIR}/all-databases.sql&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Export container volumes (Podman or Docker)&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;volume&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;export&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;myapp-data&lt;span class="w"&gt; &lt;/span&gt;&amp;gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;BACKUP_DIR&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/myapp-data.tar&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Sync everything to rsync.net&lt;/span&gt;
rsync&lt;span class="w"&gt; &lt;/span&gt;-avz&lt;span class="w"&gt; &lt;/span&gt;--delete&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh -i &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/etc/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;RSYNC_TARGET&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/etc/&amp;quot;&lt;/span&gt;

rsync&lt;span class="w"&gt; &lt;/span&gt;-avz&lt;span class="w"&gt; &lt;/span&gt;--delete&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh -i &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/var/backups/pre-rsync/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;RSYNC_TARGET&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/dumps/&amp;quot;&lt;/span&gt;

rsync&lt;span class="w"&gt; &lt;/span&gt;-avz&lt;span class="w"&gt; &lt;/span&gt;--delete&lt;span class="w"&gt; &lt;/span&gt;-e&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ssh -i &lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;SSH_KEY&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;/opt/app-data/&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;RSYNC_TARGET&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;/app-data/&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# If we got here, everything succeeded (set -e exits on any failure)&lt;/span&gt;
curl&lt;span class="w"&gt; &lt;/span&gt;-fsS&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;&lt;span class="si"&gt;${&lt;/span&gt;&lt;span class="nv"&gt;MONITOR_URL&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The database dump ensures you get a consistent, point-in-time export rather than a half-written data&amp;nbsp;directory. &lt;code&gt;podman volume export&lt;/code&gt; gives you a clean tarball of container volumes - useful when your application runs in containers and the actual state lives in named volumes rather than bind&amp;nbsp;mounts.&lt;/p&gt;
&lt;p&gt;For containers specifically, don&amp;#8217;t forget that the compose file or quadlet unit and any environment files are just as important as the volume data. Losing your database is bad; losing your database &lt;em&gt;and&lt;/em&gt; the configuration that tells you how it was deployed is&amp;nbsp;worse.&lt;/p&gt;
&lt;p&gt;The rsync.net destination is just a directory on a &lt;span class="caps"&gt;ZFS&lt;/span&gt; filesystem, so you get the benefit of rsync.net&amp;#8217;s own &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots on the remote side. Even&amp;nbsp;if &lt;code&gt;--delete&lt;/code&gt; removes a file during sync, the previous version survives in a snapshot. It&amp;#8217;s belt and suspenders, and it costs nothing&amp;nbsp;extra.&lt;/p&gt;
&lt;p&gt;Boring? Yes. Reliable? Also yes. Sometimes the best backup tool is the one that&amp;#8217;s been working since&amp;nbsp;1996.&lt;/p&gt;
&lt;h3 id="option-c-the-podman-trick-when-packages-dont-exist"&gt;Option C: The Podman Trick (When Packages Don&amp;#8217;t&amp;nbsp;Exist)&lt;/h3&gt;
&lt;p&gt;And then there&amp;#8217;s the creative&amp;nbsp;part.&lt;/p&gt;
&lt;p&gt;I have a &lt;span class="caps"&gt;RHEL&lt;/span&gt; 9 server running Forgejo (a self-hosted Git forge) in Podman containers. This is a production system with SELinux in enforcing mode. I wanted to back it up to my existing &lt;span class="caps"&gt;PBS&lt;/span&gt; infrastructure, but here&amp;#8217;s the&amp;nbsp;problem: &lt;code&gt;proxmox-backup-client&lt;/code&gt; doesn&amp;#8217;t ship as an &lt;span class="caps"&gt;RPM&lt;/span&gt; for &lt;span class="caps"&gt;RHEL&lt;/span&gt;. There&amp;#8217;s no package in &lt;span class="caps"&gt;EPEL&lt;/span&gt;, no Copr repo, nothing in the Proxmox repositories for Red Hat-based&amp;nbsp;systems.&lt;/p&gt;
&lt;p&gt;The solution?&amp;nbsp;Run &lt;code&gt;proxmox-backup-client&lt;/code&gt; &lt;em&gt;inside&lt;/em&gt; a Podman container, and pass the entire root filesystem in&amp;nbsp;read-only:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;--rm&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;pbs-backup-job&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--security-opt&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;type:spc_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--hostname&lt;span class="w"&gt; &lt;/span&gt;hermes&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--entrypoint&lt;span class="w"&gt; &lt;/span&gt;/usr/bin/proxmox-backup-client&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--secret&lt;span class="w"&gt; &lt;/span&gt;pbs_repo_password,type&lt;span class="o"&gt;=&lt;/span&gt;env,target&lt;span class="o"&gt;=&lt;/span&gt;PBS_PASSWORD&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--secret&lt;span class="w"&gt; &lt;/span&gt;pbs_key_passphrase,type&lt;span class="o"&gt;=&lt;/span&gt;env,target&lt;span class="o"&gt;=&lt;/span&gt;PBS_ENCRYPTION_PASSWORD&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/:/mnt/root:ro&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/root/backup.key:/etc/proxmox/backup.key:ro,Z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;docker.io/ayufan/proxmox-backup-server:v3.4.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;backup&lt;span class="w"&gt; &lt;/span&gt;root.pxar:/mnt/root&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--repository&lt;span class="w"&gt; &lt;/span&gt;backup@pbs@backup-vps.example.com:data&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;--keyfile&lt;span class="w"&gt; &lt;/span&gt;/etc/proxmox/backup.key
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Let&amp;#8217;s break down why this actually works&amp;nbsp;well:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;--security-opt label=type:spc_t&lt;/code&gt;&lt;/strong&gt; is the key ingredient on a SELinux system.&amp;nbsp;The &lt;code&gt;spc_t&lt;/code&gt; (super privileged container) type allows the container to read host files that would normally be blocked by SELinux labels. Without this, the container would get a wall of &amp;#8220;permission denied&amp;#8221; errors trying to read the host filesystem, even though it&amp;#8217;s mounted read-only. If you&amp;#8217;re running SELinux in enforcing mode (and &lt;a href="https://hofstede.it/selinux-a-practical-guide-for-fedora-and-rhel/"&gt;you should be&lt;/a&gt;), this flag is&amp;nbsp;non-negotiable.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;-v /:/mnt/root:ro&lt;/code&gt;&lt;/strong&gt; mounts the entire host root filesystem into the container as read-only. The container can see everything but modify nothing. The backup client reads files&amp;nbsp;from &lt;code&gt;/mnt/root&lt;/code&gt; and creates&amp;nbsp;a &lt;code&gt;root.pxar&lt;/code&gt; archive (Proxmox&amp;#8217;s archive format) that gets sent to &lt;span class="caps"&gt;PBS&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Podman secrets&lt;/strong&gt; handle the credentials. The &lt;span class="caps"&gt;PBS&lt;/span&gt; repository password and encryption key passphrase are stored as Podman secrets and injected as environment variables - they never appear in the command line, process list, or container inspect&amp;nbsp;output.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;--hostname hermes&lt;/code&gt;&lt;/strong&gt; sets the hostname inside the container to match the actual host. &lt;span class="caps"&gt;PBS&lt;/span&gt; uses the hostname to identify backup sources, so this ensures the backups show up correctly in the &lt;span class="caps"&gt;PBS&lt;/span&gt; &lt;span class="caps"&gt;UI&lt;/span&gt;.&lt;/p&gt;
&lt;p&gt;The result: a full file-level backup of a &lt;span class="caps"&gt;RHEL&lt;/span&gt; server to Proxmox Backup Server, with encryption, deduplication, and incremental transfers - all without installing a single package on the host. It runs nightly via a systemd timer, and it has been&amp;nbsp;rock-solid.&lt;/p&gt;
&lt;p&gt;This is absolutely a workaround, not a vendor-supported path. But it&amp;#8217;s contained, repeatable, and has proven stable in practice. Sometimes the ugliest solutions are the most reliable&amp;nbsp;ones.&lt;/p&gt;
&lt;h2 id="the-glue-dead-mans-switch-monitoring"&gt;The Glue: Dead Man&amp;#8217;s Switch&amp;nbsp;Monitoring&lt;/h2&gt;
&lt;p&gt;The most dangerous failure mode for backups isn&amp;#8217;t failure. It&amp;#8217;s &lt;em&gt;silent&lt;/em&gt; failure. The backup that stopped working three months ago and nobody noticed until the restore attempt. By the time you need it, you discover that your last good backup is from February and it&amp;#8217;s now&amp;nbsp;April.&lt;/p&gt;
&lt;p&gt;This is why every backup script ends with a monitoring ping. I run an &lt;a href="https://github.com/louislam/uptime-kuma"&gt;Uptime Kuma&lt;/a&gt; instance that monitors these pings via &lt;strong&gt;push-based&lt;/strong&gt; checks (sometimes called dead man&amp;#8217;s switches or heartbeat monitors). The logic is inverted from normal monitoring: instead of checking whether a service is &lt;em&gt;up&lt;/em&gt;, it checks whether a backup has &lt;em&gt;reported in&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Each backup job pings a unique &lt;span class="caps"&gt;URL&lt;/span&gt; on&amp;nbsp;success:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;curl&lt;span class="w"&gt; &lt;/span&gt;-fsS&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;https://uptime.example.com/api/push/xxxxxxxxxx?status=up&amp;amp;msg=OK&amp;amp;ping=&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If the backup fails, the curl never fires. If the backup script doesn&amp;#8217;t run at all (cron misconfigured, server down, disk full, whatever), the curl never fires. Uptime Kuma is configured with a 28-hour timeout (24 hours for the daily schedule + 4 hours of tolerance). If 28 hours pass without a ping, I get an&amp;nbsp;alert.&lt;/p&gt;
&lt;p&gt;This catches every failure mode:
- Backup script crashes → no ping → alert
- Cron job stops running → no ping → alert
- Server goes offline → no ping → alert
- &lt;span class="caps"&gt;SSH&lt;/span&gt; key expires → backup fails → no ping → alert
- Remote target is full → backup fails → no ping →&amp;nbsp;alert&lt;/p&gt;
&lt;p&gt;And yes - who monitors the monitor? Uptime Kuma itself is monitored by an external service (a simple &lt;span class="caps"&gt;HTTP&lt;/span&gt; check), so if it goes down, I know about that&amp;nbsp;too.&lt;/p&gt;
&lt;p&gt;The only thing push monitoring doesn&amp;#8217;t catch is a backup that &lt;em&gt;appears&lt;/em&gt; to succeed but produces corrupt or incomplete data. For that, you need test&amp;nbsp;restores.&lt;/p&gt;
&lt;h2 id="quarterly-restore-tests"&gt;Quarterly Restore&amp;nbsp;Tests&lt;/h2&gt;
&lt;p&gt;A mentor told me at one of my first jobs: &amp;#8220;Untested backups are just expensive hopes and dreams.&amp;#8221; That stuck with me, and I take it&amp;nbsp;seriously.&lt;/p&gt;
&lt;p&gt;Every quarter, I run full restore tests for every machine I consider production. These are scheduled in my calendar as recurring appointments, and I treat them with the same priority as any other production maintenance window. The process is straightforward: spin up a fresh &lt;span class="caps"&gt;VM&lt;/span&gt;, restore the system entirely from backups, and verify that it comes up in a functionally working state - services running, data intact, applications&amp;nbsp;responding.&lt;/p&gt;
&lt;p&gt;I record every restore session with &lt;a href="https://asciinema.org/"&gt;asciinema&lt;/a&gt; and document the results in my DokuWiki, with the terminal recording attached. This gives me three things at once: confidence that the backups actually work, regular practice with the restore procedure so I&amp;#8217;m not fumbling through it for the first time during an actual emergency, and a living library of documentation I can refer to when things go wrong at 3 &lt;span class="caps"&gt;AM&lt;/span&gt;. Under pressure is the worst possible time to be reading man pages and guessing at flags. Having a recording of yourself successfully restoring the exact same system last quarter is worth more than any&amp;nbsp;runbook.&lt;/p&gt;
&lt;p&gt;The quarterly cadence is a compromise. Monthly would be better but hard to justify the time. Yearly would be negligent. Quarterly is frequent enough that procedural drift doesn&amp;#8217;t accumulate, and any backup issue gets caught within 90 days at&amp;nbsp;most.&lt;/p&gt;
&lt;h3 id="how-restores-actually-work"&gt;How Restores Actually&amp;nbsp;Work&lt;/h3&gt;
&lt;p&gt;Since each layer uses different tools, the restore path differs per layer. A quick&amp;nbsp;overview:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; local snapshots&lt;/strong&gt;&amp;nbsp;→ &lt;code&gt;zfs rollback&lt;/code&gt; to revert a dataset in place,&amp;nbsp;or &lt;code&gt;zfs clone&lt;/code&gt; to mount a snapshot as a separate dataset for selective file recovery without affecting the live&amp;nbsp;system&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Syncoid / rsync.net&lt;/strong&gt;&amp;nbsp;→ &lt;code&gt;zfs send&lt;/code&gt; from the remote pool back to the local host via &lt;span class="caps"&gt;SSH&lt;/span&gt;. Same tool, reverse direction. For a full rebuild: install the &lt;span class="caps"&gt;OS&lt;/span&gt;, create the&amp;nbsp;pool, &lt;code&gt;zfs recv&lt;/code&gt; the&amp;nbsp;datasets&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;PBS&lt;/span&gt; (Proxmox Backup Server)&lt;/strong&gt; → restore via the &lt;span class="caps"&gt;PBS&lt;/span&gt; web &lt;span class="caps"&gt;UI&lt;/span&gt;&amp;nbsp;or &lt;code&gt;proxmox-backup-client restore&lt;/code&gt; on the &lt;span class="caps"&gt;CLI&lt;/span&gt;. For VMs on Proxmox &lt;span class="caps"&gt;VE&lt;/span&gt;, the restore is a one-click operation from the &lt;span class="caps"&gt;PVE&lt;/span&gt;&amp;nbsp;interface&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Plain rsync&lt;/strong&gt; → copy the dumps and files back, re-import database dumps&amp;nbsp;with &lt;code&gt;psql&lt;/code&gt; or &lt;code&gt;mysql&lt;/code&gt;, re-create container volumes from the tarballs. The most manual path, but also the most&amp;nbsp;transparent&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This is exactly what I walk through during the quarterly restore tests. Each layer&amp;#8217;s restore procedure is documented and recorded, so I&amp;#8217;m not figuring it out under&amp;nbsp;pressure.&lt;/p&gt;
&lt;h2 id="putting-it-all-together"&gt;Putting It All&amp;nbsp;Together&lt;/h2&gt;
&lt;p&gt;Here&amp;#8217;s the full&amp;nbsp;picture:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;┌─────────────────────────────────────────────────────────────┐
│                     FreeBSD Servers (~10)                   │
│                                                             │
│  sanoid (local snapshots) ──► syncoid ──► rsync.net (ZFS)   │
│          daily/weekly              nightly    5 TB pool     │
│          retention                 encrypted  remote prune  │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                  Proxmox VE Nodes (2)                       │
│                                                             │
│  PVE backup schedule ──► PBS VM ──► Backblaze B2 (S3)       │
│       nightly              1C/1G    deduplicated            │
│       incremental          proxy    encrypted               │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                     Linux Servers                           │
│                                                             │
│  proxmox-backup-client ──► PBS ──► Backblaze B2             │
│  rsync over SSH ──► rsync.net                               │
│  podman + pbs-client ──► PBS  (RHEL/no native package)      │
└─────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────┐
│                      Monitoring                             │
│                                                             │
│  Every backup job ──► Uptime Kuma (push / dead man switch)  │
│                        28h timeout ──► alert on silence     │
└─────────────────────────────────────────────────────────────┘
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="what-this-costs"&gt;What This&amp;nbsp;Costs&lt;/h2&gt;
&lt;p&gt;One of the most common excuses for not having proper backups is cost. So let&amp;#8217;s put actual numbers on&amp;nbsp;this:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Component&lt;/th&gt;
&lt;th&gt;What It Does&lt;/th&gt;
&lt;th&gt;Cost&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Scaleway &lt;span class="caps"&gt;STARDUST1&lt;/span&gt;-S (IPv6, 1 &lt;span class="caps"&gt;GB&lt;/span&gt; &lt;span class="caps"&gt;RAM&lt;/span&gt;, 1 vCPU, 10 &lt;span class="caps"&gt;GB&lt;/span&gt; disk)&lt;/td&gt;
&lt;td&gt;Runs Proxmox Backup Server&lt;/td&gt;
&lt;td&gt;€0.10/month&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Backblaze B2 object storage&lt;/td&gt;
&lt;td&gt;Stores &lt;span class="caps"&gt;VM&lt;/span&gt; backups from &lt;span class="caps"&gt;PBS&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;$6/&lt;span class="caps"&gt;TB&lt;/span&gt;/month&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;rsync.net &lt;span class="caps"&gt;ZFS&lt;/span&gt; account (5 &lt;span class="caps"&gt;TB&lt;/span&gt;)&lt;/td&gt;
&lt;td&gt;Off-site &lt;span class="caps"&gt;ZFS&lt;/span&gt; replication target for all FreeBSD servers&lt;/td&gt;
&lt;td&gt;$60/month&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The Scaleway instance is almost comically cheap. It&amp;#8217;s a &lt;span class="caps"&gt;STARDUST1&lt;/span&gt;-S - the smallest instance they offer - and it&amp;#8217;s more than enough for &lt;span class="caps"&gt;PBS&lt;/span&gt;, because &lt;span class="caps"&gt;PBS&lt;/span&gt; is doing deduplication and proxying to S3, not storing data locally. Ten cents a month. That&amp;#8217;s not a&amp;nbsp;typo.&lt;/p&gt;
&lt;p&gt;Backblaze B2 scales linearly with what you store. For a lab environment with a few hundred gigabytes of &lt;span class="caps"&gt;VM&lt;/span&gt; data after deduplication, you&amp;#8217;re looking at low single digits per month. Egress is free if you route through Cloudflare&amp;#8217;s Bandwidth Alliance, which matters a lot if you ever actually need to&amp;nbsp;restore.&lt;/p&gt;
&lt;p&gt;The rsync.net line item looks expensive at first glance - $60/month for 5 &lt;span class="caps"&gt;TB&lt;/span&gt;. But you&amp;#8217;re not just buying storage. You get a fully managed FreeBSD virtual host with root access, sitting on top of a tolerant raidz3 pool with &lt;span class="caps"&gt;ZFS&lt;/span&gt; snapshots. rsync.net handles all hardware maintenance, &lt;span class="caps"&gt;OS&lt;/span&gt; patches, upgrades, and provides unlimited support. You can optionally add geo-redundancy. For a managed service with that feature set and reliability, it&amp;#8217;s genuinely good value. Try pricing a comparable setup at any other provider where you get a dedicated &lt;span class="caps"&gt;ZFS&lt;/span&gt; pool that&amp;nbsp;accepts &lt;code&gt;zfs recv&lt;/code&gt; natively - the options are slim, and they aren&amp;#8217;t&amp;nbsp;cheaper.&lt;/p&gt;
&lt;p&gt;All in, the entire backup infrastructure for about a dozen servers costs roughly &lt;strong&gt;$70/month&lt;/strong&gt;. That&amp;#8217;s less than one hour of emergency consulting when you lose data you can&amp;#8217;t&amp;nbsp;recover.&lt;/p&gt;
&lt;p&gt;Is it perfect? No. There are things I&amp;#8217;d improve: better alerting on partial failures, maybe a dashboard that shows backup age across all hosts at a glance, and more automation around the restore tests. But the fundamentals are solid: every server has local snapshots for fast rollback, every server has off-site copies for disaster recovery, every backup job is monitored, restores are tested quarterly, and the whole thing runs&amp;nbsp;unattended.&lt;/p&gt;
&lt;p&gt;The most important property of a backup strategy isn&amp;#8217;t that it&amp;#8217;s clever. It&amp;#8217;s that it&amp;#8217;s boring, consistent, and actually running. Fancy architectures that you never finish implementing protect exactly zero bytes of data. A simple cron job that runs every night and phones home when it&amp;#8217;s done? That protects&amp;nbsp;everything.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/jimsalterjrs/sanoid"&gt;Sanoid/Syncoid - Policy-driven snapshot&amp;nbsp;management&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://rsync.net/products/zfsintro.html"&gt;rsync.net - &lt;span class="caps"&gt;ZFS&lt;/span&gt;-capable off-site&amp;nbsp;storage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pbs.proxmox.com/docs/"&gt;Proxmox Backup Server&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.backblaze.com/cloud-storage"&gt;Backblaze B2 - S3-compatible object&amp;nbsp;storage&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://github.com/louislam/uptime-kuma"&gt;Uptime Kuma - Self-hosted&amp;nbsp;monitoring&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hub.docker.com/r/ayufan/proxmox-backup-server"&gt;ayufan/proxmox-backup-server container&amp;nbsp;image&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="backup"/><category term="zfs"/><category term="freebsd"/><category term="linux"/><category term="proxmox"/><category term="sanoid"/><category term="syncoid"/><category term="rsync"/><category term="podman"/><category term="infrastructure"/></entry><entry><title>Why I Prefer CentOS Stream Over Old CentOS</title><link href="https://hofstede.it/why-i-prefer-centos-stream-over-old-centos/" rel="alternate"/><published>2026-03-15T00:00:00+01:00</published><updated>2026-03-15T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-03-15:/why-i-prefer-centos-stream-over-old-centos/</id><summary type="html">&lt;p&gt;Old CentOS rebuilt &lt;span class="caps"&gt;RHEL&lt;/span&gt; faithfully, but its downstream position meant it could only follow, never contribute back. CentOS Stream changes that. Sitting upstream of &lt;span class="caps"&gt;RHEL&lt;/span&gt; and downstream of Fedora, it combines enterprise-grade stability with a genuine feedback loop into &lt;span class="caps"&gt;RHEL&lt;/span&gt; development. After years of running it in production, I&amp;#8217;m convinced it&amp;#8217;s the better&amp;nbsp;model.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Logo" src="https://hofstede.it/images/2026-03-15-centos-stream-better-than-old-centos.png" title="CentOS Stream"&gt;&lt;/p&gt;
&lt;p&gt;When Red Hat announced in December 2020 that CentOS Linux would shift to CentOS Stream, the community response was swift and polarized. Many users who had built infrastructure around CentOS&amp;#8217;s bug-for-bug &lt;span class="caps"&gt;RHEL&lt;/span&gt; compatibility were understandably concerned about what this change meant for their&amp;nbsp;workflows.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;Disclosure: I work for Red Hat, but the opinions here are my own, based on running my personal&amp;nbsp;infrastructure.&lt;/em&gt;&lt;/p&gt;
&lt;p&gt;Several years later, it&amp;#8217;s worth looking at what CentOS Stream actually delivers - and why I think it&amp;#8217;s a stronger model than what came&amp;nbsp;before.&lt;/p&gt;
&lt;h2 id="what-old-centos-actually-was"&gt;What Old CentOS Actually&amp;nbsp;Was&lt;/h2&gt;
&lt;p&gt;CentOS Linux was a community-driven rebuild of Red Hat Enterprise Linux, stripped of branding and trademarks. Every binary, every library version, every kernel patch was matched as closely as possible to &lt;span class="caps"&gt;RHEL&lt;/span&gt;, bug for bug. It was a significant engineering effort, and the project served millions of users&amp;nbsp;well.&lt;/p&gt;
&lt;p&gt;And that&amp;#8217;s exactly where its limitations&amp;nbsp;lived.&lt;/p&gt;
&lt;p&gt;But by design, old CentOS couldn&amp;#8217;t diverge from &lt;span class="caps"&gt;RHEL&lt;/span&gt;. It couldn&amp;#8217;t ship a bug fix before &lt;span class="caps"&gt;RHEL&lt;/span&gt; did, or adopt a newer library version independently, because any divergence would break the compatibility guarantee users depended on. The project waited for each &lt;span class="caps"&gt;RHEL&lt;/span&gt; release, rebuilt it, and shipped - with no mechanism to feed improvements back&amp;nbsp;upstream.&lt;/p&gt;
&lt;p&gt;That model delivered real value. But it also meant CentOS could only ever&amp;nbsp;follow.&lt;/p&gt;
&lt;h2 id="what-centos-stream-actually-is"&gt;What CentOS Stream Actually&amp;nbsp;Is&lt;/h2&gt;
&lt;p&gt;CentOS Stream sits between Fedora and &lt;span class="caps"&gt;RHEL&lt;/span&gt; in the development&amp;nbsp;pipeline:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Fedora   -&amp;gt;     CentOS Stream  -&amp;gt;   RHEL
  (upstream)      (staging)     (stable release)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Fedora is where new packages, features, and technologies land first. It moves fast, breaks things occasionally, and serves as the proving ground for the broader Red Hat ecosystem. &lt;span class="caps"&gt;RHEL&lt;/span&gt; is the polished, commercially supported product - stable, certified, and conservative by&amp;nbsp;design.&lt;/p&gt;
&lt;p&gt;CentOS Stream occupies the middle ground. It&amp;#8217;s the continuous delivery branch where the &lt;em&gt;next&lt;/em&gt; minor release of &lt;span class="caps"&gt;RHEL&lt;/span&gt; is developed. When Red Hat prepares an upcoming release like &lt;span class="caps"&gt;RHEL&lt;/span&gt; 9.9 or &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10.2, the changes flow through the corresponding CentOS Stream branch first. By the time those changes reach &lt;span class="caps"&gt;RHEL&lt;/span&gt;, they&amp;#8217;ve been tested, validated, and refined in&amp;nbsp;Stream.&lt;/p&gt;
&lt;p&gt;This is fundamentally different from old CentOS, which sat &lt;em&gt;downstream&lt;/em&gt; of &lt;span class="caps"&gt;RHEL&lt;/span&gt; - receiving changes only after they&amp;#8217;d been released. Stream flips that relationship&amp;nbsp;entirely.&lt;/p&gt;
&lt;h2 id="why-this-is-better"&gt;Why This Is&amp;nbsp;Better&lt;/h2&gt;
&lt;h3 id="you-see-whats-coming"&gt;You See What&amp;#8217;s&amp;nbsp;Coming&lt;/h3&gt;
&lt;p&gt;With old CentOS, updates arrived as a fait accompli. &lt;span class="caps"&gt;RHEL&lt;/span&gt; released a point release, CentOS rebuilt it, and you got whatever Red Hat decided you should have. No preview, no preparation, no&amp;nbsp;input.&lt;/p&gt;
&lt;p&gt;CentOS Stream lets you look slightly ahead into what&amp;#8217;s coming to &lt;span class="caps"&gt;RHEL&lt;/span&gt;. The packages in Stream today are the packages that will ship in the next &lt;span class="caps"&gt;RHEL&lt;/span&gt; minor release. If you&amp;#8217;re running &lt;span class="caps"&gt;RHEL&lt;/span&gt; in production alongside Stream in development or staging, you get advance visibility into every change heading your way. That&amp;#8217;s not instability - that&amp;#8217;s&amp;nbsp;intelligence.&lt;/p&gt;
&lt;h3 id="the-stability-is-real"&gt;The Stability Is&amp;nbsp;Real&lt;/h3&gt;
&lt;p&gt;This is the part that surprises people most. CentOS Stream is not Fedora. It&amp;#8217;s not a bleeding-edge rolling release. It&amp;#8217;s not&amp;nbsp;experimental.&lt;/p&gt;
&lt;p&gt;CentOS Stream contains well-tested software. Most components entering Stream have already matured through Fedora or internal &lt;span class="caps"&gt;RHEL&lt;/span&gt; development processes. Stream reflects the current development branch for the next &lt;span class="caps"&gt;RHEL&lt;/span&gt; minor release, staged for final&amp;nbsp;validation.&lt;/p&gt;
&lt;p&gt;In my personal experience running CentOS Stream since Stream 8 was released in September 2019, the stability has been remarkably good. For the types of workloads I run, I haven&amp;#8217;t encountered stability issues attributable to Stream&amp;#8217;s position in the pipeline. Stream stages already-proven changes, leaving the comprehensive hardware certification and extended &lt;span class="caps"&gt;QA&lt;/span&gt; sweeps for the official &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;release.&lt;/p&gt;
&lt;p&gt;The &amp;#8220;development branch&amp;#8221; label scares people, but the reality doesn&amp;#8217;t match the fear. Stream isn&amp;#8217;t where experiments happen - that&amp;#8217;s Fedora&amp;#8217;s job. Stream is where already-proven changes are staged for &lt;span class="caps"&gt;RHEL&lt;/span&gt; release. The delta between Stream and the next &lt;span class="caps"&gt;RHEL&lt;/span&gt; point release is intentionally&amp;nbsp;small.&lt;/p&gt;
&lt;h3 id="a-genuine-feedback-loop"&gt;A Genuine Feedback&amp;nbsp;Loop&lt;/h3&gt;
&lt;p&gt;Because Stream feeds &lt;em&gt;into&lt;/em&gt; &lt;span class="caps"&gt;RHEL&lt;/span&gt; rather than copying &lt;em&gt;from&lt;/em&gt; it, community contributions can influence what ships in &lt;span class="caps"&gt;RHEL&lt;/span&gt;. Bug fixes can be proposed and tested in Stream before they reach the enterprise release. The distribution is an active participant in its own development rather than a downstream&amp;nbsp;consumer.&lt;/p&gt;
&lt;p&gt;Stream is still closely aligned with &lt;span class="caps"&gt;RHEL&lt;/span&gt; - the packages are &lt;span class="caps"&gt;RHEL&lt;/span&gt; packages, the kernel is the &lt;span class="caps"&gt;RHEL&lt;/span&gt; kernel, the ecosystem is the &lt;span class="caps"&gt;RHEL&lt;/span&gt; ecosystem - but there&amp;#8217;s room for the community to contribute meaningful improvements that benefit both Stream and &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;users.&lt;/p&gt;
&lt;h3 id="the-support-contract-argument"&gt;The Support Contract&amp;nbsp;Argument&lt;/h3&gt;
&lt;p&gt;The most common objection to CentOS Stream is the lack of a commercial support contract. But let&amp;#8217;s be clear: &lt;strong&gt;old CentOS didn&amp;#8217;t have a support contract either.&lt;/strong&gt; That was always the trade-off. You got &lt;span class="caps"&gt;RHEL&lt;/span&gt;-compatible binaries without paying for a subscription, but you also got them without commercial support, without certified hardware matrices, and without the ability to open a support case when something went&amp;nbsp;sideways.&lt;/p&gt;
&lt;p&gt;CentOS Stream has the same arrangement. If you need a support contract, you need &lt;span class="caps"&gt;RHEL&lt;/span&gt; - and that was true before Stream existed too. Nothing changed&amp;nbsp;here.&lt;/p&gt;
&lt;p&gt;What you &lt;em&gt;do&lt;/em&gt; get with Stream is access to the CentOS community, Red Hat&amp;#8217;s public issue trackers, and the ability to see and test changes before they hit your &lt;span class="caps"&gt;RHEL&lt;/span&gt; production systems. Arguably, that&amp;#8217;s more than old CentOS ever&amp;nbsp;offered.&lt;/p&gt;
&lt;h2 id="stream-as-a-server-platform"&gt;Stream as a Server&amp;nbsp;Platform&lt;/h2&gt;
&lt;p&gt;CentOS Stream is a valid choice for running production servers. Not a compromise, not a stopgap - a legitimate, well-engineered server operating&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;Consider what you&amp;#8217;re actually&amp;nbsp;getting:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt;-grade packages&lt;/strong&gt;: The software in Stream is built from the same sources as &lt;span class="caps"&gt;RHEL&lt;/span&gt;, with the same compilation flags, the same security policies, and the same enterprise-focused&amp;nbsp;defaults.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Continuous updates&lt;/strong&gt;: Rather than waiting for the next point release, you receive a steady flow of tested updates. Many fixes and improvements appear earlier than they did on old CentOS, which had to wait for &lt;span class="caps"&gt;RHEL&lt;/span&gt; to release them &lt;em&gt;and then&lt;/em&gt; rebuild. (Embargoed CVEs may reach &lt;span class="caps"&gt;RHEL&lt;/span&gt; subscribers first, but routine fixes flow through Stream without the old rebuild&amp;nbsp;lag.)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;SELinux, systemd, firewalld&lt;/strong&gt;: The full enterprise Linux stack, configured and tuned the same way &lt;span class="caps"&gt;RHEL&lt;/span&gt; configures&amp;nbsp;it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Container and virtualization support&lt;/strong&gt;: Podman, Buildah, and the container ecosystem work exactly as they do on &lt;span class="caps"&gt;RHEL&lt;/span&gt;.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;~5-year lifecycle&lt;/strong&gt;: CentOS Stream tracks the Full Support phase of its corresponding &lt;span class="caps"&gt;RHEL&lt;/span&gt; release - roughly five years. Stream 8 reached end of life in May 2024, and Stream 9 is expected to follow in 2027. That&amp;#8217;s shorter than &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s full 10-year lifecycle, but still excellent for a continuous delivery&amp;nbsp;platform.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;CentOS Stream is not a drop-in replacement for every workload that historically used CentOS Linux. Environments that require strict vendor certification, regulatory validation, or decade-long lifecycle guarantees will still be better served by &lt;span class="caps"&gt;RHEL&lt;/span&gt; itself. But for workloads that don&amp;#8217;t need those specific assurances, Stream provides a compelling platform - and because it tracks ahead of &lt;span class="caps"&gt;RHEL&lt;/span&gt; rather than behind, you&amp;#8217;re often running slightly newer&amp;nbsp;software.&lt;/p&gt;
&lt;h2 id="the-ecosystem-effect"&gt;The Ecosystem&amp;nbsp;Effect&lt;/h2&gt;
&lt;p&gt;CentOS Stream has become genuinely useful for the broader Linux ecosystem. ISVs can test against Stream to ensure compatibility with upcoming &lt;span class="caps"&gt;RHEL&lt;/span&gt; releases. Infrastructure teams can validate automation and deployment pipelines before &lt;span class="caps"&gt;RHEL&lt;/span&gt; updates land. Community members can contribute fixes that benefit both Stream and eventually &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;users.&lt;/p&gt;
&lt;p&gt;This collaborative model means that engineering effort goes toward improving the distribution rather than reproducing binaries. Contributions flow upstream into &lt;span class="caps"&gt;RHEL&lt;/span&gt;, creating a virtuous cycle that benefits the entire&amp;nbsp;ecosystem.&lt;/p&gt;
&lt;h2 id="the-bottom-line"&gt;The Bottom&amp;nbsp;Line&lt;/h2&gt;
&lt;p&gt;Downstream &lt;span class="caps"&gt;RHEL&lt;/span&gt; rebuilds still exist for users who want the old model, and they fill the role old CentOS occupied. But CentOS Stream offers something those rebuilds structurally cannot: an upstream role with a genuine feedback loop into &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;development.&lt;/p&gt;
&lt;p&gt;Old CentOS served its users well, but its downstream model limited it to following &lt;span class="caps"&gt;RHEL&lt;/span&gt;. CentOS Stream replaces that with a seat at the table - stable in practice, with the ability to contribute back. After years of running it in production, I&amp;#8217;m convinced it&amp;#8217;s the better&amp;nbsp;model.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://www.centos.org/centos-stream/"&gt;CentOS Stream&lt;/a&gt; - official project&amp;nbsp;page&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.redhat.com/en/topics/linux/what-is-centos-stream"&gt;Red Hat - CentOS Stream &lt;span class="caps"&gt;FAQ&lt;/span&gt;&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.centos.org/centos10/"&gt;CentOS Stream 10 Release&amp;nbsp;Notes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.fedoraproject.org/en-US/quick-docs/fedora-and-red-hat-enterprise-linux/"&gt;Fedora → CentOS Stream → &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;Pipeline&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="linux"/><category term="centos"/><category term="centos-stream"/><category term="rhel"/><category term="fedora"/><category term="server"/><category term="enterprise"/></entry><entry><title>Linux Firewalls: How to Actually Secure a Cloud Server (iptables, nftables, firewalld, ufw)</title><link href="https://hofstede.it/linux-firewalls-how-to-actually-secure-a-cloud-server-iptables-nftables-firewalld-ufw/" rel="alternate"/><published>2026-03-14T00:00:00+01:00</published><updated>2026-03-14T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-03-14:/linux-firewalls-how-to-actually-secure-a-cloud-server-iptables-nftables-firewalld-ufw/</id><summary type="html">&lt;p&gt;A practical guide to the four major Linux firewall technologies - iptables, nftables, firewalld, and ufw. Covers real-world cloud server hardening with concrete examples, from locking down &lt;span class="caps"&gt;SSH&lt;/span&gt; to building zone-based configurations. Includes an honest comparison and an entirely unbiased opinion about which firewall is actually&amp;nbsp;best.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Logo" src="https://hofstede.it/images/2026-03-14-linux-firewall-guide.png" title="Linux Firewall Technologies Guide"&gt;&lt;/p&gt;
&lt;p&gt;You&amp;#8217;ve just provisioned a fresh cloud server. It has a public IPv4 address, maybe an IPv6 range, and exactly zero protection between it and the internet&amp;#8217;s endless stream of &lt;span class="caps"&gt;SSH&lt;/span&gt; brute-force bots, port scanners, and whatever that traffic on port 5900 is. The clock is ticking - within minutes, your auth log will start filling up with connection attempts from &lt;span class="caps"&gt;IP&lt;/span&gt; addresses you&amp;#8217;ve never heard of in countries you&amp;#8217;ve never&amp;nbsp;visited.&lt;/p&gt;
&lt;p&gt;Linux gives you several ways to build a firewall between your server and this chaos. The problem isn&amp;#8217;t a lack of options - it&amp;#8217;s too many. iptables, nftables, firewalld, ufw - they all filter packets, but they approach the job differently, target different audiences, and impose different mental models. This guide walks through all four with real configurations you can actually deploy on a freshly provisioned cloud&amp;nbsp;server.&lt;/p&gt;
&lt;p&gt;&lt;em&gt;(And yes, I know - I&amp;#8217;ve spent most of this blog praising &lt;a href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; on FreeBSD&lt;/a&gt; as the pinnacle of firewall design. &lt;span class="caps"&gt;PF&lt;/span&gt;&amp;#8217;s syntax is cleaner, its stateful inspection is more elegant, and&amp;nbsp;writing &lt;code&gt;pf.conf&lt;/code&gt; genuinely brings me joy. But sometimes you&amp;#8217;re handed a Linux box and told to make it safe. So here we are, slumming it on the other side of the fence. Let&amp;#8217;s make the best of&amp;nbsp;it.)&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="the-landscape-how-we-got-here"&gt;The Landscape: How We Got&amp;nbsp;Here&lt;/h2&gt;
&lt;p&gt;Linux packet filtering has a history that mirrors Linux itself - organic, layered, and occasionally&amp;nbsp;contradictory.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;ipfwadm&lt;/strong&gt; (1994) was the first Linux firewall, basic and limited. &lt;strong&gt;ipchains&lt;/strong&gt; (1999, Linux 2.2) replaced it with chain-based filtering. &lt;strong&gt;iptables&lt;/strong&gt; (2001, Linux 2.4) brought stateful inspection and became the dominant firewall for two decades. &lt;strong&gt;nftables&lt;/strong&gt; (2014, Linux 3.13) was designed as iptables&amp;#8217; successor, with a unified framework and better performance. &lt;strong&gt;firewalld&lt;/strong&gt; and &lt;strong&gt;ufw&lt;/strong&gt; sit on top, providing higher-level&amp;nbsp;abstractions.&lt;/p&gt;
&lt;p&gt;The key thing to understand: all of these ultimately talk to &lt;strong&gt;Netfilter&lt;/strong&gt;, the packet filtering framework in the Linux kernel. iptables and nftables are different interfaces to the same underlying machinery. firewalld and ufw are frontends that generate iptables or nftables rules for you. The packets don&amp;#8217;t care which tool wrote the rules - they get filtered either&amp;nbsp;way.&lt;/p&gt;
&lt;h2 id="iptables-the-classic"&gt;iptables: The&amp;nbsp;Classic&lt;/h2&gt;
&lt;p&gt;iptables has been the standard Linux firewall since 2001. If you&amp;#8217;ve administered a Linux server in the last twenty years, you&amp;#8217;ve encountered it. Every tutorial, every Stack Overflow answer, every &amp;#8220;how to set up a &lt;span class="caps"&gt;VPS&lt;/span&gt;&amp;#8221; blog post assumes&amp;nbsp;iptables.&lt;/p&gt;
&lt;h3 id="the-mental-model"&gt;The Mental&amp;nbsp;Model&lt;/h3&gt;
&lt;p&gt;iptables organizes rules into &lt;strong&gt;tables&lt;/strong&gt; and &lt;strong&gt;chains&lt;/strong&gt;. The default table&amp;nbsp;(&lt;code&gt;filter&lt;/code&gt;) contains three built-in&amp;nbsp;chains:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;INPUT&lt;/span&gt;&lt;/strong&gt;: packets destined for this&amp;nbsp;machine&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;FORWARD&lt;/span&gt;&lt;/strong&gt;: packets being routed through this&amp;nbsp;machine&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;OUTPUT&lt;/span&gt;&lt;/strong&gt;: packets originating from this&amp;nbsp;machine&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Each chain has a &lt;strong&gt;policy&lt;/strong&gt; (default action) and a list of rules evaluated in order. The first matching rule&amp;nbsp;wins.&lt;/p&gt;
&lt;h3 id="securing-a-cloud-server-with-iptables"&gt;Securing a Cloud Server with&amp;nbsp;iptables&lt;/h3&gt;
&lt;p&gt;Here&amp;#8217;s a complete iptables configuration for a cloud server running a web application and &lt;span class="caps"&gt;SSH&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="ch"&gt;#!/bin/bash&lt;/span&gt;
&lt;span class="c1"&gt;# Cloud server firewall - iptables&lt;/span&gt;
&lt;span class="c1"&gt;# Flush existing rules&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-F
iptables&lt;span class="w"&gt; &lt;/span&gt;-X
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-F
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-X

&lt;span class="c1"&gt;# Default policies: drop everything, allow outbound&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;DROP
iptables&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;FORWARD&lt;span class="w"&gt; &lt;/span&gt;DROP
iptables&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;OUTPUT&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

ip6tables&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;DROP
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;FORWARD&lt;span class="w"&gt; &lt;/span&gt;DROP
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;OUTPUT&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

&lt;span class="c1"&gt;# Allow loopback&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;lo&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;lo&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

&lt;span class="c1"&gt;# Allow established and related connections&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;ESTABLISHED,RELATED&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;ESTABLISHED,RELATED&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

&lt;span class="c1"&gt;# Drop invalid packets&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;INVALID&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;DROP
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;INVALID&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;DROP

&lt;span class="c1"&gt;# SSH: rate-limited to slow down brute-force attempts&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;--dport&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;NEW&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;recent&lt;span class="w"&gt; &lt;/span&gt;--set&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;SSH
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;--dport&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;NEW&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;recent&lt;span class="w"&gt; &lt;/span&gt;--update&lt;span class="w"&gt; &lt;/span&gt;--seconds&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;60&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--hitcount&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;SSH&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;DROP
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;--dport&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;NEW&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;--dport&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;NEW&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;recent&lt;span class="w"&gt; &lt;/span&gt;--set&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;SSH
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;--dport&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;NEW&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;recent&lt;span class="w"&gt; &lt;/span&gt;--update&lt;span class="w"&gt; &lt;/span&gt;--seconds&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;60&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--hitcount&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;SSH&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;DROP
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;--dport&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;NEW&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

&lt;span class="c1"&gt;# HTTP and HTTPS&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;multiport&lt;span class="w"&gt; &lt;/span&gt;--dports&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;80&lt;/span&gt;,443&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;NEW&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;multiport&lt;span class="w"&gt; &lt;/span&gt;--dports&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;80&lt;/span&gt;,443&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;conntrack&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--ctstate&lt;span class="w"&gt; &lt;/span&gt;NEW&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

&lt;span class="c1"&gt;# ICMPv4: allow ping and necessary types&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;icmp&lt;span class="w"&gt; &lt;/span&gt;--icmp-type&lt;span class="w"&gt; &lt;/span&gt;echo-request&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;icmp&lt;span class="w"&gt; &lt;/span&gt;--icmp-type&lt;span class="w"&gt; &lt;/span&gt;destination-unreachable&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;icmp&lt;span class="w"&gt; &lt;/span&gt;--icmp-type&lt;span class="w"&gt; &lt;/span&gt;time-exceeded&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

&lt;span class="c1"&gt;# ICMPv6: essential for IPv6 to function&lt;/span&gt;
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;echo-request&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;echo-reply&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;destination-unreachable&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;packet-too-big&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;time-exceeded&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;parameter-problem&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;neighbour-solicitation&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;neighbour-advertisement&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;router-solicitation&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;ipv6-icmp&lt;span class="w"&gt; &lt;/span&gt;--icmpv6-type&lt;span class="w"&gt; &lt;/span&gt;router-advertisement&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;ACCEPT

&lt;span class="c1"&gt;# Log dropped packets (rate-limited to avoid log flooding)&lt;/span&gt;
iptables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;limit&lt;span class="w"&gt; &lt;/span&gt;--limit&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;/min&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;LOG&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--log-prefix&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;iptables-dropped: &amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--log-level&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;
ip6tables&lt;span class="w"&gt; &lt;/span&gt;-A&lt;span class="w"&gt; &lt;/span&gt;INPUT&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;limit&lt;span class="w"&gt; &lt;/span&gt;--limit&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;5&lt;/span&gt;/min&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;LOG&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--log-prefix&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;ip6tables-dropped: &amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--log-level&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;4&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="making-rules-persistent"&gt;Making Rules&amp;nbsp;Persistent&lt;/h3&gt;
&lt;p&gt;iptables rules live in kernel memory - they vanish on reboot. Making them persistent depends on your&amp;nbsp;distribution:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Debian/Ubuntu:&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;apt&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;iptables-persistent
netfilter-persistent&lt;span class="w"&gt; &lt;/span&gt;save
netfilter-persistent&lt;span class="w"&gt; &lt;/span&gt;reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt;/Fedora (if not using&amp;nbsp;firewalld):&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;iptables-services
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;iptables&lt;span class="w"&gt; &lt;/span&gt;ip6tables
service&lt;span class="w"&gt; &lt;/span&gt;iptables&lt;span class="w"&gt; &lt;/span&gt;save
service&lt;span class="w"&gt; &lt;/span&gt;ip6tables&lt;span class="w"&gt; &lt;/span&gt;save
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The rules are saved&amp;nbsp;to &lt;code&gt;/etc/iptables/rules.v4&lt;/code&gt; and &lt;code&gt;/etc/iptables/rules.v6&lt;/code&gt; (Debian)&amp;nbsp;or &lt;code&gt;/etc/sysconfig/iptables&lt;/code&gt; and &lt;code&gt;/etc/sysconfig/ip6tables&lt;/code&gt; (&lt;span class="caps"&gt;RHEL&lt;/span&gt;).&lt;/p&gt;
&lt;h3 id="the-good-and-the-ugly"&gt;The Good and the&amp;nbsp;Ugly&lt;/h3&gt;
&lt;p&gt;iptables works. It&amp;#8217;s battle-tested, universally available, and documented to the point of exhaustion. But the syntax is verbose - especially with dual-stack configurations where you need to duplicate everything for IPv4 and IPv6. There&amp;#8217;s no native concept of sets (you&amp;nbsp;need &lt;code&gt;ipset&lt;/code&gt; for that), and large rulesets become linear chains that the kernel evaluates rule by rule, which doesn&amp;#8217;t scale&amp;nbsp;elegantly.&lt;/p&gt;
&lt;p&gt;Technically, iptables is considered deprecated in favor of nftables. Modern kernels include&amp;nbsp;an &lt;code&gt;iptables-nft&lt;/code&gt; compatibility layer that translates iptables commands into nftables rules behind the scenes. On many current distributions, when you&amp;nbsp;run &lt;code&gt;iptables&lt;/code&gt;, you&amp;#8217;re actually&amp;nbsp;running &lt;code&gt;iptables-nft&lt;/code&gt; without knowing it. Check&amp;nbsp;with &lt;code&gt;iptables -V&lt;/code&gt; - if it&amp;nbsp;says &lt;code&gt;nf_tables&lt;/code&gt;, you&amp;#8217;re already on the&amp;nbsp;bridge.&lt;/p&gt;
&lt;h2 id="nftables-the-modern-replacement"&gt;nftables: The Modern&amp;nbsp;Replacement&lt;/h2&gt;
&lt;p&gt;nftables is iptables&amp;#8217; designated successor, included in the Linux kernel since 3.13. It addresses iptables&amp;#8217; design limitations: a single tool handles both IPv4 and IPv6, the rule language is more expressive, and the kernel representation uses a virtual machine that evaluates rulesets more&amp;nbsp;efficiently.&lt;/p&gt;
&lt;h3 id="the-mental-model_1"&gt;The Mental&amp;nbsp;Model&lt;/h3&gt;
&lt;p&gt;nftables replaces the fixed table/chain structure with a flexible one you define yourself. You create your own tables, your own chains, and configure their hook points. This sounds more complex, but it means you can organize rules however makes sense for your&amp;nbsp;infrastructure.&lt;/p&gt;
&lt;h3 id="the-same-cloud-server-in-nftables"&gt;The Same Cloud Server, in&amp;nbsp;nftables&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="p"&gt;!&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;usr&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;sbin&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;nft&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;f&lt;/span&gt;

&lt;span class="nx"&gt;flush&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ruleset&lt;/span&gt;

&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;firewall&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;for&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;tracking&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;SSH&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;brute&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;force&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;attempts&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ssh_meter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv4_addr&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dynamic&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ssh_meter6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv6_addr&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dynamic&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="nx"&gt;s&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;chain&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;filter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hook&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;priority&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Loopback&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;iif&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;lo&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Established&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;related&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;connections&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;ct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;established&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="nx"&gt;related&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Drop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;invalid&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;ct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;invalid&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;SSH&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;with&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;limiting&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nx"&gt;max&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;connections&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;per&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;60&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;seconds&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;The&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="nx"&gt;update&lt;/span&gt;&lt;span class="err"&gt;&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;statement&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dynamically&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;adds&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;source&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;address&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;set&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;checks&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;the&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;limit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;in&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;a&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;single&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;operation&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dport&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;update&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;ssh_meter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;saddr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;limit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;over&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;minute&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dport&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;update&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;ssh_meter6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ip6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;saddr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;limit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;over&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;4&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;minute&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dport&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;HTTP&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;HTTPS&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dport&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;80&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;443&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ct&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;state&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;new&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ICMP&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;and&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ICMPv6&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;ip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;protocol&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;echo&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;destination&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;unreachable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;time&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;exceeded&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;ip6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;nexthdr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv6&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;icmp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;icmpv6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;echo&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;request&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;echo&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;reply&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;destination&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;unreachable&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;packet&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;too&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;big&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;time&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;exceeded&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;parameter&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;problem&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;nd&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;neighbor&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;solicit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;nd&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;neighbor&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;advert&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;nd&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;router&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;solicit&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="nx"&gt;nd&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;router&lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="nx"&gt;advert&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dropped&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;packets&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;limit&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rate&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="o"&gt;/&lt;/span&gt;&lt;span class="nx"&gt;minute&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;log&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;prefix&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;nft-dropped: &amp;quot;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;level&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;warn&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;chain&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;forward&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;filter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hook&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;forward&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;priority&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;chain&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;output&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;filter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hook&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;output&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;priority&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Notice&amp;nbsp;the &lt;code&gt;inet&lt;/code&gt; family - a single table handles both IPv4 and IPv6. No more duplicating every rule. The set syntax for rate limiting is built into nftables natively, no&amp;nbsp;external &lt;code&gt;ipset&lt;/code&gt; needed.&lt;/p&gt;
&lt;h3 id="loading-and-persisting"&gt;Loading and&amp;nbsp;Persisting&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Validate syntax&lt;/span&gt;
nft&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/nftables.conf

&lt;span class="c1"&gt;# Load rules&lt;/span&gt;
nft&lt;span class="w"&gt; &lt;/span&gt;-f&lt;span class="w"&gt; &lt;/span&gt;/etc/nftables.conf

&lt;span class="c1"&gt;# Enable at boot&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;nftables
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The entire ruleset lives&amp;nbsp;in &lt;code&gt;/etc/nftables.conf&lt;/code&gt;. One file, one syntax, both address families. Compared to iptables&amp;#8217; separate save files for v4 and v6, this is a breath of fresh&amp;nbsp;air.&lt;/p&gt;
&lt;h3 id="named-sets-ip-allowlists-and-blocklists"&gt;Named Sets: &lt;span class="caps"&gt;IP&lt;/span&gt; Allowlists and&amp;nbsp;Blocklists&lt;/h3&gt;
&lt;p&gt;One of nftables&amp;#8217; strongest features is native support for sets - named collections of addresses, ports, or interfaces that can be referenced in rules and updated at&amp;nbsp;runtime:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nx"&gt;table&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;inet&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;firewall&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Trusted&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;management&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;IPs&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;trusted_admins&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv4_addr&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;elements&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m m-Double"&gt;198.51.100.22&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m m-Double"&gt;203.0.113.50&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;trusted_admins6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv6_addr&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;interval&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;elements&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;2001&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;db8&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="nx"&gt;ffff&lt;/span&gt;&lt;span class="o"&gt;::/&lt;/span&gt;&lt;span class="mi"&gt;48&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Blocklist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;-&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;populated&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dynamically&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;blocklist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ipv4_addr&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;flags&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;timeout&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;timeout&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;24&lt;/span&gt;&lt;span class="nx"&gt;h&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nx"&gt;chain&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;type&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;filter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;hook&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;input&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;priority&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;policy&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;Block&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;known&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;bad&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;actors&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;immediately&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;ip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;saddr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;blocklist&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;SSH&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;only&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;from&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;trusted&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;sources&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dport&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ip&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;saddr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;trusted_admins&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dport&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;ip6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;saddr&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="err"&gt;@&lt;/span&gt;&lt;span class="nx"&gt;trusted_admins6&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;accept&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nx"&gt;tcp&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;dport&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;drop&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="err"&gt;#&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;...&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rest&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nx"&gt;rules&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Manage sets at runtime without reloading the&amp;nbsp;ruleset:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Add to blocklist (auto-expires after 24h due to timeout flag)&lt;/span&gt;
nft&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;element&lt;span class="w"&gt; &lt;/span&gt;inet&lt;span class="w"&gt; &lt;/span&gt;firewall&lt;span class="w"&gt; &lt;/span&gt;blocklist&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.100&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Add with custom timeout&lt;/span&gt;
nft&lt;span class="w"&gt; &lt;/span&gt;add&lt;span class="w"&gt; &lt;/span&gt;element&lt;span class="w"&gt; &lt;/span&gt;inet&lt;span class="w"&gt; &lt;/span&gt;firewall&lt;span class="w"&gt; &lt;/span&gt;blocklist&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.101&lt;span class="w"&gt; &lt;/span&gt;timeout&lt;span class="w"&gt; &lt;/span&gt;1h&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# Remove from set&lt;/span&gt;
nft&lt;span class="w"&gt; &lt;/span&gt;delete&lt;span class="w"&gt; &lt;/span&gt;element&lt;span class="w"&gt; &lt;/span&gt;inet&lt;span class="w"&gt; &lt;/span&gt;firewall&lt;span class="w"&gt; &lt;/span&gt;blocklist&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.100&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;}&lt;/span&gt;

&lt;span class="c1"&gt;# List set contents&lt;/span&gt;
nft&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;set&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;inet&lt;span class="w"&gt; &lt;/span&gt;firewall&lt;span class="w"&gt; &lt;/span&gt;blocklist
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="why-nftables-over-iptables"&gt;Why nftables Over&amp;nbsp;iptables&lt;/h3&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Unified IPv4/IPv6&lt;/strong&gt;: One ruleset, one syntax, one mental&amp;nbsp;model&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Native sets&lt;/strong&gt;: No&amp;nbsp;external &lt;code&gt;ipset&lt;/code&gt; dependency&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Better performance&lt;/strong&gt;: Binary rules compiled into a virtual machine, not linear chain&amp;nbsp;matching&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Atomic rule replacement&lt;/strong&gt;: Load an entire new ruleset atomically - no window where rules are partially&amp;nbsp;loaded&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Programmable Netlink interface&lt;/strong&gt;: nftables exposes a Netlink socket&amp;nbsp;(&lt;code&gt;NETLINK_NETFILTER&lt;/code&gt;) that lets applications manage rules directly without shelling out&amp;nbsp;to &lt;code&gt;nft&lt;/code&gt;. Libraries like Google&amp;#8217;s &lt;a href="https://github.com/google/nftables"&gt;google/nftables&lt;/a&gt; for Go make this practical - you can build fully programmable firewall tooling, including things like injecting &lt;span class="caps"&gt;BPF&lt;/span&gt; into nftables rules. ngrok&amp;#8217;s team used this approach to build &lt;a href="https://github.com/ngrok/firewall_toolkit"&gt;firewall_toolkit&lt;/a&gt;, a host-level DDoS mitigation&amp;nbsp;library&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Cleaner syntax&lt;/strong&gt;: Subjective, but ask anyone who&amp;#8217;s written a 200-line iptables&amp;nbsp;script&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The learning curve is real if you&amp;#8217;re coming from iptables, but I&amp;#8217;ve found the transition is worth the effort. Once the nftables mental model clicks, going back to iptables feels like writing assembly after learning a proper&amp;nbsp;language.&lt;/p&gt;
&lt;h2 id="firewalld-the-enterprise-linux-firewall"&gt;firewalld: The Enterprise Linux&amp;nbsp;Firewall&lt;/h2&gt;
&lt;p&gt;firewalld is the default firewall manager on Fedora, &lt;span class="caps"&gt;RHEL&lt;/span&gt;, CentOS Stream, and their derivatives - which means it&amp;#8217;s also the firewall running on the majority of production Linux servers in enterprise environments. Rather than asking you to write packet filter rules, firewalld introduces &lt;strong&gt;zones&lt;/strong&gt; - named security contexts that you assign to network&amp;nbsp;interfaces.&lt;/p&gt;
&lt;p&gt;What sets firewalld apart from the other tools in this article is its architecture. It runs as a &lt;strong&gt;system daemon&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;firewalld.service&lt;/code&gt;) with a &lt;strong&gt;D-Bus &lt;span class="caps"&gt;API&lt;/span&gt;&lt;/strong&gt;, which means any authorized application - Cockpit, NetworkManager, libvirt, Ansible, or your own tooling - can query and modify firewall rules programmatically without parsing text files or shelling out to command-line tools. It also cleanly separates &lt;strong&gt;runtime&lt;/strong&gt; state (what&amp;#8217;s active right now) from &lt;strong&gt;permanent&lt;/strong&gt; configuration (what survives a reboot), giving you a built-in safety net for testing changes on remote&amp;nbsp;systems.&lt;/p&gt;
&lt;h3 id="the-mental-model_2"&gt;The Mental&amp;nbsp;Model&lt;/h3&gt;
&lt;p&gt;Zones represent trust levels. An interface in&amp;nbsp;the &lt;code&gt;public&lt;/code&gt; zone gets restrictive rules. An interface in&amp;nbsp;the &lt;code&gt;trusted&lt;/code&gt; zone allows everything. Instead of thinking about chains and rules, you think about &amp;#8220;this interface is in this zone, and this zone allows these&amp;nbsp;services.&amp;#8221;&lt;/p&gt;
&lt;p&gt;The predefined zones, from most restrictive to most&amp;nbsp;permissive:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Zone&lt;/th&gt;
&lt;th&gt;Default Behavior&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;drop&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Drop all incoming, no reply sent&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;block&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Reject all incoming with &lt;span class="caps"&gt;ICMP&lt;/span&gt; response&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;public&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Only selected services allowed (default zone)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;external&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Like public, with masquerading enabled&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;dmz&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Selected services, limited incoming&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;work&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Trust most networked machines&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;home&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Trust most networked machines&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;internal&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Trust most networked machines&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;trusted&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Accept everything&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="cloud-server-with-firewalld"&gt;Cloud Server with&amp;nbsp;firewalld&lt;/h3&gt;
&lt;p&gt;On &lt;span class="caps"&gt;RHEL&lt;/span&gt;/Fedora, firewalld is likely already running. Here&amp;#8217;s how to configure it for our cloud server&amp;nbsp;scenario:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check current state&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--state
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--get-active-zones

&lt;span class="c1"&gt;# Set the default zone&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--set-default-zone&lt;span class="o"&gt;=&lt;/span&gt;public

&lt;span class="c1"&gt;# Allow SSH (usually already enabled in public zone)&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;--permanent

&lt;span class="c1"&gt;# Allow HTTP and HTTPS&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;http&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;https&lt;span class="w"&gt; &lt;/span&gt;--permanent

&lt;span class="c1"&gt;# Reload to apply permanent changes&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--reload

&lt;span class="c1"&gt;# Verify&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--list-all
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Pro tip:&lt;/strong&gt; Instead of&amp;nbsp;adding &lt;code&gt;--permanent&lt;/code&gt; to every command and then reloading, you can test rules in &lt;em&gt;runtime&lt;/em&gt; first. Runtime rules are active immediately but disappear on reboot&amp;nbsp;(or &lt;code&gt;--reload&lt;/code&gt;). If you accidentally lock yourself out, a reboot reverts the damage. Once you&amp;#8217;ve confirmed everything works, save the running state in one&amp;nbsp;shot:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Test in runtime first (no --permanent flag)&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;http
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;https

&lt;span class="c1"&gt;# Everything working? Save the current runtime state as permanent&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--runtime-to-permanent
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This workflow is especially useful on remote servers where a typo in a permanent rule could mean a trip to the out-of-band&amp;nbsp;console.&lt;/p&gt;
&lt;p&gt;Output:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources:
  services: dhcpv6-client http https ssh
  ports:
  protocols:
  forward: yes
  masquerade: no
  forward-ports:
  source-ports:
  rich-rules:
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="service-definitions"&gt;Service&amp;nbsp;Definitions&lt;/h3&gt;
&lt;p&gt;firewalld uses &lt;span class="caps"&gt;XML&lt;/span&gt; service definitions&amp;nbsp;in &lt;code&gt;/usr/lib/firewalld/services/&lt;/code&gt;. Each service defines which ports and protocols it needs. When you add a &amp;#8220;service&amp;#8221; rather than a raw port, you&amp;#8217;re working at a higher abstraction&amp;nbsp;level:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# These are equivalent:&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;https&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-port&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;443&lt;/span&gt;/tcp&lt;span class="w"&gt; &lt;/span&gt;--permanent

&lt;span class="c1"&gt;# But the service approach is self-documenting and multi-port aware&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You can create custom services for your own&amp;nbsp;applications:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a custom service definition&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--permanent&lt;span class="w"&gt; &lt;/span&gt;--new-service&lt;span class="o"&gt;=&lt;/span&gt;myapp
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--permanent&lt;span class="w"&gt; &lt;/span&gt;--service&lt;span class="o"&gt;=&lt;/span&gt;myapp&lt;span class="w"&gt; &lt;/span&gt;--set-description&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;My Application&amp;quot;&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--permanent&lt;span class="w"&gt; &lt;/span&gt;--service&lt;span class="o"&gt;=&lt;/span&gt;myapp&lt;span class="w"&gt; &lt;/span&gt;--add-port&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;8080&lt;/span&gt;/tcp
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--permanent&lt;span class="w"&gt; &lt;/span&gt;--service&lt;span class="o"&gt;=&lt;/span&gt;myapp&lt;span class="w"&gt; &lt;/span&gt;--add-port&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;8443&lt;/span&gt;/tcp

&lt;span class="c1"&gt;# Use it&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;myapp&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="rich-rules-when-zones-arent-enough"&gt;Rich Rules: When Zones Aren&amp;#8217;t&amp;nbsp;Enough&lt;/h3&gt;
&lt;p&gt;For more granular control, firewalld supports &amp;#8220;rich rules&amp;#8221; - a more expressive syntax that sits between simple zone/service management and raw nftables&amp;nbsp;rules:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Allow SSH only from a trusted subnet&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-rich-rule&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
&lt;span class="s1"&gt;    rule family=&amp;quot;ipv4&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    source address=&amp;quot;198.51.100.0/24&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    service name=&amp;quot;ssh&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    accept&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--permanent

&lt;span class="c1"&gt;# Rate-limit SSH connections&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-rich-rule&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
&lt;span class="s1"&gt;    rule service name=&amp;quot;ssh&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    accept&lt;/span&gt;
&lt;span class="s1"&gt;    limit value=&amp;quot;4/m&amp;quot;&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--permanent

&lt;span class="c1"&gt;# Block a specific IP&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-rich-rule&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
&lt;span class="s1"&gt;    rule family=&amp;quot;ipv4&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    source address=&amp;quot;192.0.2.100&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    drop&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--permanent

&lt;span class="c1"&gt;# Log dropped packets from a specific source&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-rich-rule&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&lt;/span&gt;
&lt;span class="s1"&gt;    rule family=&amp;quot;ipv4&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    source address=&amp;quot;192.0.2.0/24&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    log prefix=&amp;quot;suspect-traffic: &amp;quot; level=&amp;quot;warning&amp;quot;&lt;/span&gt;
&lt;span class="s1"&gt;    drop&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--permanent

firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="multi-zone-network-segmentation"&gt;Multi-Zone Network&amp;nbsp;Segmentation&lt;/h3&gt;
&lt;p&gt;One of firewalld&amp;#8217;s most powerful enterprise features is the ability to assign different zones to different interfaces, creating proper network segmentation without touching nftables&amp;nbsp;directly:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Public-facing interface: restrictive&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--change-interface&lt;span class="o"&gt;=&lt;/span&gt;eth0&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;http&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;public&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;https&lt;span class="w"&gt; &lt;/span&gt;--permanent

&lt;span class="c1"&gt;# Internal management network: more permissive&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;internal&lt;span class="w"&gt; &lt;/span&gt;--change-interface&lt;span class="o"&gt;=&lt;/span&gt;eth1&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;internal&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;ssh&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;internal&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;cockpit&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;internal&lt;span class="w"&gt; &lt;/span&gt;--add-service&lt;span class="o"&gt;=&lt;/span&gt;dns&lt;span class="w"&gt; &lt;/span&gt;--permanent

&lt;span class="c1"&gt;# Database network: only database traffic&lt;/span&gt;
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;dmz&lt;span class="w"&gt; &lt;/span&gt;--change-interface&lt;span class="o"&gt;=&lt;/span&gt;eth2&lt;span class="w"&gt; &lt;/span&gt;--permanent
firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--zone&lt;span class="o"&gt;=&lt;/span&gt;dmz&lt;span class="w"&gt; &lt;/span&gt;--add-port&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="m"&gt;5432&lt;/span&gt;/tcp&lt;span class="w"&gt; &lt;/span&gt;--permanent

firewall-cmd&lt;span class="w"&gt; &lt;/span&gt;--reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This maps naturally to how enterprise networks are actually designed: distinct interfaces for distinct trust levels, each with its own policy. A single nftables ruleset can do the same thing, but you&amp;#8217;d be hand-coding the interface dispatch logic that firewalld provides out of the&amp;nbsp;box.&lt;/p&gt;
&lt;h3 id="firewallds-backend-nftables"&gt;firewalld&amp;#8217;s Backend:&amp;nbsp;nftables&lt;/h3&gt;
&lt;p&gt;Since firewalld 0.6.0, the default backend is nftables. firewalld translates its zone/service model into nftables rules. You can see the generated rules&amp;nbsp;with:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;nft&lt;span class="w"&gt; &lt;/span&gt;list&lt;span class="w"&gt; &lt;/span&gt;ruleset
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The output will be significantly more complex than hand-written nftables - firewalld generates a multi-table structure with dispatch chains for its zone model. This is the trade-off: you get a managed, auditable interface with a well-defined &lt;span class="caps"&gt;API&lt;/span&gt;, while the underlying nftables ruleset handles the heavy&amp;nbsp;lifting.&lt;/p&gt;
&lt;h3 id="when-firewalld-shines"&gt;When firewalld&amp;nbsp;Shines&lt;/h3&gt;
&lt;p&gt;firewalld is at its best in environments where firewall management isn&amp;#8217;t a one-person artisanal craft but a repeatable, auditable process across many&amp;nbsp;machines:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Cockpit integration&lt;/strong&gt;: The web-based Cockpit console (standard on &lt;span class="caps"&gt;RHEL&lt;/span&gt;) can manage firewalld zones, services, and ports through a graphical interface. For teams where not everyone is comfortable with &lt;span class="caps"&gt;CLI&lt;/span&gt; firewall management, this lowers the barrier without sacrificing&amp;nbsp;security.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ansible automation&lt;/strong&gt;:&amp;nbsp;The &lt;code&gt;ansible.posix.firewalld&lt;/code&gt; module maps directly to firewalld&amp;#8217;s zone/service model.&amp;nbsp;Declaring &lt;code&gt;service: https, zone: public, state: enabled, permanent: true&lt;/code&gt; in a playbook is about as clean as firewall configuration management&amp;nbsp;gets.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;D-Bus &lt;span class="caps"&gt;API&lt;/span&gt;&lt;/strong&gt;: Any application can query the current firewall state or request changes through D-Bus. libvirt uses this to dynamically open ports for &lt;span class="caps"&gt;VM&lt;/span&gt; networking. NetworkManager uses it to assign interfaces to zones when connections come up. This kind of runtime integration simply isn&amp;#8217;t possible with static ruleset&amp;nbsp;files.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Consistent across the ecosystem&lt;/strong&gt;: Whether you&amp;#8217;re managing a single &lt;span class="caps"&gt;RHEL&lt;/span&gt; server, a fleet of Fedora CoreOS nodes, or a CentOS Stream build farm, the firewall interface is identical. The same commands, the same zones, the same service&amp;nbsp;definitions.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Where firewalld is less ideal is when you need complete, low-level control over rule ordering or heavily customized chain structures. Rich rules and direct passthrough rules can handle most advanced cases, but if you&amp;#8217;re building something truly bespoke, nftables gives you more direct&amp;nbsp;control.&lt;/p&gt;
&lt;h2 id="ufw-uncomplicated-firewall"&gt;ufw: Uncomplicated&amp;nbsp;Firewall&lt;/h2&gt;
&lt;p&gt;ufw does exactly what its name promises: it makes firewall configuration uncomplicated. It&amp;#8217;s the default on Ubuntu and is popular on any Debian-based system where the administrator wants a firewall without studying Netfilter&amp;nbsp;internals.&lt;/p&gt;
&lt;h3 id="the-mental-model_3"&gt;The Mental&amp;nbsp;Model&lt;/h3&gt;
&lt;p&gt;ufw has essentially no mental model beyond &amp;#8220;allow or deny things.&amp;#8221; It&amp;#8217;s a command-line interface that generates iptables (or nftables) rules from simple, human-readable commands. It trades flexibility for&amp;nbsp;approachability.&lt;/p&gt;
&lt;h3 id="cloud-server-with-ufw"&gt;Cloud Server with&amp;nbsp;ufw&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Reset to clean state&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;reset

&lt;span class="c1"&gt;# Default policies&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;default&lt;span class="w"&gt; &lt;/span&gt;deny&lt;span class="w"&gt; &lt;/span&gt;incoming
ufw&lt;span class="w"&gt; &lt;/span&gt;default&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;outgoing

&lt;span class="c1"&gt;# Allow SSH (do this BEFORE enabling ufw, or you&amp;#39;ll lock yourself out)&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;ssh

&lt;span class="c1"&gt;# Allow HTTP and HTTPS&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;http
ufw&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;https

&lt;span class="c1"&gt;# Enable the firewall&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;

&lt;span class="c1"&gt;# Check status&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;verbose
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Output:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), disabled (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
22/tcp (v6)                ALLOW IN    Anywhere (v6)
80/tcp (v6)                ALLOW IN    Anywhere (v6)
443/tcp (v6)               ALLOW IN    Anywhere (v6)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That&amp;#8217;s it. Six commands and your server is firewalled. ufw automatically handles both IPv4 and&amp;nbsp;IPv6.&lt;/p&gt;
&lt;h3 id="slightly-more-advanced-ufw"&gt;Slightly More Advanced&amp;nbsp;ufw&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Allow SSH only from a specific subnet&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;from&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;198&lt;/span&gt;.51.100.0/24&lt;span class="w"&gt; &lt;/span&gt;to&lt;span class="w"&gt; &lt;/span&gt;any&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;22&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;proto&lt;span class="w"&gt; &lt;/span&gt;tcp

&lt;span class="c1"&gt;# Allow a port range&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;6000&lt;/span&gt;:6007/tcp

&lt;span class="c1"&gt;# Deny a specific IP&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;deny&lt;span class="w"&gt; &lt;/span&gt;from&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.100

&lt;span class="c1"&gt;# Rate-limit SSH (allows 6 connections per 30 seconds)&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;limit&lt;span class="w"&gt; &lt;/span&gt;ssh

&lt;span class="c1"&gt;# Delete a rule&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;delete&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;http

&lt;span class="c1"&gt;# Insert a rule at a specific position&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;insert&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;deny&lt;span class="w"&gt; &lt;/span&gt;from&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;192&lt;/span&gt;.0.2.0/24

&lt;span class="c1"&gt;# Allow from a specific IP to a specific port&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;from&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;198&lt;/span&gt;.51.100.22&lt;span class="w"&gt; &lt;/span&gt;to&lt;span class="w"&gt; &lt;/span&gt;any&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;5432&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;proto&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;comment&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;PostgreSQL admin&amp;#39;&lt;/span&gt;

&lt;span class="c1"&gt;# Show numbered rules for deletion&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;numbered
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="application-profiles"&gt;Application&amp;nbsp;Profiles&lt;/h3&gt;
&lt;p&gt;ufw supports application profiles - predefined port configurations stored&amp;nbsp;in &lt;code&gt;/etc/ufw/applications.d/&lt;/code&gt;. Many packages drop their own profile files into this directory automatically when installed&amp;nbsp;via &lt;code&gt;apt&lt;/code&gt; - install Nginx or Apache and the corresponding ufw profile appears without any extra work. This is part of why ufw feels so frictionless on Ubuntu: the ecosystem does the heavy&amp;nbsp;lifting.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# List available profiles&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;app&lt;span class="w"&gt; &lt;/span&gt;list

&lt;span class="c1"&gt;# Get profile details&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;app&lt;span class="w"&gt; &lt;/span&gt;info&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Nginx Full&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Profile: Nginx Full
Title: Web Server (Nginx, HTTP + HTTPS)
Description: Small, but very powerful and efficient web server

Ports:
  80,443/tcp
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Use a profile&lt;/span&gt;
ufw&lt;span class="w"&gt; &lt;/span&gt;allow&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;Nginx Full&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="behind-the-scenes"&gt;Behind the&amp;nbsp;Scenes&lt;/h3&gt;
&lt;p&gt;ufw stores its rules&amp;nbsp;in &lt;code&gt;/etc/ufw/user.rules&lt;/code&gt; and &lt;code&gt;/etc/ufw/user6.rules&lt;/code&gt;. It also&amp;nbsp;has &lt;code&gt;before.rules&lt;/code&gt; and &lt;code&gt;after.rules&lt;/code&gt; files for advanced configurations that go beyond ufw&amp;#8217;s command-line interface - for example, &lt;span class="caps"&gt;NAT&lt;/span&gt; rules or custom &lt;span class="caps"&gt;FORWARD&lt;/span&gt;&amp;nbsp;chains:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# /etc/ufw/before.rules - add NAT for containers/VMs&lt;/span&gt;
*nat
:POSTROUTING&lt;span class="w"&gt; &lt;/span&gt;ACCEPT&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;[&lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;:0&lt;span class="o"&gt;]&lt;/span&gt;
-A&lt;span class="w"&gt; &lt;/span&gt;POSTROUTING&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.0.0.0/24&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;eth0&lt;span class="w"&gt; &lt;/span&gt;-j&lt;span class="w"&gt; &lt;/span&gt;MASQUERADE
COMMIT
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="when-to-use-ufw"&gt;When to Use&amp;nbsp;ufw&lt;/h3&gt;
&lt;p&gt;ufw is perfect for single-purpose servers, personal &lt;span class="caps"&gt;VPS&lt;/span&gt; instances, and situations where you just need basic allow/deny rules without ceremony. It&amp;#8217;s genuinely the fastest path from &amp;#8220;naked server&amp;#8221; to &amp;#8220;firewalled server,&amp;#8221; and there&amp;#8217;s no shame in using it. Not everything needs to be a hand-crafted artisanal nftables&amp;nbsp;configuration.&lt;/p&gt;
&lt;p&gt;Where it falls short: complex multi-zone setups, advanced &lt;span class="caps"&gt;NAT&lt;/span&gt;, detailed logging configurations, or anything where you need to understand and control the exact rule structure. ufw deliberately hides that complexity, which is a feature until it&amp;nbsp;isn&amp;#8217;t.&lt;/p&gt;
&lt;h2 id="choosing-your-firewall"&gt;Choosing Your&amp;nbsp;Firewall&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Aspect&lt;/th&gt;
&lt;th&gt;iptables&lt;/th&gt;
&lt;th&gt;nftables&lt;/th&gt;
&lt;th&gt;firewalld&lt;/th&gt;
&lt;th&gt;ufw&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Learning curve&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Moderate&lt;/td&gt;
&lt;td&gt;Moderate&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;Very low&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;IPv4/IPv6 unified&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Syntax&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Verbose&lt;/td&gt;
&lt;td&gt;Clean&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;CLI&lt;/span&gt;/&lt;span class="caps"&gt;XML&lt;/span&gt;/D-Bus&lt;/td&gt;
&lt;td&gt;Minimal&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Best for&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Legacy systems&lt;/td&gt;
&lt;td&gt;New deployments&lt;/td&gt;
&lt;td&gt;Enterprise/fleet management&lt;/td&gt;
&lt;td&gt;Ubuntu/quick setups&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Native sets&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No (needs ipset)&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Via zones/ipsets&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Atomic updates&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Runtime/permanent split&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Programmatic &lt;span class="caps"&gt;API&lt;/span&gt;&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Netlink&lt;/td&gt;
&lt;td&gt;D-Bus&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Web &lt;span class="caps"&gt;UI&lt;/span&gt; (Cockpit)&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Config management&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Manual&lt;/td&gt;
&lt;td&gt;Manual&lt;/td&gt;
&lt;td&gt;Ansible module&lt;/td&gt;
&lt;td&gt;Manual&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; default&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Backend&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Ubuntu default&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Backend&lt;/td&gt;
&lt;td&gt;No&lt;/td&gt;
&lt;td&gt;Yes&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;&lt;span class="caps"&gt;CLI&lt;/span&gt; complexity&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;High&lt;/td&gt;
&lt;td&gt;Medium&lt;/td&gt;
&lt;td&gt;Low&lt;/td&gt;
&lt;td&gt;Very low&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;My general recommendation for new Linux&amp;nbsp;deployments:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Running &lt;span class="caps"&gt;RHEL&lt;/span&gt;, Fedora, or CentOS Stream?&lt;/strong&gt; Use firewalld. It&amp;#8217;s the native choice and the integration benefits are substantial - Cockpit, Ansible, NetworkManager, and libvirt all speak firewalld natively. If you&amp;#8217;re managing more than a handful of servers, the zone/service abstraction and D-Bus &lt;span class="caps"&gt;API&lt;/span&gt; will save you significant time compared to hand-rolled nftables across every&amp;nbsp;machine.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Running Ubuntu or Debian and just need basic rules?&lt;/strong&gt; Use ufw. It gets the job done in under a&amp;nbsp;minute.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Need precise control, custom rulesets, or complex &lt;span class="caps"&gt;NAT&lt;/span&gt;?&lt;/strong&gt; Use nftables directly. It&amp;#8217;s the future of Linux packet&amp;nbsp;filtering.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Maintaining an existing iptables setup that works?&lt;/strong&gt; Keep it.&amp;nbsp;The &lt;code&gt;iptables-nft&lt;/code&gt; compatibility layer means your rules are already running on nftables under the hood. Migrate when you have a reason, not because someone on Reddit told you&amp;nbsp;to.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="common-patterns-for-cloud-servers"&gt;Common Patterns for Cloud&amp;nbsp;Servers&lt;/h2&gt;
&lt;p&gt;Regardless of which tool you choose, certain patterns apply to every cloud server&amp;nbsp;firewall:&lt;/p&gt;
&lt;h3 id="1-default-deny-incoming"&gt;1. Default Deny&amp;nbsp;Incoming&lt;/h3&gt;
&lt;p&gt;Every firewall configuration should start with a default deny policy for incoming traffic. This is the single most important rule. If you forget to allow a service, it&amp;#8217;s unreachable (you notice immediately). If you forget to block a service with a default-allow policy, it&amp;#8217;s exposed (you might never&amp;nbsp;notice).&lt;/p&gt;
&lt;h3 id="2-always-allow-established-connections"&gt;2. Always Allow Established&amp;nbsp;Connections&lt;/h3&gt;
&lt;p&gt;Stateful tracking is essential. A rule&amp;nbsp;like &lt;code&gt;ct state established,related accept&lt;/code&gt; (nftables)&amp;nbsp;or &lt;code&gt;-m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT&lt;/code&gt; (iptables) ensures that return traffic for allowed outbound connections gets through without explicit rules for every possible&amp;nbsp;response.&lt;/p&gt;
&lt;h3 id="3-rate-limit-ssh"&gt;3. Rate-Limit &lt;span class="caps"&gt;SSH&lt;/span&gt;&lt;/h3&gt;
&lt;p&gt;Every public-facing &lt;span class="caps"&gt;SSH&lt;/span&gt; server gets brute-forced. Rate limiting won&amp;#8217;t stop a determined attacker, but it dramatically slows down automated scanners. Combine it with key-only authentication&amp;nbsp;(&lt;code&gt;PasswordAuthentication no&lt;/code&gt; in &lt;code&gt;sshd_config&lt;/code&gt;) for real&amp;nbsp;security.&lt;/p&gt;
&lt;h3 id="4-never-block-essential-icmpv6"&gt;4. Never Block Essential&amp;nbsp;ICMPv6&lt;/h3&gt;
&lt;p&gt;Blocking all ICMPv6 will break IPv6 connectivity. Neighbor Discovery (the IPv6 equivalent of &lt;span class="caps"&gt;ARP&lt;/span&gt;) uses ICMPv6. Path &lt;span class="caps"&gt;MTU&lt;/span&gt; Discovery uses ICMPv6. If you&amp;nbsp;block &lt;code&gt;neighbour-solicitation&lt;/code&gt; or &lt;code&gt;packet-too-big&lt;/code&gt;, your IPv6 connectivity will fail in mysterious&amp;nbsp;ways.&lt;/p&gt;
&lt;h3 id="5-log-selectively"&gt;5. Log&amp;nbsp;Selectively&lt;/h3&gt;
&lt;p&gt;Logging every dropped packet will fill your disk. Log with rate limiting, and log the things that matter - unexpected traffic on specific ports, connections from specific sources, or rules that should never trigger but&amp;nbsp;do.&lt;/p&gt;
&lt;h3 id="6-the-docker-trap"&gt;6. The Docker&amp;nbsp;Trap&lt;/h3&gt;
&lt;p&gt;This one burns almost every Linux admin at least once: &lt;strong&gt;Docker bypasses your firewall&amp;nbsp;rules.&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;When you&amp;nbsp;run &lt;code&gt;docker run -p 8080:80 nginx&lt;/code&gt;, Docker doesn&amp;#8217;t politely ask ufw or firewalld to open port 8080. Instead, it manipulates the&amp;nbsp;iptables &lt;code&gt;PREROUTING&lt;/code&gt; and &lt;code&gt;DOCKER&lt;/code&gt; chains directly, inserting rules that take effect &lt;em&gt;before&lt;/em&gt; your &lt;span class="caps"&gt;INPUT&lt;/span&gt; chain is evaluated. The result: that container port is exposed to the entire internet, completely ignoring your carefully&amp;nbsp;crafted &lt;code&gt;ufw default deny incoming&lt;/code&gt; policy.&lt;/p&gt;
&lt;p&gt;This happens because Docker needs to implement its port-forwarding via &lt;span class="caps"&gt;NAT&lt;/span&gt;, and it does so by writing directly to Netfilter - bypassing any frontend that manages the &lt;span class="caps"&gt;INPUT&lt;/span&gt;&amp;nbsp;chain.&lt;/p&gt;
&lt;p&gt;Mitigations:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Option 1: Bind to localhost only (container accessible only from the host)&lt;/span&gt;
docker&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;127&lt;/span&gt;.0.0.1:8080:80&lt;span class="w"&gt; &lt;/span&gt;nginx

&lt;span class="c1"&gt;# Option 2: Disable Docker&amp;#39;s iptables manipulation entirely&lt;/span&gt;
&lt;span class="c1"&gt;# In /etc/docker/daemon.json:&lt;/span&gt;
&lt;span class="o"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;iptables&amp;quot;&lt;/span&gt;:&lt;span class="w"&gt; &lt;/span&gt;false&lt;span class="o"&gt;}&lt;/span&gt;
&lt;span class="c1"&gt;# WARNING: You are now responsible for all container networking rules.&lt;/span&gt;
&lt;span class="c1"&gt;# Containers will lose outbound internet access until you manually&lt;/span&gt;
&lt;span class="c1"&gt;# configure NAT masquerading for the docker0 bridge.&lt;/span&gt;

&lt;span class="c1"&gt;# Option 3: Use Docker&amp;#39;s internal network and reverse-proxy through&lt;/span&gt;
&lt;span class="c1"&gt;# a host-level web server (Nginx, Caddy) that IS behind your firewall&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The safest pattern for production: don&amp;#8217;t publish ports directly&amp;nbsp;with &lt;code&gt;-p&lt;/code&gt;. Instead, put a reverse proxy on the host that listens on the public interface (behind your firewall), and have it forward traffic to containers on Docker&amp;#8217;s internal bridge network. This way, your firewall rules actually apply to the traffic reaching your&amp;nbsp;services.&lt;/p&gt;
&lt;p&gt;Podman, notably, doesn&amp;#8217;t have this problem in rootless mode - it&amp;nbsp;uses &lt;code&gt;slirp4netns&lt;/code&gt; or &lt;code&gt;pasta&lt;/code&gt; for networking, which doesn&amp;#8217;t touch the host&amp;#8217;s iptables chains. One more reason to consider it for server&amp;nbsp;workloads.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;All four tools solve the same fundamental problem: controlling which packets get into your server and which don&amp;#8217;t. The differences are in ergonomics, abstraction level, and ecosystem integration. iptables is the weathered veteran - it works everywhere but shows its age. nftables is the well-designed successor that hasn&amp;#8217;t fully taken over yet because the old guard is entrenched. firewalld wraps everything in Red Hat&amp;#8217;s zone model, which is either a helpful abstraction or an unnecessary layer depending on your perspective. ufw strips it all down to the bare essentials, which is either refreshingly simple or frustratingly&amp;nbsp;limited.&lt;/p&gt;
&lt;p&gt;Pick the tool that matches your distribution, your team, and your complexity requirements. Then configure it, test it, and - most importantly - actually turn it on. A perfectly designed ruleset sitting in a text file isn&amp;#8217;t protecting&amp;nbsp;anything.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="corrections"&gt;Corrections&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;2026-03-14&lt;/strong&gt;: An earlier version of this article stated that nftables has no programmatic &lt;span class="caps"&gt;API&lt;/span&gt;. This is incorrect - nftables exposes a Netlink interface&amp;nbsp;(&lt;code&gt;NETLINK_NETFILTER&lt;/code&gt;) that allows applications to manage rules directly without&amp;nbsp;the &lt;code&gt;nft&lt;/code&gt; command-line tool. The comparison table and nftables section have been updated to reflect this. Thanks to &lt;a href="https://hachyderm.io/@joew"&gt;Joe Chilliams (@joew@hachyderm.io)&lt;/a&gt; for the correction and the excellent real-world&amp;nbsp;examples.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://wiki.nftables.org/"&gt;nftables wiki&lt;/a&gt; - the canonical nftables&amp;nbsp;documentation&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man7.org/linux/man-pages/man8/iptables.8.html"&gt;iptables man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.netfilter.org/"&gt;Netfilter project&lt;/a&gt; - the umbrella project for Linux packet&amp;nbsp;filtering&lt;/li&gt;
&lt;li&gt;&lt;a href="https://firewalld.org/documentation/"&gt;firewalld&amp;nbsp;documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://help.ubuntu.com/community/UFW"&gt;ufw community help&amp;nbsp;(Ubuntu)&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10/html/configuring_firewalls_and_packet_filters/using-and-configuring-firewalld_firewall-packet-filters"&gt;Red Hat - Using&amp;nbsp;firewalld&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://wiki.debian.org/nftables"&gt;Debian wiki -&amp;nbsp;nftables&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;Every Linux firewall tool on this page gets the job done. But if you really want to see packet filtering done with elegance - a single, clean configuration file, last-match-wins semantics, tables that scale to hundreds of thousands of entries without breaking a sweat, and a syntax that reads like it was designed by someone who actually enjoys writing firewall rules - well, you know where I stand. &lt;a href="https://hofstede.it/pf-firewall-on-freebsd-a-practical-guide/"&gt;&lt;span class="caps"&gt;PF&lt;/span&gt; on FreeBSD&lt;/a&gt; remains the gold standard. The rest of us are just approximating&amp;nbsp;greatness.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="firewall"/><category term="iptables"/><category term="nftables"/><category term="firewalld"/><category term="ufw"/><category term="security"/><category term="networking"/><category term="cloud"/></entry><entry><title>Neovim Crash Course for Sysadmins: The 20% That Solve 80% of the Pain</title><link href="https://hofstede.it/neovim-crash-course-for-sysadmins-the-20-that-solve-80-of-the-pain/" rel="alternate"/><published>2026-02-27T00:00:00+01:00</published><updated>2026-02-27T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-02-27:/neovim-crash-course-for-sysadmins-the-20-that-solve-80-of-the-pain/</id><summary type="html">&lt;p&gt;Not another hjkl tutorial. This is the stuff you still get wrong after years of using Vim - efficient navigation, copy/paste that actually works, &lt;span class="caps"&gt;YAML&lt;/span&gt;-specific workflows, and the motions that turn config file editing from a chore into a&amp;nbsp;joy.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Neovim Crash Course for Sysadmins" src="https://hofstede.it/images/2026-02-27-neovim-crashkurs-sysadmin-devops.png" title="Neovim Crash Course for Sysadmins"&gt;&lt;/p&gt;
&lt;p&gt;This is not a beginner&amp;#8217;s guide. If you&amp;#8217;ve used Vim daily for a year and still occasionally fight with paste behavior, you&amp;#8217;re the target audience. This article covers the things I got wrong (or never properly learned) after fifteen years of daily Vim usage - and the moment everything&amp;nbsp;clicked.&lt;/p&gt;
&lt;p&gt;The focus is practical: editing configuration files, &lt;span class="caps"&gt;YAML&lt;/span&gt;, shell scripts, and infrastructure code. No Vim philosophy lectures. No &amp;#8220;Vim is a language&amp;#8221; metaphors. Just the patterns that make the biggest difference for sysadmins and DevOps engineers who spend their days in&amp;nbsp;terminals.&lt;/p&gt;
&lt;h2 id="table-of-contents"&gt;Table of&amp;nbsp;Contents&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="#motions-that-actually-matter"&gt;Motions That Actually&amp;nbsp;Matter&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#jumping-not-crawling"&gt;Jumping, Not&amp;nbsp;Crawling&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-power-of-text-objects"&gt;The Power of Text&amp;nbsp;Objects&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#copypaste-the-thing-youve-been-doing-wrong"&gt;Copy/Paste: The Thing You&amp;#8217;ve Been Doing&amp;nbsp;Wrong&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#search-and-replace-scoped-to-what-you-need"&gt;Search and Replace: Scoped to What You&amp;nbsp;Need&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#yaml-the-pain-and-the-antidote"&gt;&lt;span class="caps"&gt;YAML&lt;/span&gt;: The Pain and the&amp;nbsp;Antidote&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#registers-and-macros-automation-for-the-lazy"&gt;Registers and Macros: Automation for the&amp;nbsp;Lazy&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#splits-buffers-and-efficient-file-navigation"&gt;Splits, Buffers, and Efficient File&amp;nbsp;Navigation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#the-neovim-advantage-lsp-and-treesitter"&gt;The Neovim Advantage: &lt;span class="caps"&gt;LSP&lt;/span&gt; and&amp;nbsp;Treesitter&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#practical-combos-the-cheat-sheet"&gt;Practical Combos: The Cheat&amp;nbsp;Sheet&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="#wrapping-up"&gt;Wrapping&amp;nbsp;Up&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="motions-that-actually-matter"&gt;Motions That Actually&amp;nbsp;Matter&lt;/h2&gt;
&lt;p&gt;Everyone&amp;nbsp;knows &lt;code&gt;hjkl&lt;/code&gt;. Most people&amp;nbsp;know &lt;code&gt;w&lt;/code&gt; and &lt;code&gt;b&lt;/code&gt;. Here&amp;#8217;s what separates fast editing from &amp;#8220;I&amp;#8217;ll just use&amp;nbsp;sed&amp;#8221;:&lt;/p&gt;
&lt;h3 id="jumping-not-crawling"&gt;Jumping, Not&amp;nbsp;Crawling&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Motion&lt;/th&gt;
&lt;th&gt;What it does&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;f{char}&lt;/code&gt; / &lt;code&gt;F{char}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Jump to next/previous occurrence&amp;nbsp;of &lt;code&gt;{char}&lt;/code&gt; on the line&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;t{char}&lt;/code&gt; / &lt;code&gt;T{char}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Jump to just before&amp;nbsp;next/previous &lt;code&gt;{char}&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;;&lt;/code&gt; and &lt;code&gt;,&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Repeat&amp;nbsp;last &lt;code&gt;f&lt;/code&gt;/&lt;code&gt;t&lt;/code&gt; forward/backward&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;%&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Jump to matching bracket/parenthesis&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;{&lt;/code&gt; / &lt;code&gt;}&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Jump to previous/next empty line (paragraph)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;Ctrl-d&lt;/code&gt; / &lt;code&gt;Ctrl-u&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Half-page down/up&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;*&lt;/code&gt; / &lt;code&gt;#&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Search word under cursor forward/backward&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;gd&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Go to local definition of word under cursor&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The &lt;code&gt;f&lt;/code&gt;/&lt;code&gt;t&lt;/code&gt; family is criminally underused. Editing a firewall rule and need to change the port&amp;nbsp;number? &lt;code&gt;f:&lt;/code&gt; jumps to the&amp;nbsp;colon, &lt;code&gt;l&lt;/code&gt; moves one&amp;nbsp;right, &lt;code&gt;cw&lt;/code&gt; changes the word. Three keystrokes instead of&amp;nbsp;mashing &lt;code&gt;w&lt;/code&gt; twelve&amp;nbsp;times.&lt;/p&gt;
&lt;h3 id="the-power-of-text-objects"&gt;The Power of Text&amp;nbsp;Objects&lt;/h3&gt;
&lt;p&gt;This is where Vim stops being a text editor and starts being a scalpel. Text objects work with any operator&amp;nbsp;(&lt;code&gt;d&lt;/code&gt;, &lt;code&gt;c&lt;/code&gt;, &lt;code&gt;y&lt;/code&gt;, &lt;code&gt;v&lt;/code&gt;):&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Object&lt;/th&gt;
&lt;th&gt;Meaning&lt;/th&gt;
&lt;th&gt;Example&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;iw&lt;/code&gt; / &lt;code&gt;aw&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;inner word / a word (includes trailing space)&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ciw&lt;/code&gt; - change word under cursor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;i"&lt;/code&gt; / &lt;code&gt;a"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;inside quotes / around quotes&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ci"&lt;/code&gt; - change contents of quoted string&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;i(&lt;/code&gt; / &lt;code&gt;a(&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;inside parentheses / around parens&lt;/td&gt;
&lt;td&gt;&lt;code&gt;di(&lt;/code&gt; - delete contents&amp;nbsp;between &lt;code&gt;()&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;i{&lt;/code&gt; / &lt;code&gt;a{&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;inside braces / around braces&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ci{&lt;/code&gt; - change block contents&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;ip&lt;/code&gt; / &lt;code&gt;ap&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;inner paragraph / a paragraph&lt;/td&gt;
&lt;td&gt;&lt;code&gt;yip&lt;/code&gt; - yank entire paragraph&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;it&lt;/code&gt; / &lt;code&gt;at&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;inside &lt;span class="caps"&gt;XML&lt;/span&gt;/&lt;span class="caps"&gt;HTML&lt;/span&gt; tag / around tag&lt;/td&gt;
&lt;td&gt;&lt;code&gt;cit&lt;/code&gt; - change tag contents&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h3 id="visualizing-text-objects"&gt;Visualizing Text&amp;nbsp;Objects&lt;/h3&gt;
&lt;p&gt;Text objects are easier to grasp with a concrete example. Given this&amp;nbsp;code:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;function_name() {
    some_value = true
}
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;code&gt;di{&lt;/code&gt; deletes &lt;code&gt;some_value = true&lt;/code&gt; - everything &lt;em&gt;between&lt;/em&gt; the braces,&amp;nbsp;leaving &lt;code&gt;{ }&lt;/code&gt; intact&lt;/li&gt;
&lt;li&gt;&lt;code&gt;da{&lt;/code&gt; deletes &lt;code&gt;{ some_value = true }&lt;/code&gt; - the braces &lt;span class="caps"&gt;AND&lt;/span&gt; their&amp;nbsp;contents&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Same logic applies to&amp;nbsp;quotes: &lt;code&gt;di"&lt;/code&gt; deletes just the string&amp;nbsp;contents, &lt;code&gt;da"&lt;/code&gt; deletes the quotes too. For&amp;nbsp;paragraphs, &lt;code&gt;dip&lt;/code&gt; deletes the&amp;nbsp;text, &lt;code&gt;dap&lt;/code&gt; also removes the trailing blank&amp;nbsp;line.&lt;/p&gt;
&lt;p&gt;Real-world example - you have a Jinja2 template&amp;nbsp;variable &lt;code&gt;{{ old_value }}&lt;/code&gt; and need to change&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ci{  →  deletes old_value, leaves you in insert mode between the braces
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;One combo instead of navigating, selecting, deleting,&amp;nbsp;typing.&lt;/p&gt;
&lt;h2 id="copypaste-the-thing-youve-been-doing-wrong"&gt;Copy/Paste: The Thing You&amp;#8217;ve Been Doing&amp;nbsp;Wrong&lt;/h2&gt;
&lt;p&gt;This was my fifteen-year blind&amp;nbsp;spot.&lt;/p&gt;
&lt;h3 id="the-problem"&gt;The&amp;nbsp;Problem&lt;/h3&gt;
&lt;p&gt;You visually select some lines&amp;nbsp;with &lt;code&gt;v&lt;/code&gt;, yank&amp;nbsp;with &lt;code&gt;y&lt;/code&gt;, move somewhere,&amp;nbsp;hit &lt;code&gt;p&lt;/code&gt; - and the text lands in the middle of a line instead of on a new line below. You undo,&amp;nbsp;try &lt;code&gt;P&lt;/code&gt;, and it ends up in the middle of the line &lt;em&gt;above&lt;/em&gt;.&amp;nbsp;Familiar?&lt;/p&gt;
&lt;h3 id="why-it-happens"&gt;Why It&amp;nbsp;Happens&lt;/h3&gt;
&lt;p&gt;Vim tracks whether a yank was &lt;strong&gt;characterwise&lt;/strong&gt;, &lt;strong&gt;linewise&lt;/strong&gt;, or &lt;strong&gt;blockwise&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;v&lt;/code&gt; (characterwise)&lt;/strong&gt;&amp;nbsp;→ &lt;code&gt;y&lt;/code&gt; creates a characterwise register&amp;nbsp;→ &lt;code&gt;p&lt;/code&gt; inserts&amp;nbsp;inline&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;V&lt;/code&gt; (linewise)&lt;/strong&gt;&amp;nbsp;→ &lt;code&gt;y&lt;/code&gt; creates a linewise register&amp;nbsp;→ &lt;code&gt;p&lt;/code&gt; inserts below current&amp;nbsp;line&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;code&gt;Ctrl-v&lt;/code&gt; (blockwise)&lt;/strong&gt;&amp;nbsp;→ &lt;code&gt;y&lt;/code&gt; creates a blockwise register&amp;nbsp;→ &lt;code&gt;p&lt;/code&gt; inserts as a&amp;nbsp;column&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The yank type determines the paste behavior. That&amp;#8217;s it. That&amp;#8217;s the whole&amp;nbsp;mystery.&lt;/p&gt;
&lt;h3 id="the-fix"&gt;The&amp;nbsp;Fix&lt;/h3&gt;
&lt;p&gt;&lt;strong&gt;Use &lt;code&gt;V&lt;/code&gt; (Visual Line) when you want to copy/paste lines.&lt;/strong&gt; This is the single biggest quality-of-life&amp;nbsp;improvement:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;V       → enter visual line mode
jjj     → select lines downward
y       → yank (linewise)
}       → jump to where you want it
p       → paste below current line. Clean. New line. No surprises.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="quick-reference"&gt;Quick&amp;nbsp;Reference&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Want to&amp;#8230;&lt;/th&gt;
&lt;th&gt;Use&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Yank current line&lt;/td&gt;
&lt;td&gt;&lt;code&gt;yy&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Yank 5 lines&lt;/td&gt;
&lt;td&gt;&lt;code&gt;5yy&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Yank a paragraph&lt;/td&gt;
&lt;td&gt;&lt;code&gt;yip&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Yank lines visually&lt;/td&gt;
&lt;td&gt;&lt;code&gt;V&lt;/code&gt; + move&amp;nbsp;+ &lt;code&gt;y&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Paste below&lt;/td&gt;
&lt;td&gt;&lt;code&gt;p&lt;/code&gt; (with linewise yank)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Paste above&lt;/td&gt;
&lt;td&gt;&lt;code&gt;P&lt;/code&gt; (with linewise yank)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Force paste as new line&lt;/td&gt;
&lt;td&gt;&lt;code&gt;:put&lt;/code&gt; (below)&amp;nbsp;/ &lt;code&gt;:put!&lt;/code&gt; (above)&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The &lt;code&gt;:put&lt;/code&gt; command is your safety net - it &lt;em&gt;always&lt;/em&gt; pastes as a new line, regardless of how you yanked. Useful when you&amp;#8217;ve already yanked characterwise and don&amp;#8217;t want to redo&amp;nbsp;it.&lt;/p&gt;
&lt;h3 id="named-registers-your-clipboard-slots"&gt;Named Registers: Your Clipboard&amp;nbsp;Slots&lt;/h3&gt;
&lt;p&gt;Vim has 26 named registers&amp;nbsp;(&lt;code&gt;"a&lt;/code&gt; through &lt;code&gt;"z&lt;/code&gt;). Use&amp;nbsp;them:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="s"&gt;&amp;quot;ayy    → yank current line into register a&lt;/span&gt;
&lt;span class="s"&gt;&amp;quot;bV3jy  → yank 4 lines into register b&lt;/span&gt;
&lt;span class="s"&gt;&amp;quot;ap     → paste from register a&lt;/span&gt;
&lt;span class="s"&gt;&amp;quot;bp     → paste from register b&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The system clipboard is&amp;nbsp;register &lt;code&gt;"+&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&amp;quot;+yy    → yank line to system clipboard
&amp;quot;+p     → paste from system clipboard
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For Neovim, you can unify the clipboard by adding to your&amp;nbsp;config:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;clipboard&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;unnamedplus&amp;#39;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now &lt;code&gt;y&lt;/code&gt; and &lt;code&gt;p&lt;/code&gt; use the system clipboard directly. Whether you want this is a matter of taste - I prefer keeping Vim&amp;#8217;s registers separate and&amp;nbsp;using &lt;code&gt;"+&lt;/code&gt; explicitly when I need the system&amp;nbsp;clipboard.&lt;/p&gt;
&lt;h2 id="search-and-replace-scoped-to-what-you-need"&gt;Search and Replace: Scoped to What You&amp;nbsp;Need&lt;/h2&gt;
&lt;h3 id="global-replace"&gt;Global&amp;nbsp;Replace&lt;/h3&gt;
&lt;p&gt;The classic everyone&amp;nbsp;knows:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:&lt;/span&gt;%s&lt;span class="sr"&gt;/old/&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt;/&lt;span class="k"&gt;g&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;%&lt;/code&gt; means the entire&amp;nbsp;file. &lt;code&gt;g&lt;/code&gt; means all occurrences per&amp;nbsp;line.&lt;/p&gt;
&lt;h3 id="replace-only-in-visual-selection"&gt;Replace Only in Visual&amp;nbsp;Selection&lt;/h3&gt;
&lt;p&gt;Select a block&amp;nbsp;with &lt;code&gt;V&lt;/code&gt; (or &lt;code&gt;v&lt;/code&gt;), then&amp;nbsp;press &lt;code&gt;:&lt;/code&gt;. Vim auto-fills the&amp;nbsp;range:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;lt;,&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;s&lt;span class="sr"&gt;/old/&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt;/&lt;span class="k"&gt;g&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This replaces only within the selected &lt;strong&gt;lines&lt;/strong&gt;. But here&amp;#8217;s the subtlety:&amp;nbsp;with &lt;code&gt;V&lt;/code&gt; (Visual Line), this replaces in the entire lines. If you selected&amp;nbsp;with &lt;code&gt;v&lt;/code&gt; (characterwise) and want to restrict the match to &lt;em&gt;exactly&lt;/em&gt; the highlighted&amp;nbsp;text:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;&amp;lt;,&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;s&lt;span class="sr"&gt;/\%Vold/&lt;/span&gt;&lt;span class="k"&gt;new&lt;/span&gt;/&lt;span class="k"&gt;g&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;\%V&lt;/code&gt; atom constrains the match to the visual selection boundary, not just the lines it spans. Niche, but invaluable when you need surgical&amp;nbsp;precision.&lt;/p&gt;
&lt;h3 id="useful-flags"&gt;Useful&amp;nbsp;Flags&lt;/h3&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Flag&lt;/th&gt;
&lt;th&gt;Effect&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;g&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;All occurrences per line (not just first)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;c&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Confirm each replacement&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;i&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Case insensitive&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;I&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Case sensitive&amp;nbsp;(overrides &lt;code&gt;ignorecase&lt;/code&gt; setting)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;n&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Count matches without replacing&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;The &lt;code&gt;c&lt;/code&gt; flag is&amp;nbsp;underrated. &lt;code&gt;:%s/foo/bar/gc&lt;/code&gt; lets you step through every match and decide&amp;nbsp;with &lt;code&gt;y&lt;/code&gt;/&lt;code&gt;n&lt;/code&gt;. Much safer than blind replace on a production&amp;nbsp;config.&lt;/p&gt;
&lt;h3 id="multi-file-replace-with-argdo-cfdo"&gt;Multi-File Replace with :argdo /&amp;nbsp;:cfdo&lt;/h3&gt;
&lt;p&gt;Need to replace across multiple files? Open them&amp;nbsp;and:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:&lt;/span&gt;args *.yaml
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;argdo&lt;/span&gt; %s&lt;span class="sr"&gt;/old_value/&lt;/span&gt;new_value/&lt;span class="k"&gt;g&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt; &lt;span class="k"&gt;update&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Or use the quickfix list after a&amp;nbsp;grep:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;vimgrep&lt;/span&gt; &lt;span class="sr"&gt;/pattern/&lt;/span&gt; **/*.yaml
&lt;span class="p"&gt;:&lt;/span&gt;cfdo %s&lt;span class="sr"&gt;/pattern/&lt;/span&gt;replacement/&lt;span class="k"&gt;g&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt; &lt;span class="k"&gt;update&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="yaml-the-pain-and-the-antidote"&gt;&lt;span class="caps"&gt;YAML&lt;/span&gt;: The Pain and the&amp;nbsp;Antidote&lt;/h2&gt;
&lt;p&gt;&lt;span class="caps"&gt;YAML&lt;/span&gt; editing is where Vim either shines or makes you want to throw your keyboard. Here&amp;#8217;s how to make it&amp;nbsp;shine.&lt;/p&gt;
&lt;h3 id="indentation-the-basics"&gt;Indentation: The&amp;nbsp;Basics&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;&amp;gt;&amp;gt;&lt;/span&gt;      → indent current line one &lt;span class="nb"&gt;shiftwidth&lt;/span&gt;
&lt;span class="p"&gt;&amp;lt;&amp;lt;&lt;/span&gt;      → unindent current line
&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;ip     → indent entire paragraph
&lt;span class="m"&gt;5&lt;/span&gt;&lt;span class="p"&gt;&amp;gt;&amp;gt;&lt;/span&gt;     → indent &lt;span class="m"&gt;5&lt;/span&gt; &lt;span class="nb"&gt;lines&lt;/span&gt;
V5j&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;    → visual select &lt;span class="m"&gt;6&lt;/span&gt; &lt;span class="nb"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; indent
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;In Visual&amp;nbsp;mode, &lt;code&gt;&amp;gt;&lt;/code&gt; indents and then &lt;em&gt;exits&lt;/em&gt; Visual mode.&amp;nbsp;Use &lt;code&gt;gv&lt;/code&gt; to reselect the same area, or better - just&amp;nbsp;use &lt;code&gt;.&lt;/code&gt; to&amp;nbsp;repeat:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;V5j&amp;gt;    → select and indent
.       → indent the same lines again (still selected by `&amp;#39;&amp;lt;,&amp;#39;&amp;gt;`)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="set-up-your-yaml-defaults"&gt;Set Up Your &lt;span class="caps"&gt;YAML&lt;/span&gt;&amp;nbsp;Defaults&lt;/h3&gt;
&lt;p&gt;Essential in your Neovim&amp;nbsp;config:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;api&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;nvim_create_autocmd&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;FileType&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
  &lt;span class="n"&gt;pattern&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;yaml&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
  &lt;span class="n"&gt;callback&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kr"&gt;function&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
    &lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;opt_local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;shiftwidth&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
    &lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;opt_local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;tabstop&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
    &lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;opt_local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;softtabstop&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;2&lt;/span&gt;
    &lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;opt_local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;expandtab&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;
    &lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;opt_local&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;cursorcolumn&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt;  &lt;span class="c1"&gt;-- vertical line at cursor column&lt;/span&gt;
  &lt;span class="kr"&gt;end&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;cursorcolumn&lt;/code&gt; draws a vertical highlight at your cursor&amp;#8217;s column position - an instant visual guide for &lt;span class="caps"&gt;YAML&lt;/span&gt; indentation&amp;nbsp;alignment.&lt;/p&gt;
&lt;h3 id="moving-blocks-up-and-down"&gt;Moving Blocks Up and&amp;nbsp;Down&lt;/h3&gt;
&lt;p&gt;Rearranging &lt;span class="caps"&gt;YAML&lt;/span&gt; keys or Ansible&amp;nbsp;tasks:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;m&lt;/span&gt;&lt;span class="p"&gt;+&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;    → move current line down one
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;m&lt;/span&gt;&lt;span class="m"&gt;-2&lt;/span&gt;    → move current line &lt;span class="k"&gt;up&lt;/span&gt; one
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Or visually select a block and move&amp;nbsp;it:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;V3j     → select &lt;span class="m"&gt;4&lt;/span&gt; &lt;span class="nb"&gt;lines&lt;/span&gt;
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;m&lt;/span&gt; &amp;#39;&lt;span class="p"&gt;&amp;gt;+&lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt; → move block down one line
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;m&lt;/span&gt; &amp;#39;&lt;span class="p"&gt;&amp;lt;&lt;/span&gt;&lt;span class="m"&gt;-2&lt;/span&gt; → move block &lt;span class="k"&gt;up&lt;/span&gt; one line
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Bind these for&amp;nbsp;convenience:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;-- Move lines up/down in visual mode&lt;/span&gt;
&lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keymap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;v&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;J&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;:m &amp;#39;&amp;gt;+1&amp;lt;CR&amp;gt;gv=gv&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;keymap&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;v&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;K&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;&amp;quot;:m &amp;#39;&amp;lt;-2&amp;lt;CR&amp;gt;gv=gv&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Now &lt;code&gt;J&lt;/code&gt;/&lt;code&gt;K&lt;/code&gt; in Visual mode moves the selected block and re-indents. Invaluable for reordering Ansible&amp;nbsp;tasks.&lt;/p&gt;
&lt;h3 id="folding-yaml-sections"&gt;Folding &lt;span class="caps"&gt;YAML&lt;/span&gt;&amp;nbsp;Sections&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;foldmethod&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;indent&amp;#39;&lt;/span&gt;   &lt;span class="c1"&gt;-- YAML&amp;#39;s structure IS its indentation&lt;/span&gt;
&lt;span class="n"&gt;vim&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;opt&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;foldlevelstart&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="mi"&gt;99&lt;/span&gt;     &lt;span class="c1"&gt;-- start with everything unfolded&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Key&lt;/th&gt;
&lt;th&gt;Action&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;za&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Toggle fold under cursor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;zM&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Close all folds&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;zR&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Open all folds&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;zc&lt;/code&gt; / &lt;code&gt;zo&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Close / open one fold&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Folding by indent works perfectly for &lt;span class="caps"&gt;YAML&lt;/span&gt; because indentation &lt;em&gt;is&lt;/em&gt; the structure. A folded Ansible playbook shows you just the task names - like a table of&amp;nbsp;contents.&lt;/p&gt;
&lt;h2 id="registers-and-macros-automation-for-the-lazy"&gt;Registers and Macros: Automation for the&amp;nbsp;Lazy&lt;/h2&gt;
&lt;h3 id="the-dot-command"&gt;The Dot&amp;nbsp;Command&lt;/h3&gt;
&lt;p&gt;The most powerful single key in&amp;nbsp;Vim: &lt;code&gt;.&lt;/code&gt; repeats the last&amp;nbsp;change.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ciw&amp;quot;new_value&amp;quot;&amp;lt;Esc&amp;gt;   → change word to &amp;quot;new_value&amp;quot;
n                      → jump to next search match
.                      → apply the same change
n.n.n.                 → repeat across all matches
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This pattern - search, change, repeat - handles 90% of &amp;#8220;I need to change this in five places&amp;#8221; without reaching&amp;nbsp;for &lt;code&gt;:%s&lt;/code&gt;.&lt;/p&gt;
&lt;h3 id="quick-macros"&gt;Quick&amp;nbsp;Macros&lt;/h3&gt;
&lt;p&gt;Record&amp;nbsp;with &lt;code&gt;q{register}&lt;/code&gt;, replay&amp;nbsp;with &lt;code&gt;@{register}&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="n"&gt;qa&lt;/span&gt;&lt;span class="w"&gt;              &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;start&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;recording&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;into&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;register&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;a&lt;/span&gt;
&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="nl"&gt;f&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="n"&gt;w&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;jump&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;start&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;of&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;find&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;colon&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;move&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;next&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;
&lt;span class="n"&gt;ciw&lt;/span&gt;&lt;span class="ss"&gt;&amp;quot;&amp;lt;C-r&amp;gt;&amp;quot;&lt;/span&gt;&lt;span class="err"&gt;&amp;quot;&lt;/span&gt;&lt;span class="w"&gt;     &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;change&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;insert&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;paste&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;deleted&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;word&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;close&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;quote&lt;/span&gt;
&lt;span class="n"&gt;Esc&lt;/span&gt;&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;back&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;to&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;normal&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mode&lt;/span&gt;
&lt;span class="n"&gt;j&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;move&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;down&lt;/span&gt;
&lt;span class="n"&gt;q&lt;/span&gt;&lt;span class="w"&gt;               &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;stop&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;recording&lt;/span&gt;

&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="nv"&gt;@a&lt;/span&gt;&lt;span class="w"&gt;             &lt;/span&gt;&lt;span class="err"&gt;→&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;replay&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;times&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Note: &lt;code&gt;ciw"&amp;lt;C-r&amp;gt;""&lt;/code&gt; is the native way&amp;nbsp;- &lt;code&gt;ciw&lt;/code&gt; deletes the&amp;nbsp;word, &lt;code&gt;"&lt;/code&gt; starts the quoted&amp;nbsp;string, &lt;code&gt;&amp;lt;C-r&amp;gt;"&lt;/code&gt; (Ctrl-r followed by double-quote) pastes the deleted word from the default register, and the&amp;nbsp;final &lt;code&gt;"&lt;/code&gt; closes&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;For &lt;span class="caps"&gt;YAML&lt;/span&gt; quoting, this is also a great use case for &lt;strong&gt;vim-surround&lt;/strong&gt; or &lt;strong&gt;mini.surround&lt;/strong&gt;. With&amp;nbsp;mini.surround, &lt;code&gt;ysiw"&lt;/code&gt; wraps the word under cursor in quotes in a single command. No macro&amp;nbsp;needed.&lt;/p&gt;
&lt;h3 id="replaying-the-last-macro"&gt;Replaying the Last&amp;nbsp;Macro&lt;/h3&gt;
&lt;p&gt;&lt;code&gt;@@&lt;/code&gt; repeats the last played macro. Combined with a&amp;nbsp;count: &lt;code&gt;20@@&lt;/code&gt; - fire and&amp;nbsp;forget.&lt;/p&gt;
&lt;h2 id="splits-buffers-and-efficient-file-navigation"&gt;Splits, Buffers, and Efficient File&amp;nbsp;Navigation&lt;/h2&gt;
&lt;p&gt;Sysadmins often edit multiple related files: the playbook &lt;em&gt;and&lt;/em&gt; the inventory, the nginx config &lt;em&gt;and&lt;/em&gt; the upstream&amp;nbsp;block, &lt;code&gt;pf.conf&lt;/code&gt; &lt;em&gt;and&lt;/em&gt; the anchor&amp;nbsp;file.&lt;/p&gt;
&lt;h3 id="splits"&gt;Splits&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;sp&lt;/span&gt; filename    → horizontal split
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;vsp&lt;/span&gt; filename   → &lt;span class="k"&gt;vertical&lt;/span&gt; split
Ctrl&lt;span class="p"&gt;-&lt;/span&gt;&lt;span class="k"&gt;w&lt;/span&gt; &lt;span class="k"&gt;h&lt;/span&gt;&lt;span class="sr"&gt;/j/&lt;/span&gt;&lt;span class="k"&gt;k&lt;/span&gt;/&lt;span class="k"&gt;l&lt;/span&gt; → navigate between splits
Ctrl&lt;span class="p"&gt;-&lt;/span&gt;&lt;span class="k"&gt;w&lt;/span&gt; &lt;span class="p"&gt;=&lt;/span&gt;        → equalize split sizes
Ctrl&lt;span class="p"&gt;-&lt;/span&gt;&lt;span class="k"&gt;w&lt;/span&gt; _        → maximize current horizontal split
Ctrl&lt;span class="p"&gt;-&lt;/span&gt;&lt;span class="k"&gt;w&lt;/span&gt; &lt;span class="p"&gt;|&lt;/span&gt;        → maximize current &lt;span class="k"&gt;vertical&lt;/span&gt; split
Ctrl&lt;span class="p"&gt;-&lt;/span&gt;&lt;span class="k"&gt;w&lt;/span&gt; &lt;span class="k"&gt;o&lt;/span&gt;        → &lt;span class="k"&gt;close&lt;/span&gt; &lt;span class="k"&gt;all&lt;/span&gt; other splits &lt;span class="p"&gt;(&lt;/span&gt;:&lt;span class="k"&gt;only&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="buffers"&gt;Buffers&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;e&lt;/span&gt; filename     → open &lt;span class="k"&gt;file&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; current buffer
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;ls&lt;/span&gt;             → &lt;span class="nb"&gt;list&lt;/span&gt; &lt;span class="k"&gt;all&lt;/span&gt; open &lt;span class="k"&gt;buffers&lt;/span&gt;
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;bn&lt;/span&gt; &lt;span class="sr"&gt;/ :bp       → next /&lt;/span&gt; &lt;span class="k"&gt;previous&lt;/span&gt; buffer
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;b&lt;/span&gt; &lt;span class="p"&gt;&amp;lt;&lt;/span&gt;partial&lt;span class="p"&gt;&amp;gt;&lt;/span&gt;    → switch &lt;span class="k"&gt;to&lt;/span&gt; buffer by partial name &lt;span class="k"&gt;match&lt;/span&gt;
&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="k"&gt;bd&lt;/span&gt;             → &lt;span class="k"&gt;close&lt;/span&gt; buffer
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;:b&lt;/code&gt; with tab completion and partial matching is surprisingly&amp;nbsp;fast: &lt;code&gt;:b pf&amp;lt;Tab&amp;gt;&lt;/code&gt; jumps&amp;nbsp;to &lt;code&gt;pf.conf&lt;/code&gt; if it&amp;#8217;s&amp;nbsp;open.&lt;/p&gt;
&lt;h3 id="jumping-between-recent-files"&gt;Jumping Between Recent&amp;nbsp;Files&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Ctrl&lt;span class="p"&gt;-&lt;/span&gt;&lt;span class="k"&gt;o&lt;/span&gt;  → &lt;span class="k"&gt;jump&lt;/span&gt; back through &lt;span class="k"&gt;jump&lt;/span&gt; &lt;span class="nb"&gt;list&lt;/span&gt;
Ctrl&lt;span class="p"&gt;-&lt;/span&gt;&lt;span class="k"&gt;i&lt;/span&gt;  → &lt;span class="k"&gt;jump&lt;/span&gt; forward
``      → &lt;span class="k"&gt;jump&lt;/span&gt; &lt;span class="k"&gt;to&lt;/span&gt; last position before latest &lt;span class="k"&gt;jump&lt;/span&gt;
`&amp;quot;      → &lt;span class="k"&gt;jump&lt;/span&gt; &lt;span class="k"&gt;to&lt;/span&gt; position when last editing this &lt;span class="k"&gt;file&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;code&gt;Ctrl-o&lt;/code&gt; is the &amp;#8220;undo for navigation.&amp;#8221; Jumped somewhere&amp;nbsp;with &lt;code&gt;*&lt;/code&gt; or &lt;code&gt;gd&lt;/code&gt; and want to go&amp;nbsp;back? &lt;code&gt;Ctrl-o&lt;/code&gt;. Multiple times if&amp;nbsp;needed.&lt;/p&gt;
&lt;h2 id="the-neovim-advantage-lsp-and-treesitter"&gt;The Neovim Advantage: &lt;span class="caps"&gt;LSP&lt;/span&gt; and&amp;nbsp;Treesitter&lt;/h2&gt;
&lt;p&gt;If you&amp;#8217;re still on vanilla Vim, here&amp;#8217;s what you&amp;#8217;re missing. Neovim&amp;#8217;s built-in &lt;span class="caps"&gt;LSP&lt;/span&gt; client and Treesitter integration transform &lt;span class="caps"&gt;YAML&lt;/span&gt;&amp;nbsp;editing:&lt;/p&gt;
&lt;h3 id="yaml-language-server"&gt;&lt;span class="caps"&gt;YAML&lt;/span&gt; Language&amp;nbsp;Server&lt;/h3&gt;
&lt;p&gt;With &lt;code&gt;yaml-language-server&lt;/code&gt; configured, you&amp;nbsp;get:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Schema validation&lt;/strong&gt; - red squiggles when your Kubernetes manifest has a wrong&amp;nbsp;field&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Auto-completion&lt;/strong&gt;&amp;nbsp;- &lt;code&gt;Ctrl-x Ctrl-o&lt;/code&gt; suggests valid keys for your&amp;nbsp;schema&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Hover docs&lt;/strong&gt;&amp;nbsp;- &lt;code&gt;K&lt;/code&gt; shows documentation for the key under&amp;nbsp;cursor&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;A minimal &lt;span class="caps"&gt;LSP&lt;/span&gt; setup&amp;nbsp;in &lt;code&gt;init.lua&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;-- Requires nvim-lspconfig plugin&lt;/span&gt;
&lt;span class="nb"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;lspconfig&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;yamlls&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;setup&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="n"&gt;settings&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="n"&gt;yaml&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
      &lt;span class="n"&gt;schemas&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;https://json.schemastore.org/ansible-playbook&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;playbook*.yml&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
        &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;https://json.schemastore.org/github-workflow&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;.github/workflows/*.yml&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
      &lt;span class="p"&gt;},&lt;/span&gt;
    &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="treesitter"&gt;Treesitter&lt;/h3&gt;
&lt;p&gt;Treesitter gives you syntax-aware text objects and&amp;nbsp;highlighting:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nb"&gt;require&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;nvim-treesitter.configs&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="n"&gt;setup&lt;/span&gt;&lt;span class="p"&gt;({&lt;/span&gt;
  &lt;span class="n"&gt;ensure_installed&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;yaml&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;bash&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;lua&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;json&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;toml&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s1"&gt;&amp;#39;python&amp;#39;&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="n"&gt;highlight&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;enable&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
  &lt;span class="n"&gt;indent&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt; &lt;span class="n"&gt;enable&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="kc"&gt;true&lt;/span&gt; &lt;span class="p"&gt;},&lt;/span&gt;
&lt;span class="p"&gt;})&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With Treesitter, indentation and folding become structure-aware instead of relying on raw indent levels. It&amp;#8217;s the difference between &amp;#8220;this line has 4 spaces&amp;#8221; and &amp;#8220;this is a mapping value inside a sequence&amp;nbsp;item.&amp;#8221;&lt;/p&gt;
&lt;h2 id="practical-combos-the-cheat-sheet"&gt;Practical Combos: The Cheat&amp;nbsp;Sheet&lt;/h2&gt;
&lt;p&gt;A reference for the patterns that come up daily in sysadmin&amp;nbsp;work:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Situation&lt;/th&gt;
&lt;th&gt;Keystrokes&lt;/th&gt;
&lt;th&gt;What happens&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Change a value in quotes&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ci"&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Deletes contents between quotes, enters insert mode&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Delete everything inside braces&lt;/td&gt;
&lt;td&gt;&lt;code&gt;di{&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Clears the block&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Yank a whole &lt;span class="caps"&gt;YAML&lt;/span&gt; section&lt;/td&gt;
&lt;td&gt;&lt;code&gt;yap&lt;/code&gt; or &lt;code&gt;yip&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Direct paragraph yank (no visual mode needed)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Comment 10 lines&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Ctrl-v 9j I# &amp;lt;Esc&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Block&amp;nbsp;insert &lt;code&gt;#&lt;/code&gt; at start of 10 lines&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Uncomment 10 lines&lt;/td&gt;
&lt;td&gt;&lt;code&gt;Ctrl-v 9j ll x&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Block&amp;nbsp;select &lt;code&gt;#&lt;/code&gt; (two columns) and delete&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Replace in selection only&lt;/td&gt;
&lt;td&gt;&lt;code&gt;V&lt;/code&gt; select, &lt;code&gt;:'&amp;lt;,'&amp;gt;s/a/b/g&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Scoped search-replace&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Indent a block more&lt;/td&gt;
&lt;td&gt;&lt;code&gt;V&lt;/code&gt; select, &lt;code&gt;&amp;gt;&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Shift right&amp;nbsp;by &lt;code&gt;shiftwidth&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Sort lines&lt;/td&gt;
&lt;td&gt;&lt;code&gt;V&lt;/code&gt; select, &lt;code&gt;:sort&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Alphabetical sort&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Remove duplicate lines&lt;/td&gt;
&lt;td&gt;&lt;code&gt;V&lt;/code&gt; select, &lt;code&gt;:sort u&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Sort and deduplicate&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Reformat a long line&lt;/td&gt;
&lt;td&gt;&lt;code&gt;gq&lt;/code&gt; + motion&amp;nbsp;(&lt;code&gt;gqip&lt;/code&gt; for paragraph)&lt;/td&gt;
&lt;td&gt;Wrap&amp;nbsp;to &lt;code&gt;textwidth&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Repeat last change everywhere&lt;/td&gt;
&lt;td&gt;&lt;code&gt;n.n.n.&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Search next, apply same edit&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Save file you opened without sudo&lt;/td&gt;
&lt;td&gt;&lt;code&gt;:w !sudo tee %&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Writes via sudo without restarting the editor&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Undo by time&lt;/td&gt;
&lt;td&gt;&lt;code&gt;:earlier 5m&lt;/code&gt; / &lt;code&gt;:later 5m&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Jump to file state 5 minutes ago/ahead&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="wrapping-up"&gt;Wrapping&amp;nbsp;Up&lt;/h2&gt;
&lt;p&gt;Vim rewards investment on a curve - the first week is brutal, the first year is productive, and the next decade is about discovering that you&amp;#8217;ve been doing basic things inefficiently the entire time. The patterns in this article represent the subset that matters most for infrastructure work: efficient navigation, correct yank/paste semantics, scoped replacements, and &lt;span class="caps"&gt;YAML&lt;/span&gt;-specific&amp;nbsp;workflows.&lt;/p&gt;
&lt;p&gt;The best way to internalize these is not to memorize the table above but to deliberately use &lt;em&gt;one new pattern&lt;/em&gt; each day until it becomes muscle memory. Start&amp;nbsp;with &lt;code&gt;V&lt;/code&gt; instead&amp;nbsp;of &lt;code&gt;v&lt;/code&gt;. Then graduate&amp;nbsp;to &lt;code&gt;ci"&lt;/code&gt;. Then text objects. Each one&amp;nbsp;compounds.&lt;/p&gt;
&lt;p&gt;If you want a ready-made Neovim configuration optimized for Ansible, Python, and &lt;span class="caps"&gt;YAML&lt;/span&gt; work, check out &lt;a href="https://codeberg.org/Larvitz/nvim-ansible"&gt;nvim-ansible on Codeberg&lt;/a&gt; - it implements most of the patterns and plugins discussed in this&amp;nbsp;article.&lt;/p&gt;
&lt;p&gt;And if you&amp;#8217;ve been using Vim for fifteen years and just learned&amp;nbsp;that &lt;code&gt;V&lt;/code&gt; solves your paste problems - you&amp;#8217;re in good&amp;nbsp;company.&lt;/p&gt;</content><category term="Linux"/><category term="neovim"/><category term="vim"/><category term="sysadmin"/><category term="devops"/><category term="yaml"/><category term="ansible"/><category term="productivity"/></entry><entry><title>RHEL on ZFS Root: An Unholy Experiment</title><link href="https://hofstede.it/rhel-on-zfs-root-an-unholy-experiment/" rel="alternate"/><published>2026-02-14T00:00:00+01:00</published><updated>2026-02-14T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-02-14:/rhel-on-zfs-root-an-unholy-experiment/</id><summary type="html">&lt;p&gt;Running Red Hat Enterprise Linux on &lt;span class="caps"&gt;ZFS&lt;/span&gt; root is not supported. I did it anyway. Here&amp;#8217;s how this cursed configuration came to be, why you shouldn&amp;#8217;t replicate it, and what the proper alternative looks&amp;nbsp;like.&lt;/p&gt;</summary><content type="html">&lt;p&gt;Some experiments are born from necessity. Others from curiosity. This one was born from a Friday evening, a homelab &lt;span class="caps"&gt;VM&lt;/span&gt; with nothing to lose, and the thought: &amp;#8220;I wonder if convert2rhel will just&amp;#8230; let me do&amp;nbsp;this.&amp;#8221;&lt;/p&gt;
&lt;p&gt;&lt;img alt="RHEL on ZFS" src="https://hofstede.it/images/2026-02-14-rhel-zfs-root.png" title="Red Hat Enterprise Linux 9.7 on ZFS Root - because why not"&gt;&lt;/p&gt;
&lt;p&gt;Let me be absolutely clear from the start: &lt;strong&gt;this is a terrible idea&lt;/strong&gt;. Red Hat does not support &lt;span class="caps"&gt;ZFS&lt;/span&gt; on root. The supported solution in the &lt;span class="caps"&gt;RHEL&lt;/span&gt; ecosystem is Stratis. If you do this in production, you&amp;#8217;re on your own. If you call Red Hat support with &lt;span class="caps"&gt;ZFS&lt;/span&gt; issues, they will politely suggest you reproduce the problem on a supported configuration before they even look at&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;This was purely an experiment for the lolz. Here&amp;#8217;s what I did and why you&amp;nbsp;shouldn&amp;#8217;t.&lt;/p&gt;
&lt;h2 id="the-proof-it-works-for-now"&gt;The Proof It Works (For&amp;nbsp;Now)&lt;/h2&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;[root@testhost ~]# uname -a
Linux testhost 5.14.0-611.16.1.el9_7.x86_64 #1 SMP PREEMPT_DYNAMIC Sun Dec 7 05:52:24 EST 2025 x86_64 x86_64 x86_64 GNU/Linux

[root@testhost ~]# cat /etc/os-release 
NAME=&amp;quot;Red Hat Enterprise Linux&amp;quot;
VERSION=&amp;quot;9.7 (Plow)&amp;quot;
ID=&amp;quot;rhel&amp;quot;
ID_LIKE=&amp;quot;fedora&amp;quot;
VERSION_ID=&amp;quot;9.7&amp;quot;
PRETTY_NAME=&amp;quot;Red Hat Enterprise Linux 9.7 (Plow)&amp;quot;

[root@testhost ~]# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
rpool  67.5G  4.09G  63.4G        -         -     3%     6%  1.00x    ONLINE  -

[root@testhost ~]# zfs list
NAME         USED  AVAIL  REFER  MOUNTPOINT
rpool       4.09G  61.3G    96K  none
rpool/home   176K  61.3G    96K  legacy
rpool/root  4.09G  61.3G  2.65G  /

[root@testhost ~]# lsmod | grep zfs
zfs                  7024640  8
spl                   217088  1 zfs

[root@testhost ~]# dmesg | grep zfs
[    1.466549] zfs: module license &amp;#39;CDDL&amp;#39; taints kernel.
[    1.467131] zfs: module license taints kernel.
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Yes, that&amp;#8217;s genuine &lt;span class="caps"&gt;RHEL&lt;/span&gt; 9.7 running on a &lt;span class="caps"&gt;ZFS&lt;/span&gt; root pool. No, I don&amp;#8217;t recommend&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="the-how-a-tale-of-two-bad-decisions"&gt;The How: A Tale of Two Bad&amp;nbsp;Decisions&lt;/h2&gt;
&lt;p&gt;The path to this abomination was&amp;nbsp;straightforward:&lt;/p&gt;
&lt;h3 id="step-1-rocky-linux-9-on-zfs-root"&gt;Step 1: Rocky Linux 9 on &lt;span class="caps"&gt;ZFS&lt;/span&gt;&amp;nbsp;Root&lt;/h3&gt;
&lt;p&gt;Start with Rocky Linux 9 installed on &lt;span class="caps"&gt;ZFS&lt;/span&gt; root using the excellent &lt;a href="https://openzfs.github.io/openzfs-docs/Getting%20Started/RHEL-based%20distro/Root%20on%20ZFS.html"&gt;OpenZFS documentation&lt;/a&gt;. This guide is well-maintained and produces a working&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;At this point, you have a perfectly functional Rocky Linux system with &lt;span class="caps"&gt;ZFS&lt;/span&gt; root. You could stop here. You &lt;em&gt;should&lt;/em&gt; stop here. I did not stop&amp;nbsp;here.&lt;/p&gt;
&lt;h3 id="step-2-convert2rhel-with-extreme-prejudice"&gt;Step 2: Convert2RHEL (With Extreme&amp;nbsp;Prejudice)&lt;/h3&gt;
&lt;p&gt;This is where things get cursed. Convert2RHEL is designed to convert CentOS, Rocky, or AlmaLinux systems to genuine &lt;span class="caps"&gt;RHEL&lt;/span&gt;. It performs extensive pre-flight checks to ensure the system is in a supported state before&amp;nbsp;conversion.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;ZFS&lt;/span&gt; root is very much &lt;em&gt;not&lt;/em&gt; a supported&amp;nbsp;state.&lt;/p&gt;
&lt;p&gt;The tool correctly identified numerous&amp;nbsp;problems:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Non-standard bootloader configuration (rEFInd instead of &lt;span class="caps"&gt;GRUB&lt;/span&gt; in the usual&amp;nbsp;location)&lt;/li&gt;
&lt;li&gt;Unexpected partition&amp;nbsp;layout&lt;/li&gt;
&lt;li&gt;Missing dracut modules for standard&amp;nbsp;setups&lt;/li&gt;
&lt;li&gt;Various &amp;#8220;are you sure this is a real system?&amp;#8221; sanity&amp;nbsp;checks&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The solution? Disable the checks. All of them. Every single one that stands between you and your bad&amp;nbsp;decision.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;convert2rhel&lt;span class="w"&gt; &lt;/span&gt;--disable-submgr-checks&lt;span class="w"&gt; &lt;/span&gt;--no-rpm-va&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--disablerepo&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;*&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--enablerepo&lt;span class="w"&gt; &lt;/span&gt;rhel-9-for-x86_64-baseos-rpms&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;--enablerepo&lt;span class="w"&gt; &lt;/span&gt;rhel-9-for-x86_64-appstream-rpms
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;That handles some of them. For the rest, I had to patch convert2rhel directly, commenting out validation functions that insisted on standard configurations. If you&amp;#8217;ve ever edited a vendor tool&amp;#8217;s source code to make it stop telling you &amp;#8220;no,&amp;#8221; you know the feeling. It&amp;#8217;s the system administration equivalent of cutting the warning label off a&amp;nbsp;mattress.&lt;/p&gt;
&lt;h3 id="step-3-reboot-and-pray"&gt;Step 3: Reboot and&amp;nbsp;Pray&lt;/h3&gt;
&lt;p&gt;After the conversion completes, you need to manually clean up the remnants of the Rocky release packages. A&amp;nbsp;few &lt;code&gt;rpm -e&lt;/code&gt; invocations,&amp;nbsp;some &lt;code&gt;dnf distro-sync&lt;/code&gt; runs, and a quick check that the &lt;span class="caps"&gt;ZFS&lt;/span&gt; dracut modules are still intact in the initramfs. Nothing too&amp;nbsp;dramatic.&lt;/p&gt;
&lt;p&gt;Then comes the moment of truth. You&amp;nbsp;type &lt;code&gt;reboot&lt;/code&gt;, stare at the console, and&amp;nbsp;wait.&lt;/p&gt;
&lt;p&gt;The rEFInd menu appears. The kernel loads. &lt;span class="caps"&gt;ZFS&lt;/span&gt; imports the pool. Systemd starts doing its thing. And then - a login prompt. &lt;span class="caps"&gt;RHEL&lt;/span&gt; 9.7 on &lt;span class="caps"&gt;ZFS&lt;/span&gt; root, fully subscribed and receiving updates from Red Hat&amp;#8217;s repositories. It actually&amp;nbsp;worked.&lt;/p&gt;
&lt;p&gt;I&amp;#8217;m not going to pretend I wasn&amp;#8217;t&amp;nbsp;surprised.&lt;/p&gt;
&lt;h2 id="why-this-is-a-bad-idea"&gt;Why This Is A Bad&amp;nbsp;Idea&lt;/h2&gt;
&lt;p&gt;Let me count the&amp;nbsp;ways:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;No Support&lt;/strong&gt;: Red Hat will not help you with &lt;span class="caps"&gt;ZFS&lt;/span&gt; issues. Period. The first thing support will ask is &amp;#8220;can you reproduce this on a supported&amp;nbsp;filesystem?&amp;#8221;&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Kernel Updates&lt;/strong&gt;: &lt;span class="caps"&gt;ZFS&lt;/span&gt; is out-of-tree. Every kernel update triggers a &lt;span class="caps"&gt;DKMS&lt;/span&gt; rebuild of the &lt;span class="caps"&gt;ZFS&lt;/span&gt; modules. This usually works, but when it doesn&amp;#8217;t, you&amp;#8217;re debugging it yourself with no root&amp;nbsp;filesystem.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Dracut Complexity&lt;/strong&gt;: The initramfs needs &lt;span class="caps"&gt;ZFS&lt;/span&gt; modules and tools to mount the root pool. Any dracut configuration drift could leave you with an unbootable&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Convert2RHEL Wasn&amp;#8217;t Designed For This&lt;/strong&gt;: By bypassing validation checks, you&amp;#8217;re in uncharted territory. Future convert2rhel updates might break or refuse to&amp;nbsp;run.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;SELinux Contexts&lt;/strong&gt;: &lt;span class="caps"&gt;ZFS&lt;/span&gt; supports xattrs&amp;nbsp;(and &lt;code&gt;xattr=sa&lt;/code&gt; is the default on Linux), so basic SELinux labeling works. But edge cases will bite you: dataset send/receive can lose security labels, full relabels are slow, and some &lt;span class="caps"&gt;ZFS&lt;/span&gt; operations don&amp;#8217;t trigger the expected SELinux hooks. It mostly works, but &amp;#8220;mostly&amp;#8221; is doing a lot of heavy lifting in that&amp;nbsp;sentence.&lt;/p&gt;
&lt;h2 id="the-proper-solution-stratis"&gt;The Proper Solution:&amp;nbsp;Stratis&lt;/h2&gt;
&lt;p&gt;If you want advanced storage features on &lt;span class="caps"&gt;RHEL&lt;/span&gt;, the supported path is &lt;a href="https://stratis-storage.github.io/"&gt;Stratis&lt;/a&gt;. Stratis&amp;nbsp;provides:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Pool-based storage&amp;nbsp;management&lt;/li&gt;
&lt;li&gt;Thin&amp;nbsp;provisioning&lt;/li&gt;
&lt;li&gt;Snapshots&lt;/li&gt;
&lt;li&gt;Encryption&amp;nbsp;support&lt;/li&gt;
&lt;li&gt;Full Red Hat&amp;nbsp;support&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Stratis uses device-mapper for thin provisioning and &lt;span class="caps"&gt;XFS&lt;/span&gt; as the filesystem layer, providing a modern storage management interface while staying within the supported stack. It&amp;#8217;s what Red Hat engineers will recommend when you ask about advanced filesystem&amp;nbsp;features.&lt;/p&gt;
&lt;p&gt;To use Stratis on &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;9:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;stratisd&lt;span class="w"&gt; &lt;/span&gt;stratis-cli
sudo&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;stratisd
sudo&lt;span class="w"&gt; &lt;/span&gt;stratis&lt;span class="w"&gt; &lt;/span&gt;pool&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;mypool&lt;span class="w"&gt; &lt;/span&gt;/dev/sdb
sudo&lt;span class="w"&gt; &lt;/span&gt;stratis&lt;span class="w"&gt; &lt;/span&gt;filesystem&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;mypool&lt;span class="w"&gt; &lt;/span&gt;myfs
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The resulting filesystem can then be added&amp;nbsp;to &lt;code&gt;/etc/fstab&lt;/code&gt; with&amp;nbsp;the &lt;code&gt;x-systemd.requires=stratisd.service&lt;/code&gt; mount&amp;nbsp;option.&lt;/p&gt;
&lt;h2 id="what-actually-works"&gt;What Actually&amp;nbsp;Works&lt;/h2&gt;
&lt;p&gt;Surprisingly, most things just work. Package updates from &lt;span class="caps"&gt;RHEL&lt;/span&gt; repositories install without complaint. Systemd services start and stop like they should. Networking, user management, firewalld - all blissfully unaware that they&amp;#8217;re sitting on a filesystem Oracle has lawyers for. SELinux mostly cooperates too, since &lt;span class="caps"&gt;ZFS&lt;/span&gt; defaults&amp;nbsp;to &lt;code&gt;xattr=sa&lt;/code&gt; on Linux and handles standard labeling&amp;nbsp;fine.&lt;/p&gt;
&lt;h2 id="what-might-not-work"&gt;What Might Not&amp;nbsp;Work&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Major version upgrades&lt;/strong&gt;: Leapp (&lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s in-place upgrade tool) will absolutely not appreciate what you&amp;#8217;ve done to this system. Going to &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10 will require creative problem-solving, or more likely, a fresh&amp;nbsp;install&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rescue mode&lt;/strong&gt;: &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s rescue environments assume your root is on ext4 or &lt;span class="caps"&gt;XFS&lt;/span&gt;. Booting into rescue mode on a &lt;span class="caps"&gt;ZFS&lt;/span&gt; root system gives you a very unhelpful&amp;nbsp;shell&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automated diagnostics&lt;/strong&gt;: Sosreport and friends&amp;nbsp;expect &lt;code&gt;/&lt;/code&gt; to be on a filesystem they recognize. Some checks will simply fail or produce&amp;nbsp;nonsense&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Support tickets&lt;/strong&gt;: &amp;#8220;Can you reproduce this on a supported filesystem?&amp;#8221; will be the response to every case you&amp;nbsp;open&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="the-honest-assessment"&gt;The Honest&amp;nbsp;Assessment&lt;/h2&gt;
&lt;p&gt;This works because the Linux kernel doesn&amp;#8217;t particularly care what filesystem&amp;nbsp;holds &lt;code&gt;/&lt;/code&gt;. As long as the initramfs can mount it and userspace can find its libraries, the system boots. &lt;span class="caps"&gt;ZFS&lt;/span&gt; is mature, well-tested software. It handles the filesystem layer just&amp;nbsp;fine.&lt;/p&gt;
&lt;p&gt;But &amp;#8220;works&amp;#8221; and &amp;#8220;supported&amp;#8221; are very different things. This configuration works today, will probably work tomorrow, and could break spectacularly on the next kernel update. I wouldn&amp;#8217;t run anything I care about on&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Red Hat Enterprise Linux on &lt;span class="caps"&gt;ZFS&lt;/span&gt; root is technically possible. It&amp;#8217;s also a bad idea. I did it because I wanted to see if I could, and because Friday evenings in the homelab are for experiments that would get you fired if you tried them at&amp;nbsp;work.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;re reading this thinking &amp;#8220;I should try this&amp;#8221; - do it on a &lt;span class="caps"&gt;VM&lt;/span&gt; you&amp;#8217;re prepared to destroy. And maybe don&amp;#8217;t mention it in your next job&amp;nbsp;interview.&lt;/p&gt;
&lt;p&gt;For everyone else: Stratis exists, it&amp;#8217;s supported, and Red Hat engineers won&amp;#8217;t quietly judge you for using&amp;nbsp;it.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://openzfs.github.io/openzfs-docs/Getting%20Started/RHEL-based%20distro/Root%20on%20ZFS.html"&gt;OpenZFS: Root on &lt;span class="caps"&gt;ZFS&lt;/span&gt; for &lt;span class="caps"&gt;RHEL&lt;/span&gt;-based&amp;nbsp;distros&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/converting_from_an_rpm-based_linux_distribution_to_rhel/"&gt;Convert2RHEL&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://stratis-storage.github.io/"&gt;Stratis Storage&amp;nbsp;Management&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/managing_file_systems/managing-layered-local-storage-with-stratis_managing-file-systems"&gt;Red Hat Stratis&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;My homelab &lt;span class="caps"&gt;VM&lt;/span&gt; didn&amp;#8217;t deserve this treatment, but it got it anyway. The things we do for a blog&amp;nbsp;post.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="rhel"/><category term="zfs"/><category term="rocky"/><category term="convert2rhel"/><category term="unsupported"/></entry><entry><title>SELinux: A Practical Guide for Fedora and RHEL</title><link href="https://hofstede.it/selinux-a-practical-guide-for-fedora-and-rhel/" rel="alternate"/><published>2026-02-13T00:00:00+01:00</published><updated>2026-02-13T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-02-13:/selinux-a-practical-guide-for-fedora-and-rhel/</id><summary type="html">&lt;p&gt;Moving beyond &amp;#8220;setenforce 0&amp;#8221; - a practical guide to understanding, troubleshooting, and working with SELinux on Fedora and Red Hat Enterprise&amp;nbsp;Linux.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="SELinux" src="https://hofstede.it/images/2026-02-13-selinux-practical-guide.png" title="SELinux Architecture"&gt;&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;ve spent time on Fedora or &lt;span class="caps"&gt;RHEL&lt;/span&gt; systems, you&amp;#8217;ve encountered SELinux. And if you&amp;#8217;re like many administrators, your first instinct when something doesn&amp;#8217;t work is to check if SELinux is the culprit. The temptation to&amp;nbsp;run &lt;code&gt;setenforce 0&lt;/code&gt; is strong - but it&amp;#8217;s also the wrong&amp;nbsp;approach.&lt;/p&gt;
&lt;p&gt;SELinux (Security-Enhanced Linux) implements mandatory access control (&lt;span class="caps"&gt;MAC&lt;/span&gt;) at the kernel level. Unlike traditional Unix permissions (discretionary access control), SELinux policies are enforced regardless of file ownership or traditional permissions. A process running as root can still be blocked from accessing files if the SELinux policy doesn&amp;#8217;t permit&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;This guide focuses on practical skills: understanding what SELinux is doing, troubleshooting denials, and configuring policies correctly. We&amp;#8217;ll cover the concepts you need without getting lost in policy&amp;nbsp;theory.&lt;/p&gt;
&lt;p&gt;Most of the tools in this guide&amp;nbsp;(&lt;code&gt;semanage&lt;/code&gt;, &lt;code&gt;audit2allow&lt;/code&gt;, &lt;code&gt;restorecon&lt;/code&gt;) come from&amp;nbsp;the &lt;code&gt;policycoreutils-python-utils&lt;/code&gt; package, which isn&amp;#8217;t always installed on minimal&amp;nbsp;systems:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;policycoreutils-python-utils&lt;span class="w"&gt; &lt;/span&gt;setroubleshoot-server
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="why-selinux-matters"&gt;Why SELinux&amp;nbsp;Matters&lt;/h2&gt;
&lt;p&gt;SELinux isn&amp;#8217;t security theater. It provides genuine&amp;nbsp;defense-in-depth:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Confinement&lt;/strong&gt;: Even compromised root processes are limited by SELinux policy. A web server exploit that gains root can&amp;#8217;t suddenly read &lt;span class="caps"&gt;SSH&lt;/span&gt; keys or modify system&amp;nbsp;binaries.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Least privilege&lt;/strong&gt;: Policies define exactly what each process type can access. Apache doesn&amp;#8217;t need to&amp;nbsp;read &lt;code&gt;/etc/shadow&lt;/code&gt;, so it&amp;nbsp;can&amp;#8217;t.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Audit trail&lt;/strong&gt;: Every denial is logged. Attempted intrusions leave fingerprints even when&amp;nbsp;blocked.&lt;/p&gt;
&lt;p&gt;The alternative - running in permissive mode or disabling SELinux entirely - removes this protection layer. The goal is to work &lt;em&gt;with&lt;/em&gt; SELinux, not around&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="selinux-modes"&gt;SELinux&amp;nbsp;Modes&lt;/h2&gt;
&lt;p&gt;SELinux operates in three&amp;nbsp;modes:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Mode&lt;/th&gt;
&lt;th&gt;Behavior&lt;/th&gt;
&lt;th&gt;Use Case&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Enforcing&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Policy is enforced, denials logged and blocked&lt;/td&gt;
&lt;td&gt;Production systems&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Permissive&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;Policy checked, denials logged but not blocked&lt;/td&gt;
&lt;td&gt;Debugging, policy development&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;strong&gt;Disabled&lt;/strong&gt;&lt;/td&gt;
&lt;td&gt;SELinux entirely inactive&lt;/td&gt;
&lt;td&gt;Not recommended&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;Check current&amp;nbsp;status:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sestatus
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Output shows the mode, policy name, and other&amp;nbsp;details:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;SELinux status:                 enabled
SELinuxfs mount:                /sys/fs/selinux
SELinux root directory:         /etc/selinux
Loaded policy name:             targeted
Current mode:                   enforcing
Mode from config file:          enforcing
Policy MLS status:              enabled
Policy deny_unknown status:     allowed
Memory protection checking:     actual (secure)
Max kernel policy version:      33
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Temporarily switch&amp;nbsp;modes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Switch to permissive (survives until reboot)&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;setenforce&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;

&lt;span class="c1"&gt;# Return to enforcing&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;setenforce&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For permanent changes,&amp;nbsp;edit &lt;code&gt;/etc/selinux/config&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;SELINUX&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;enforcing&lt;/span&gt;
&lt;span class="na"&gt;SELINUXTYPE&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;targeted&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;targeted&lt;/code&gt; policy is standard on Fedora and &lt;span class="caps"&gt;RHEL&lt;/span&gt; - it confines specific daemons while leaving user processes largely unrestricted. Other policy types exist&amp;nbsp;(&lt;code&gt;mls&lt;/code&gt; for multi-level security) but are&amp;nbsp;niche.&lt;/p&gt;
&lt;h2 id="understanding-selinux-contexts"&gt;Understanding SELinux&amp;nbsp;Contexts&lt;/h2&gt;
&lt;p&gt;Everything in SELinux revolves around &lt;strong&gt;contexts&lt;/strong&gt; (also called labels). Every process, file, socket, and network port has a security context that determines what can access&amp;nbsp;what.&lt;/p&gt;
&lt;h3 id="viewing-contexts"&gt;Viewing&amp;nbsp;Contexts&lt;/h3&gt;
&lt;p&gt;Files and&amp;nbsp;directories:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ls&lt;span class="w"&gt; &lt;/span&gt;-Z&lt;span class="w"&gt; &lt;/span&gt;/var/www/html/index.html
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/html/index.html
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Processes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;ps&lt;span class="w"&gt; &lt;/span&gt;-Z&lt;span class="w"&gt; &lt;/span&gt;-C&lt;span class="w"&gt; &lt;/span&gt;httpd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;LABEL                              PID TTY          TIME CMD
system_u:system_r:httpd_t:s0      1234 ?        00:00:01 httpd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Ports:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;semanage&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;http
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;http_cache_port_t              tcp      8080, 8118, 8123, 10001-10010
http_port_t                    tcp      80, 81, 443, 488, 8008, 8009, 8443, 9000
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="context-format"&gt;Context&amp;nbsp;Format&lt;/h3&gt;
&lt;p&gt;Contexts have four colon-separated&amp;nbsp;fields:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;user:role:type:level
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For practical administration, the &lt;strong&gt;type&lt;/strong&gt; field&amp;nbsp;(&lt;code&gt;httpd_sys_content_t&lt;/code&gt;, &lt;code&gt;httpd_t&lt;/code&gt;) matters most. Types ending&amp;nbsp;in &lt;code&gt;_t&lt;/code&gt; are what policies match&amp;nbsp;against.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;User&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;system_u&lt;/code&gt;, &lt;code&gt;unconfined_u&lt;/code&gt;): SELinux user, maps to Linux&amp;nbsp;users&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Role&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;system_r&lt;/code&gt;, &lt;code&gt;object_r&lt;/code&gt;): Groups types for role-based&amp;nbsp;access&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Type&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;httpd_t&lt;/code&gt;, &lt;code&gt;httpd_sys_content_t&lt;/code&gt;): The primary policy&amp;nbsp;discriminator&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Level&lt;/strong&gt;&amp;nbsp;(&lt;code&gt;s0&lt;/code&gt;): &lt;span class="caps"&gt;MLS&lt;/span&gt;/&lt;span class="caps"&gt;MCS&lt;/span&gt; sensitivity level (usually s0 in targeted&amp;nbsp;policy)&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="how-policy-works"&gt;How Policy&amp;nbsp;Works&lt;/h3&gt;
&lt;p&gt;The policy defines rules&amp;nbsp;like:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;A process with&amp;nbsp;type &lt;code&gt;httpd_t&lt;/code&gt; can read files with&amp;nbsp;type &lt;code&gt;httpd_sys_content_t&lt;/code&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;This is why placing a file&amp;nbsp;in &lt;code&gt;/var/www/html/&lt;/code&gt; doesn&amp;#8217;t automatically make it readable by Apache - the context must match. A file&amp;nbsp;with &lt;code&gt;user_home_t&lt;/code&gt; context won&amp;#8217;t be accessible even if owned&amp;nbsp;by &lt;code&gt;apache:apache&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="common-administrative-tasks"&gt;Common Administrative&amp;nbsp;Tasks&lt;/h2&gt;
&lt;h3 id="changing-file-contexts"&gt;Changing File&amp;nbsp;Contexts&lt;/h3&gt;
&lt;p&gt;When you move or create files outside standard locations, they won&amp;#8217;t have the correct context. This often catches people off&amp;nbsp;guard: &lt;code&gt;cp&lt;/code&gt; inherits the destination directory&amp;#8217;s context,&amp;nbsp;but &lt;code&gt;mv&lt;/code&gt; preserves the original context. Moving a file from your home directory&amp;nbsp;into &lt;code&gt;/var/www/html/&lt;/code&gt; leaves it&amp;nbsp;with &lt;code&gt;user_home_t&lt;/code&gt; - and Apache can&amp;#8217;t read it despite the file being in the right&amp;nbsp;place.&lt;/p&gt;
&lt;p&gt;Two approaches exist to fix&amp;nbsp;contexts:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Temporary (relabel gets reset)&lt;/strong&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;chcon&lt;span class="w"&gt; &lt;/span&gt;-R&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;httpd_sys_content_t&lt;span class="w"&gt; &lt;/span&gt;/srv/www/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Persistent (survives relabels)&lt;/strong&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Define the context mapping&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;fcontext&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;httpd_sys_content_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/srv/www(/.*)?&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# Apply the context&lt;/span&gt;
restorecon&lt;span class="w"&gt; &lt;/span&gt;-Rv&lt;span class="w"&gt; &lt;/span&gt;/srv/www/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;semanage fcontext&lt;/code&gt; command adds rules to the&amp;nbsp;policy. &lt;code&gt;restorecon&lt;/code&gt; applies those rules. This is the correct approach for&amp;nbsp;production.&lt;/p&gt;
&lt;p&gt;List custom context&amp;nbsp;mappings:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;semanage&lt;span class="w"&gt; &lt;/span&gt;fcontext&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;srv
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="managing-port-labels"&gt;Managing Port&amp;nbsp;Labels&lt;/h3&gt;
&lt;p&gt;Running a service on a non-standard port? The port needs the correct&amp;nbsp;label:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Allow Apache to listen on port 8081&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;http_port_t&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8081&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Remove a custom port&amp;nbsp;mapping:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;semanage&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;-d&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;http_port_t&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;8081&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="selinux-booleans"&gt;SELinux&amp;nbsp;Booleans&lt;/h3&gt;
&lt;p&gt;Booleans toggle specific policy behaviors without rewriting policy. They&amp;#8217;re the preferred way to enable common&amp;nbsp;functionality:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# List booleans related to HTTP&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;boolean&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;httpd

&lt;span class="c1"&gt;# Or using getsebool&lt;/span&gt;
getsebool&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;httpd
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Common &lt;span class="caps"&gt;HTTP&lt;/span&gt;&amp;nbsp;booleans:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Allow Apache to connect to network (for reverse proxy)&lt;/span&gt;
setsebool&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;httpd_can_network_connect&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;

&lt;span class="c1"&gt;# Allow Apache to send email&lt;/span&gt;
setsebool&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;httpd_can_sendmail&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;

&lt;span class="c1"&gt;# Allow home directory access&lt;/span&gt;
setsebool&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;httpd_enable_homedirs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;

&lt;span class="c1"&gt;# Allow network relay (for mod_proxy)&lt;/span&gt;
setsebool&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;httpd_can_network_relay&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-P&lt;/code&gt; flag makes changes persistent. Without it, changes reset on&amp;nbsp;reboot.&lt;/p&gt;
&lt;h2 id="troubleshooting-selinux-denials"&gt;Troubleshooting SELinux&amp;nbsp;Denials&lt;/h2&gt;
&lt;p&gt;When something doesn&amp;#8217;t work and you suspect SELinux, here&amp;#8217;s the systematic&amp;nbsp;approach:&lt;/p&gt;
&lt;h3 id="step-1-confirm-its-selinux"&gt;Step 1: Confirm It&amp;#8217;s&amp;nbsp;SELinux&lt;/h3&gt;
&lt;p&gt;Temporarily switch to&amp;nbsp;permissive:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;setenforce&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;0&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If the problem goes away, SELinux is the cause. Switch back to enforcing while you&amp;nbsp;debug:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;setenforce&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="step-2-find-the-denial"&gt;Step 2: Find the&amp;nbsp;Denial&lt;/h3&gt;
&lt;p&gt;SELinux denials are logged&amp;nbsp;to &lt;code&gt;/var/log/audit/audit.log&lt;/code&gt;. The format is dense, but tools&amp;nbsp;help:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check recent denials&lt;/span&gt;
ausearch&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;AVC&lt;span class="w"&gt; &lt;/span&gt;-ts&lt;span class="w"&gt; &lt;/span&gt;recent

&lt;span class="c1"&gt;# More readable format&lt;/span&gt;
ausearch&lt;span class="w"&gt; &lt;/span&gt;-m&lt;span class="w"&gt; &lt;/span&gt;AVC&lt;span class="w"&gt; &lt;/span&gt;-ts&lt;span class="w"&gt; &lt;/span&gt;recent&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;audit2allow&lt;span class="w"&gt; &lt;/span&gt;-w
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Example&amp;nbsp;output:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Type: AVC
Summary:

SELinux is preventing /usr/sbin/httpd from write access on the directory /var/www/uploads.

*****  Plugin catchall_boolean (89.3 confidence) suggests   ******************

If you want to allow httpd to have unified access to all httpd content
Then you must tell SELinux about this by enabling the &amp;#39;httpd_unified&amp;#39; boolean.

Do
setsebool -P httpd_unified 1
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For systems&amp;nbsp;without &lt;code&gt;auditd&lt;/code&gt;,&amp;nbsp;check &lt;code&gt;/var/log/messages&lt;/code&gt; for &lt;code&gt;setroubleshoot&lt;/code&gt; messages.&lt;/p&gt;
&lt;h3 id="step-3-understand-the-denial"&gt;Step 3: Understand the&amp;nbsp;Denial&lt;/h3&gt;
&lt;p&gt;Raw audit&amp;nbsp;entry:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;type=AVC msg=audit(1708032000.123:456): avc:  denied  { write } for  pid=1234 comm=&amp;quot;httpd&amp;quot; name=&amp;quot;uploads&amp;quot; dev=&amp;quot;dm-0&amp;quot; ino=789 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:httpd_sys_content_t:s0 tclass=dir permissive=0
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Key&amp;nbsp;fields:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;scontext&lt;/code&gt;: Source context (the&amp;nbsp;process)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;tcontext&lt;/code&gt;: Target context (what was&amp;nbsp;accessed)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;tclass&lt;/code&gt;: Object class (file, dir,&amp;nbsp;socket)&lt;/li&gt;
&lt;li&gt;&lt;code&gt;{ write }&lt;/code&gt;: The denied&amp;nbsp;permission&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="step-4-fix-the-denial"&gt;Step 4: Fix the&amp;nbsp;Denial&lt;/h3&gt;
&lt;p&gt;Options, from best to&amp;nbsp;worst:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;1. Boolean&lt;/strong&gt; (preferred for common&amp;nbsp;cases):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;audit2allow&lt;span class="w"&gt; &lt;/span&gt;-w&lt;span class="w"&gt; &lt;/span&gt;-a
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Often suggests a boolean. Enable&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;2. Context fix&lt;/strong&gt; (for mislabeled&amp;nbsp;files):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check current context&lt;/span&gt;
ls&lt;span class="w"&gt; &lt;/span&gt;-Z&lt;span class="w"&gt; &lt;/span&gt;/var/www/uploads/

&lt;span class="c1"&gt;# Fix to appropriate type&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;fcontext&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;httpd_sys_rw_content_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/var/www/uploads(/.*)?&amp;quot;&lt;/span&gt;
restorecon&lt;span class="w"&gt; &lt;/span&gt;-Rv&lt;span class="w"&gt; &lt;/span&gt;/var/www/uploads/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;3. Custom policy module&lt;/strong&gt; (for application-specific&amp;nbsp;needs):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Generate policy from audit log&lt;/span&gt;
ausearch&lt;span class="w"&gt; &lt;/span&gt;-c&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s1"&gt;&amp;#39;myapp&amp;#39;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--raw&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;audit2allow&lt;span class="w"&gt; &lt;/span&gt;-M&lt;span class="w"&gt; &lt;/span&gt;myapp

&lt;span class="c1"&gt;# Apply the module&lt;/span&gt;
semodule&lt;span class="w"&gt; &lt;/span&gt;-i&lt;span class="w"&gt; &lt;/span&gt;myapp.pp
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;4. Per-domain permissive&lt;/strong&gt; (temporary&amp;nbsp;workaround):&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Make httpd permissive while keeping system enforcing&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;permissive&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;httpd_t
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Never disable SELinux globally for an application-specific&amp;nbsp;issue.&lt;/p&gt;
&lt;h2 id="selinux-and-containers"&gt;SELinux and&amp;nbsp;Containers&lt;/h2&gt;
&lt;p&gt;Podman and containers have specific SELinux considerations. The container runtime automatically sets up isolation, but volume mounts require&amp;nbsp;attention.&lt;/p&gt;
&lt;h3 id="container-file-labels"&gt;Container File&amp;nbsp;Labels&lt;/h3&gt;
&lt;p&gt;When mounting host directories into containers, SELinux contexts matter. Podman provides flags to handle&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# The :z flag relabels the directory for container use&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/srv/data:/data:z&lt;span class="w"&gt; &lt;/span&gt;nginx

&lt;span class="c1"&gt;# The :Z flag makes the volume private to this container&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;run&lt;span class="w"&gt; &lt;/span&gt;-v&lt;span class="w"&gt; &lt;/span&gt;/srv/data:/data:Z&lt;span class="w"&gt; &lt;/span&gt;nginx
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;:z&lt;/strong&gt; (lowercase): Relabels&amp;nbsp;with &lt;code&gt;container_file_t&lt;/code&gt;, shared among&amp;nbsp;containers&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;:Z&lt;/strong&gt; (uppercase): Private labeling, only this container can&amp;nbsp;access&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;Warning&lt;/strong&gt;: Both flags relabel the &lt;em&gt;host&lt;/em&gt; directory. If another service on the host depends on the original&amp;nbsp;label, &lt;code&gt;:z&lt;/code&gt; or &lt;code&gt;:Z&lt;/code&gt; will break it. Only use these on directories dedicated to container&amp;nbsp;use.&lt;/p&gt;
&lt;h3 id="quadlet-example"&gt;Quadlet&amp;nbsp;Example&lt;/h3&gt;
&lt;p&gt;In Podman Quadlet&amp;nbsp;files:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/library/nginx:latest&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/srv/web:/usr/share/nginx/html:z&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;:z&lt;/code&gt; flag ensures SELinux doesn&amp;#8217;t block the container from reading mounted&amp;nbsp;files.&lt;/p&gt;
&lt;h3 id="checking-container-contexts"&gt;Checking Container&amp;nbsp;Contexts&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Container processes run in container_t&lt;/span&gt;
ps&lt;span class="w"&gt; &lt;/span&gt;-Z&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;container

&lt;span class="c1"&gt;# Files in container volumes should have container_file_t&lt;/span&gt;
ls&lt;span class="w"&gt; &lt;/span&gt;-Z&lt;span class="w"&gt; &lt;/span&gt;/srv/web
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="relabeling-the-filesystem"&gt;Relabeling the&amp;nbsp;Filesystem&lt;/h2&gt;
&lt;p&gt;If contexts get completely mangled (or after restoring from backup), force a full&amp;nbsp;relabel:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create relabel trigger&lt;/span&gt;
touch&lt;span class="w"&gt; &lt;/span&gt;/.autorelabel

&lt;span class="c1"&gt;# Reboot&lt;/span&gt;
reboot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The system relabels all files on next boot. This takes time on large&amp;nbsp;filesystems.&lt;/p&gt;
&lt;p&gt;For targeted&amp;nbsp;relabeling:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;restorecon&lt;span class="w"&gt; &lt;/span&gt;-Rv&lt;span class="w"&gt; &lt;/span&gt;/path/to/directory
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="common-scenarios"&gt;Common&amp;nbsp;Scenarios&lt;/h2&gt;
&lt;h3 id="web-server-with-non-standard-document-root"&gt;Web Server with Non-Standard Document&amp;nbsp;Root&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Set context for new web root&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;fcontext&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;httpd_sys_content_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/srv/www(/.*)?&amp;quot;&lt;/span&gt;
restorecon&lt;span class="w"&gt; &lt;/span&gt;-Rv&lt;span class="w"&gt; &lt;/span&gt;/srv/www

&lt;span class="c1"&gt;# For writable directories (uploads, cache)&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;fcontext&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;httpd_sys_rw_content_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/srv/www/uploads(/.*)?&amp;quot;&lt;/span&gt;
restorecon&lt;span class="w"&gt; &lt;/span&gt;-Rv&lt;span class="w"&gt; &lt;/span&gt;/srv/www/uploads
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="database-with-custom-data-directory"&gt;Database with Custom Data&amp;nbsp;Directory&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# PostgreSQL&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;fcontext&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;postgresql_db_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/srv/pgdata(/.*)?&amp;quot;&lt;/span&gt;
restorecon&lt;span class="w"&gt; &lt;/span&gt;-Rv&lt;span class="w"&gt; &lt;/span&gt;/srv/pgdata

&lt;span class="c1"&gt;# MariaDB/MySQL&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;fcontext&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;mysqld_db_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/srv/mysqldata(/.*)?&amp;quot;&lt;/span&gt;
restorecon&lt;span class="w"&gt; &lt;/span&gt;-Rv&lt;span class="w"&gt; &lt;/span&gt;/srv/mysqldata
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="ssh-on-non-standard-port"&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; on Non-Standard&amp;nbsp;Port&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Add port to ssh_port_t&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;ssh_port_t&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;tcp&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;2222&lt;/span&gt;

&lt;span class="c1"&gt;# Verify&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;port&lt;span class="w"&gt; &lt;/span&gt;-l&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;grep&lt;span class="w"&gt; &lt;/span&gt;ssh
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="home-directory-web-access"&gt;Home Directory Web&amp;nbsp;Access&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Enable userdir module access&lt;/span&gt;
setsebool&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;httpd_enable_homedirs&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;

&lt;span class="c1"&gt;# For specific user home access (more restrictive)&lt;/span&gt;
setsebool&lt;span class="w"&gt; &lt;/span&gt;-P&lt;span class="w"&gt; &lt;/span&gt;httpd_read_user_content&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="useful-commands-reference"&gt;Useful Commands&amp;nbsp;Reference&lt;/h2&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Task&lt;/th&gt;
&lt;th&gt;Command&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Check status&lt;/td&gt;
&lt;td&gt;&lt;code&gt;sestatus&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List file contexts&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ls -Z&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List process contexts&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ps -Z&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Change file context&lt;/td&gt;
&lt;td&gt;&lt;code&gt;chcon -t type file&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Add persistent context&lt;/td&gt;
&lt;td&gt;&lt;code&gt;semanage fcontext -a -t type "/path(/.*)?"&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Apply contexts&lt;/td&gt;
&lt;td&gt;&lt;code&gt;restorecon -Rv /path&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List booleans&lt;/td&gt;
&lt;td&gt;&lt;code&gt;getsebool -a&lt;/code&gt; or &lt;code&gt;semanage boolean -l&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Set boolean&lt;/td&gt;
&lt;td&gt;&lt;code&gt;setsebool -P boolean_name 1&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List port labels&lt;/td&gt;
&lt;td&gt;&lt;code&gt;semanage port -l&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Add port label&lt;/td&gt;
&lt;td&gt;&lt;code&gt;semanage port -a -t type -p tcp port&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Search denials&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ausearch -m AVC -ts recent&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Explain denials&lt;/td&gt;
&lt;td&gt;&lt;code&gt;audit2allow -w -a&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Create policy module&lt;/td&gt;
&lt;td&gt;&lt;code&gt;ausearch --raw \| audit2allow -M name&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Install module&lt;/td&gt;
&lt;td&gt;&lt;code&gt;semodule -i name.pp&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;List modules&lt;/td&gt;
&lt;td&gt;&lt;code&gt;semodule -l&lt;/code&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="best-practices"&gt;Best&amp;nbsp;Practices&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Never disable SELinux globally&lt;/strong&gt;. If an application has issues, troubleshoot the specific&amp;nbsp;denial.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Use booleans first&lt;/strong&gt;. Many common use cases have pre-built&amp;nbsp;toggles.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Prefer context changes over policy changes&lt;/strong&gt;. Labeling files correctly is cleaner than adding policy&amp;nbsp;rules.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Document customizations&lt;/strong&gt;.&amp;nbsp;Track &lt;code&gt;semanage&lt;/code&gt; commands in configuration management (Ansible, Salt) so systems remain&amp;nbsp;reproducible.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Test in permissive first&lt;/strong&gt;. When developing policy, run in permissive to collect all denials before switching to&amp;nbsp;enforcing.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Use &lt;code&gt;audit2allow -w&lt;/code&gt; for explanations&lt;/strong&gt;. Before creating a policy module, understand what you&amp;#8217;re&amp;nbsp;allowing.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;SELinux is a powerful security layer that protects systems even when traditional permissions are bypassed. The learning curve is real, but the skills are essential for Fedora and &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;nbsp;administration.&lt;/p&gt;
&lt;p&gt;The key insight: SELinux isn&amp;#8217;t trying to break your applications. When denials occur, it&amp;#8217;s usually because files or ports are mislabeled, or a boolean needs to be set. The tools exist to diagnose and fix these issues correctly - use them instead of disabling the&amp;nbsp;protection.&lt;/p&gt;
&lt;p&gt;Once you understand contexts, booleans, and the troubleshooting workflow, SELinux becomes a tool you work with rather than an obstacle to work&amp;nbsp;around.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10/html/using_selinux/"&gt;Red Hat SELinux&amp;nbsp;Guide&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man7.org/linux/man-pages/man1/audit2allow.1.html"&gt;audit2allow man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man7.org/linux/man-pages/man8/semanage.8.html"&gt;semanage man&amp;nbsp;page&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;SELinux was born from the &lt;span class="caps"&gt;NSA&lt;/span&gt;&amp;#8217;s research into mandatory access controls - military-grade security that somehow became a default on every Fedora workstation and &lt;span class="caps"&gt;RHEL&lt;/span&gt; server. Most of the time it&amp;#8217;s invisible, doing its job without asking. When it does push back, it&amp;#8217;s specific enough to tell you exactly what went wrong and how to fix it. In a world where privilege escalation makes regular headlines, a kernel-level layer that constrains even root isn&amp;#8217;t an inconvenience - it&amp;#8217;s quiet&amp;nbsp;insurance.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="fedora"/><category term="rhel"/><category term="selinux"/><category term="security"/><category term="containers"/><category term="podman"/></entry><entry><title>Podman 5.8: Quadlet Multi-File Install, Automatic SQLite Migration, and the Road to 6.0</title><link href="https://hofstede.it/podman-58-quadlet-multi-file-install-automatic-sqlite-migration-and-the-road-to-60/" rel="alternate"/><published>2026-02-12T00:00:00+01:00</published><updated>2026-02-12T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-02-12:/podman-58-quadlet-multi-file-install-automatic-sqlite-migration-and-the-road-to-60/</id><summary type="html">&lt;p&gt;Podman 5.8 lands with multi-file Quadlet installs, automatic BoltDB-to-SQLite migration in preparation for Podman 6.0, and several quality-of-life improvements for container&amp;nbsp;workflows.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;&lt;img alt="Logo" src="https://hofstede.it/images/2026-02-12-podman-5-8-release.png" title="Podman Logo"&gt;&lt;/p&gt;
&lt;p&gt;Podman 5.8 dropped this week, and while it&amp;#8217;s not a flashy major release, it lays important groundwork for the upcoming 6.0 transition. The headline features are a significantly improved Quadlet install workflow and an automatic database migration that moves users from BoltDB to SQLite - quietly preparing the ecosystem for the next&amp;nbsp;generation.&lt;/p&gt;
&lt;p&gt;If you&amp;#8217;ve been running Quadlet-based deployments (I wrote about that approach in detail in my &lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;production-grade Podman Quadlets guide&lt;/a&gt;), several of these changes directly improve that&amp;nbsp;workflow.&lt;/p&gt;
&lt;h2 id="quadlet-multi-file-install"&gt;Quadlet Multi-File&amp;nbsp;Install&lt;/h2&gt;
&lt;p&gt;The biggest usability improvement in 5.8 is multi-file support&amp;nbsp;for &lt;code&gt;podman quadlet install&lt;/code&gt;. Previously, installing a multi-container setup meant feeding each Quadlet unit file to the command individually. Now you can bundle multiple Quadlet definitions into a single file, separated&amp;nbsp;by &lt;code&gt;---&lt;/code&gt; delimiters, with each section identified by&amp;nbsp;a &lt;code&gt;# FileName=&amp;lt;name&amp;gt;&lt;/code&gt; header.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# FileName=myapp-db.container&lt;/span&gt;
&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/library/postgres:17&lt;/span&gt;
&lt;span class="na"&gt;...&lt;/span&gt;

&lt;span class="na"&gt;---&lt;/span&gt;

&lt;span class="c1"&gt;# FileName=myapp-web.container&lt;/span&gt;
&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/library/nginx:latest&lt;/span&gt;
&lt;span class="na"&gt;...&lt;/span&gt;

&lt;span class="na"&gt;---&lt;/span&gt;

&lt;span class="c1"&gt;# FileName=myapp.network&lt;/span&gt;
&lt;span class="k"&gt;[Network]&lt;/span&gt;
&lt;span class="na"&gt;Subnet&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;10.89.1.0/24&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This brings Quadlet closer to the &amp;#8220;single manifest&amp;#8221; experience people are used to from Docker Compose or Kubernetes &lt;span class="caps"&gt;YAML&lt;/span&gt;, while retaining the systemd-native integration that makes Quadlet compelling in the first place. For complex deployments with multiple containers, networks, and volumes, this cuts the friction&amp;nbsp;significantly.&lt;/p&gt;
&lt;h2 id="automatic-boltdb-to-sqlite-migration"&gt;Automatic BoltDB to SQLite&amp;nbsp;Migration&lt;/h2&gt;
&lt;p&gt;This is the change that matters most for the long term. Podman has been transitioning its internal database from BoltDB to SQLite, and 5.8 makes this migration automatic: legacy BoltDB databases will convert to SQLite upon system&amp;nbsp;reboot.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# If automatic migration fails, you can trigger it manually&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;system&lt;span class="w"&gt; &lt;/span&gt;migrate&lt;span class="w"&gt; &lt;/span&gt;--migrate-db
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The motivation is straightforward. SQLite offers better concurrent access, more reliable crash recovery, and a proven track record at scale. BoltDB served Podman well, but as container workloads grow more complex - particularly with pods and Quadlet-managed services - the limitations of a simple key-value store become&amp;nbsp;apparent.&lt;/p&gt;
&lt;p&gt;This is explicitly preparation for Podman 6.0, where SQLite will be the only supported backend. If you&amp;#8217;re running 5.8, the migration happens transparently. If something goes wrong, the manual migration command is your&amp;nbsp;fallback.&lt;/p&gt;
&lt;h2 id="apparmor-support-in-quadlet"&gt;AppArmor Support in&amp;nbsp;Quadlet&lt;/h2&gt;
&lt;p&gt;Container Quadlet files now accept&amp;nbsp;an &lt;code&gt;AppArmor&lt;/code&gt; key for configuring container AppArmor profiles directly in the unit&amp;nbsp;definition:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/library/nginx:latest&lt;/span&gt;
&lt;span class="na"&gt;AppArmor&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;my-custom-profile&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For distributions that use AppArmor as their &lt;span class="caps"&gt;MAC&lt;/span&gt; system (Debian, Ubuntu, &lt;span class="caps"&gt;SUSE&lt;/span&gt;), this removes a gap in Quadlet&amp;#8217;s security configuration. Previously, you&amp;#8217;d need to pass AppArmor settings through the less&amp;nbsp;ergonomic &lt;code&gt;PodmanArgs&lt;/code&gt; escape&amp;nbsp;hatch.&lt;/p&gt;
&lt;h2 id="performance-and-usability-improvements"&gt;Performance and Usability&amp;nbsp;Improvements&lt;/h2&gt;
&lt;p&gt;Several smaller changes round out the&amp;nbsp;release:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;podman exec --no-session&lt;/code&gt;&lt;/strong&gt; disables session tracking for exec commands. If you&amp;#8217;re running frequent exec calls against a container - health checks, monitoring scripts, batch operations - the session tracking overhead adds up. This flag trades tracking for raw&amp;nbsp;speed.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;podman update --ulimit&lt;/code&gt;&lt;/strong&gt; lets you modify container ulimits on a running container without recreating it. Useful when you realize your database container needs more open file descriptors and you&amp;#8217;d rather not cycle the whole&amp;nbsp;service.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;VM&lt;/span&gt; artifact path optimization&lt;/strong&gt;&amp;nbsp;improves &lt;code&gt;podman artifact add&lt;/code&gt; when working&amp;nbsp;with &lt;code&gt;podman machine&lt;/code&gt; VMs. Files on shared paths are now loaded directly from the &lt;span class="caps"&gt;VM&lt;/span&gt; filesystem rather than streaming through the &lt;span class="caps"&gt;REST&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt; - a notable speedup for large&amp;nbsp;artifacts.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;&lt;code&gt;podman secret create -&lt;/code&gt;&lt;/strong&gt; no longer requires pipe input. You can now type a secret directly at the terminal prompt, which is more intuitive for interactive&amp;nbsp;use.&lt;/p&gt;
&lt;h2 id="bugfixes"&gt;Bugfixes&lt;/h2&gt;
&lt;p&gt;The release addresses a range of issues across the&amp;nbsp;stack:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Healthcheck start period timing now works correctly with Kubernetes &lt;span class="caps"&gt;YAML&lt;/span&gt;&amp;nbsp;deployments&lt;/li&gt;
&lt;li&gt;Environment variable precedence&amp;nbsp;in &lt;code&gt;kube play&lt;/code&gt; follows the expected&amp;nbsp;order&lt;/li&gt;
&lt;li&gt;Volume mount path handling is more robust across edge&amp;nbsp;cases&lt;/li&gt;
&lt;li&gt;Authentication handling improvements for private&amp;nbsp;registries&lt;/li&gt;
&lt;li&gt;Windows path processing fixes for cross-platform&amp;nbsp;workflows&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="updated-dependencies"&gt;Updated&amp;nbsp;Dependencies&lt;/h2&gt;
&lt;p&gt;Podman 5.8 ships with Buildah v1.43.0, containers/storage v1.62.0, containers/image v5.39.1, and containers/common&amp;nbsp;v0.67.0.&lt;/p&gt;
&lt;h2 id="looking-ahead"&gt;Looking&amp;nbsp;Ahead&lt;/h2&gt;
&lt;p&gt;The SQLite migration signals where Podman is heading. Version 6.0 will likely bring more significant architectural changes, and 5.8 is doing the responsible thing: migrating users incrementally rather than forcing a flag day. If you&amp;#8217;re running Podman in production, updating to 5.8 sooner rather than later gives you a smooth transition&amp;nbsp;path.&lt;/p&gt;
&lt;p&gt;For Quadlet users in particular, the multi-file install support makes managing complex deployments meaningfully easier. Combined with the AppArmor integration and the ongoing maturation of the Quadlet system, Podman continues to build a compelling case as a production container runtime that doesn&amp;#8217;t need a daemon or an orchestrator to be&amp;nbsp;useful.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/containers/podman/releases/tag/v5.8.0"&gt;Podman 5.8.0 Release&amp;nbsp;Notes&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://linuxiac.com/podman-5-8-introduces-quadlet-multi-file-install-and-sqlite-migration/"&gt;Linuxiac: Podman 5.8 Introduces Quadlet Multi-File Install and SQLite&amp;nbsp;Migration&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html"&gt;Podman Quadlet&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/"&gt;Production-Grade Container Deployment with Podman Quadlets&lt;/a&gt; - my earlier deep-dive into Quadlet-based&amp;nbsp;deployments&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="podman"/><category term="containers"/><category term="quadlet"/><category term="systemd"/><category term="linux"/><category term="rhel"/><category term="sqlite"/></entry><entry><title>Immutable Linux Desktops: Universal Blue, OSTree, and the Future of Desktop Linux</title><link href="https://hofstede.it/immutable-linux-desktops-universal-blue-ostree-and-the-future-of-desktop-linux/" rel="alternate"/><published>2026-01-26T00:00:00+01:00</published><updated>2026-01-26T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-01-26:/immutable-linux-desktops-universal-blue-ostree-and-the-future-of-desktop-linux/</id><summary type="html">&lt;p&gt;Exploring atomic desktop Linux distributions, the technology stack behind them, and why Universal Blue&amp;#8217;s Aurora and Bazzite represent a compelling vision for reliable, maintainable desktop&amp;nbsp;systems.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;Traditional Linux desktop installations accumulate state over time. Package upgrades modify system files in place, configuration changes drift from defaults, and after a few years, systems become unique snowflakes that are difficult to reproduce or repair. Atomic desktops take a fundamentally different approach: the operating system is treated as a single, versioned image that can be deployed, rolled back, or replaced&amp;nbsp;entirely.&lt;/p&gt;
&lt;p&gt;This model has proven itself in server infrastructure through CoreOS and similar projects. Now it&amp;#8217;s coming to the desktop through Fedora&amp;#8217;s atomic variants and, more ambitiously, through Universal Blue&amp;#8217;s community-built images like Aurora and&amp;nbsp;Bazzite.&lt;/p&gt;
&lt;h2 id="the-foundation-ostree"&gt;The Foundation:&amp;nbsp;OSTree&lt;/h2&gt;
&lt;p&gt;At the heart of atomic Linux systems lies &lt;a href="https://ostreedev.github.io/ostree/"&gt;OSTree&lt;/a&gt; (also known as libostree), a content-addressed filesystem designed to store and deploy complete bootable operating system trees. Think of it as &amp;#8220;git for operating systems&amp;#8221;. Though the comparison is imperfect, it captures the essential&amp;nbsp;concept.&lt;/p&gt;
&lt;p&gt;OSTree stores filesystem trees as immutable, deduplicated objects identified by their content hash. When you deploy a new system version,&amp;nbsp;OSTree:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Downloads only the changed objects (delta&amp;nbsp;updates)&lt;/li&gt;
&lt;li&gt;Hardlinks unchanged files from the existing&amp;nbsp;deployment&lt;/li&gt;
&lt;li&gt;Creates a new boot entry pointing to the new&amp;nbsp;tree&lt;/li&gt;
&lt;li&gt;Keeps the previous deployment intact for&amp;nbsp;rollback&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The result is atomic updates - the system either fully transitions to the new state or remains unchanged. No more half-completed upgrades leaving your system in an inconsistent state. No more &amp;#8220;let me try rebooting again&amp;#8221; after a failed&amp;nbsp;update.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$ sudo rpm-ostree status
State: idle
Deployments:
  fedora:fedora/43/x86_64/silverblue
                  Version: 43.20260126.0 (2026-01-26T00:26:25Z)
                   Commit: e36246b0837cfe0d1c09e3ee60205ed5be57d57260adc10a625f9353bbef3c40
             GPGSignature: Valid signature by C6E7F081CF80E13146676E88829B606631645531
                     Diff: 522 upgraded, 11 removed, 26 added

● fedora:fedora/43/x86_64/silverblue
                  Version: 43.1.6 (2025-10-23T03:11:18Z)
                   Commit: 4d40d281be93a88f3d559b5756df602f454f932f3c809a6a4250b91049ce40e8
             GPGSignature: Valid signature by C6E7F081CF80E13146676E88829B606631645531
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The bullet point indicates your current deployment. The previous deployment remains available - a&amp;nbsp;single &lt;code&gt;rpm-ostree rollback&lt;/code&gt; and reboot returns you to the exact state you&amp;nbsp;left.&lt;/p&gt;
&lt;h2 id="rpm-ostree-hybrid-package-management"&gt;rpm-ostree: Hybrid Package&amp;nbsp;Management&lt;/h2&gt;
&lt;p&gt;While pure OSTree deployments are completely immutable, desktop users need flexibility. Enter &lt;a href="https://coreos.github.io/rpm-ostree/"&gt;rpm-ostree&lt;/a&gt;, which provides a hybrid model: you get the reliability of image-based deployments while retaining the ability to layer additional &lt;span class="caps"&gt;RPM&lt;/span&gt;&amp;nbsp;packages.&lt;/p&gt;
&lt;p&gt;The base system remains a verified, signed image from your distribution. On top of this, rpm-ostree allows you&amp;nbsp;to:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Layer packages&lt;/strong&gt;: Install additional RPMs that persist across&amp;nbsp;updates&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Override packages&lt;/strong&gt;: Replace base system packages with different&amp;nbsp;versions&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Remove packages&lt;/strong&gt;: Exclude unwanted components from the base&amp;nbsp;image&lt;/li&gt;
&lt;/ul&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Layer a package on top of the base image&lt;/span&gt;
rpm-ostree&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;htop&lt;span class="w"&gt; &lt;/span&gt;neovim

&lt;span class="c1"&gt;# Override a base package with a newer version&lt;/span&gt;
rpm-ostree&lt;span class="w"&gt; &lt;/span&gt;override&lt;span class="w"&gt; &lt;/span&gt;replace&lt;span class="w"&gt; &lt;/span&gt;./updated-package.rpm

&lt;span class="c1"&gt;# Remove a base package you don&amp;#39;t need&lt;/span&gt;
rpm-ostree&lt;span class="w"&gt; &lt;/span&gt;override&lt;span class="w"&gt; &lt;/span&gt;remove&lt;span class="w"&gt; &lt;/span&gt;firefox
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Each modification creates a new deployment, leaving the previous state intact. Updates from upstream are rebased on top of your layered packages, maintaining a coherent system&amp;nbsp;state.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;The key insight&lt;/strong&gt;: your customizations are explicit and tracked. Unlike traditional package managers where the system state is the accumulated result of thousands of operations over time, rpm-ostree maintains a clear separation between the base image and your&amp;nbsp;modifications.&lt;/p&gt;
&lt;h2 id="bootc-container-native-systems"&gt;bootc: Container-Native&amp;nbsp;Systems&lt;/h2&gt;
&lt;p&gt;The current standard in this space is &lt;a href="https://bootc-dev.github.io/bootc/"&gt;bootc&lt;/a&gt;, which takes the logical next step: if containers are the standard way to build and distribute software, why not use them for entire operating&amp;nbsp;systems?&lt;/p&gt;
&lt;p&gt;bootc systems are built as standard &lt;span class="caps"&gt;OCI&lt;/span&gt; container images. The same tools used for application containers - Podman, Docker, Buildah - can build bootable system images. The same registries that host application images can host operating system images. The entire container ecosystem becomes available for &lt;span class="caps"&gt;OS&lt;/span&gt;&amp;nbsp;development.&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;FROM&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;quay.io/fedora/fedora-bootc:43&lt;/span&gt;

&lt;span class="c"&gt;# Install additional packages&lt;/span&gt;
&lt;span class="k"&gt;RUN&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;-y&lt;span class="w"&gt; &lt;/span&gt;htop&lt;span class="w"&gt; &lt;/span&gt;neovim&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;clean&lt;span class="w"&gt; &lt;/span&gt;all

&lt;span class="c"&gt;# Add custom configuration&lt;/span&gt;
&lt;span class="k"&gt;COPY&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;custom-config.conf&lt;span class="w"&gt; &lt;/span&gt;/etc/myapp/

&lt;span class="c"&gt;# Standard container build produces a bootable system&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Building this Containerfile produces a complete, bootable operating system image. You can test it in a &lt;span class="caps"&gt;VM&lt;/span&gt;, push it to a registry, and deploy it to physical hardware. The same image works&amp;nbsp;everywhere.&lt;/p&gt;
&lt;p&gt;bootc uses OSTree under the hood for the actual deployment mechanics, but the authoring experience shifts entirely to container workflows. For organizations already invested in container tooling, this dramatically simplifies system&amp;nbsp;management.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Current state&lt;/strong&gt;: bootc is production-ready for servers and is making rapid progress on desktop support. Universal Blue&amp;#8217;s images are already built using container workflows, positioning them well for the full bootc&amp;nbsp;transition.&lt;/p&gt;
&lt;h2 id="fedora-atomic-desktops"&gt;Fedora Atomic&amp;nbsp;Desktops&lt;/h2&gt;
&lt;p&gt;Fedora provides official atomic desktop variants, each tailored to a specific desktop&amp;nbsp;environment:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Variant&lt;/th&gt;
&lt;th&gt;Desktop Environment&lt;/th&gt;
&lt;th&gt;Target Users&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Fedora Silverblue&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;GNOME&lt;/span&gt;&lt;/td&gt;
&lt;td&gt;General desktop users&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fedora Kinoite&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;KDE&lt;/span&gt; enthusiasts&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fedora Sway Atomic&lt;/td&gt;
&lt;td&gt;Sway&lt;/td&gt;
&lt;td&gt;Tiling &lt;span class="caps"&gt;WM&lt;/span&gt; users&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Fedora Budgie Atomic&lt;/td&gt;
&lt;td&gt;Budgie&lt;/td&gt;
&lt;td&gt;Budgie fans&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;p&gt;These provide the foundation: a stable, tested base with Fedora&amp;#8217;s regular release cadence. They&amp;#8217;re excellent choices for users who want the atomic model with minimal deviation from upstream&amp;nbsp;Fedora.&lt;/p&gt;
&lt;p&gt;However, Fedora&amp;#8217;s official images are intentionally conservative. They include only what&amp;#8217;s necessary for a functional desktop, leaving additional customization to the user through package&amp;nbsp;layering.&lt;/p&gt;
&lt;h2 id="universal-blue-opinionated-images-done-right"&gt;Universal Blue: Opinionated Images Done&amp;nbsp;Right&lt;/h2&gt;
&lt;p&gt;&lt;a href="https://universal-blue.org/"&gt;Universal Blue&lt;/a&gt; takes the Fedora atomic base and builds upon it with thoughtful defaults, additional hardware support, and curated software selections. Rather than starting from scratch, they leverage Fedora&amp;#8217;s extensive testing and add value through careful&amp;nbsp;curation.&lt;/p&gt;
&lt;p&gt;The project produces several image&amp;nbsp;variants:&lt;/p&gt;
&lt;h3 id="aurora"&gt;Aurora&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://getaurora.dev/"&gt;Aurora&lt;/a&gt; is Universal Blue&amp;#8217;s flagship desktop image, based on Fedora Kinoite (&lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma). It targets users who want a polished, productive desktop without manual&amp;nbsp;configuration:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Enhanced hardware support&lt;/strong&gt;: Additional firmware, codecs, and drivers included (UBlue images come with Nvidia drivers pre-installed and&amp;nbsp;signed)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Developer tooling&lt;/strong&gt;: Distrobox pre-configured for containerized development&amp;nbsp;environments&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Quality-of-life improvements&lt;/strong&gt;: Thoughtful defaults and missing pieces filled&amp;nbsp;in&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automatic updates&lt;/strong&gt;: System images update in the background and apply on&amp;nbsp;reboot&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Aurora represents what a modern Linux desktop can be when someone with taste makes the integration decisions. The &lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma experience feels cohesive and&amp;nbsp;complete.&lt;/p&gt;
&lt;h3 id="bazzite"&gt;Bazzite&lt;/h3&gt;
&lt;p&gt;&lt;a href="https://bazzite.gg/"&gt;Bazzite&lt;/a&gt; targets gaming and &lt;span class="caps"&gt;HTPC&lt;/span&gt; use cases, building on the same solid foundation with gaming-specific&amp;nbsp;optimizations:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Steam and Lutris&lt;/strong&gt;: Pre-installed and&amp;nbsp;configured&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Graphics drivers&lt;/strong&gt;: Latest Mesa, Vulkan support, and vendor-specific&amp;nbsp;drivers&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Controller support&lt;/strong&gt;: Out-of-box recognition for a wide range of game&amp;nbsp;controllers&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Performance tuning&lt;/strong&gt;: Kernel parameters and system settings optimized for&amp;nbsp;gaming&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Steam Deck compatibility&lt;/strong&gt;: Specific images for Steam Deck and similar&amp;nbsp;handhelds&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;HTPC&lt;/span&gt; mode&lt;/strong&gt;: Console-like experience for living room&amp;nbsp;setups&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Bazzite solves the perennial &amp;#8220;gaming on Linux&amp;#8221; setup problem. Instead of spending an afternoon configuring drivers, Proton, and various compatibility layers, you boot into a system where everything already&amp;nbsp;works.&lt;/p&gt;
&lt;h3 id="other-variants"&gt;Other&amp;nbsp;Variants&lt;/h3&gt;
&lt;p&gt;Universal Blue also&amp;nbsp;produces:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Bluefin&lt;/strong&gt;: &lt;span class="caps"&gt;GNOME&lt;/span&gt;-based developer workstation&amp;nbsp;image&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;uCore&lt;/strong&gt;: Server-focused images based on Fedora&amp;nbsp;CoreOS&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Framework-specific images&lt;/strong&gt;: Optimized for Framework laptop&amp;nbsp;hardware&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="the-advantages-of-atomic-desktops"&gt;The Advantages of Atomic&amp;nbsp;Desktops&lt;/h2&gt;
&lt;h3 id="reliability-and-stability"&gt;Reliability and&amp;nbsp;Stability&lt;/h3&gt;
&lt;p&gt;The most significant advantage is predictability. Every user running Aurora 43.20260126 has &lt;em&gt;exactly&lt;/em&gt; the same base system. Support becomes tractable - you&amp;#8217;re not debugging the unique accumulated state of someone&amp;#8217;s five-year-old&amp;nbsp;installation.&lt;/p&gt;
&lt;p&gt;Updates are atomic: they either complete successfully or don&amp;#8217;t happen at all. The half-updated, inconsistent states that sometimes plague traditional distributions simply cannot&amp;nbsp;occur.&lt;/p&gt;
&lt;h3 id="painless-rollback"&gt;Painless&amp;nbsp;Rollback&lt;/h3&gt;
&lt;p&gt;Every update preserves the previous system state. If an update causes&amp;nbsp;problems:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Rollback to the previous deployment&lt;/span&gt;
rpm-ostree&lt;span class="w"&gt; &lt;/span&gt;rollback
systemctl&lt;span class="w"&gt; &lt;/span&gt;reboot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After reboot, you&amp;#8217;re running the exact system you had before the update. This isn&amp;#8217;t &amp;#8220;uninstall the broken package and hope&amp;#8221; - it&amp;#8217;s a complete restoration of the previous&amp;nbsp;state.&lt;/p&gt;
&lt;p&gt;This dramatically lowers the risk of updates. You can confidently run the latest packages knowing that recovery is&amp;nbsp;trivial.&lt;/p&gt;
&lt;h3 id="rebasing-switch-distributions-instantly"&gt;Rebasing: Switch Distributions&amp;nbsp;Instantly&lt;/h3&gt;
&lt;p&gt;Perhaps the most surprising capability: you can switch between compatible images without reinstalling. Running Aurora but want to try Bazzite for&amp;nbsp;gaming?&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Rebase to Bazzite&lt;/span&gt;
rpm-ostree&lt;span class="w"&gt; &lt;/span&gt;rebase&lt;span class="w"&gt; &lt;/span&gt;ostree-unverified-registry:ghcr.io/ublue-os/bazzite:latest&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="c1"&gt;# (Select the specific tag for your hardware/GPU)&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;reboot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;After reboot, you&amp;#8217;re running Bazzite. Your home directory, user data, and Flatpak applications remain untouched - only the base system changed. Don&amp;#8217;t like it? Rebase back to&amp;nbsp;Aurora.&lt;/p&gt;
&lt;p&gt;This works across any compatible OSTree-based images. You can move between Fedora&amp;#8217;s official images and Universal Blue&amp;#8217;s variants freely. The operating system becomes as swappable as a container image - because, fundamentally, that&amp;#8217;s what it&amp;nbsp;is.&lt;/p&gt;
&lt;h3 id="separation-of-concerns"&gt;Separation of&amp;nbsp;Concerns&lt;/h3&gt;
&lt;p&gt;Atomic desktops enforce a clean&amp;nbsp;separation:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;System image&lt;/strong&gt;: Managed by OSTree, updated atomically,&amp;nbsp;reproducible&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Applications&lt;/strong&gt;: Installed via Flatpak, containerized, independent of&amp;nbsp;system&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;User data&lt;/strong&gt;: Lives in your home directory, persists across&amp;nbsp;rebases&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Development tools&lt;/strong&gt;: Run in Distrobox containers, isolated from&amp;nbsp;system&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This separation means you can experiment freely with your development environment without risking system stability. Your tooling runs in containers; your &lt;span class="caps"&gt;OS&lt;/span&gt; remains&amp;nbsp;pristine.&lt;/p&gt;
&lt;h3 id="reproducibility"&gt;Reproducibility&lt;/h3&gt;
&lt;p&gt;Need to set up a new machine identically to your current one? The process is&amp;nbsp;trivial:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Install the same base&amp;nbsp;image&lt;/li&gt;
&lt;li&gt;Apply the same layered packages (if&amp;nbsp;any)&lt;/li&gt;
&lt;li&gt;Sync your home&amp;nbsp;directory&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;There&amp;#8217;s no &amp;#8220;reinstall all the packages I&amp;#8217;ve accumulated over five years&amp;#8221; step because your system state is declaratively defined by the image plus your explicit&amp;nbsp;modifications.&lt;/p&gt;
&lt;h2 id="working-with-atomic-desktops"&gt;Working with Atomic&amp;nbsp;Desktops&lt;/h2&gt;
&lt;h3 id="daily-workflow"&gt;Daily&amp;nbsp;Workflow&lt;/h3&gt;
&lt;p&gt;Day-to-day usage feels identical to traditional Linux. You use your applications, edit documents, browse the web. The atomic nature is invisible until you need&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;Updates happen automatically in the background. When a new image is ready, you&amp;#8217;ll see a notification. The update applies on your next reboot - no waiting for package downloads during your&amp;nbsp;workday.&lt;/p&gt;
&lt;h3 id="installing-software"&gt;Installing&amp;nbsp;Software&lt;/h3&gt;
&lt;p&gt;For graphical applications, Flatpak is the primary&amp;nbsp;method:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;flatpak&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;flathub&lt;span class="w"&gt; &lt;/span&gt;org.mozilla.firefox
flatpak&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;flathub&lt;span class="w"&gt; &lt;/span&gt;com.spotify.Client
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Flatpak applications are containerized and independent of the base system. They update separately and work across all Linux&amp;nbsp;distributions.&lt;/p&gt;
&lt;p&gt;For command-line tools and development environments, Distrobox provides seamless container&amp;nbsp;integration:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create a development container&lt;/span&gt;
distrobox&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;--name&lt;span class="w"&gt; &lt;/span&gt;dev&lt;span class="w"&gt; &lt;/span&gt;--image&lt;span class="w"&gt; &lt;/span&gt;fedora:43

&lt;span class="c1"&gt;# Enter the container - it feels like native&lt;/span&gt;
distrobox&lt;span class="w"&gt; &lt;/span&gt;enter&lt;span class="w"&gt; &lt;/span&gt;dev

&lt;span class="c1"&gt;# Inside the container, install whatever you need&lt;/span&gt;
sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;gcc&lt;span class="w"&gt; &lt;/span&gt;cmake&lt;span class="w"&gt; &lt;/span&gt;python3-devel
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Distrobox containers integrate with your desktop - applications appear in your menu, files are accessible, and the experience is nearly&amp;nbsp;transparent.&lt;/p&gt;
&lt;p&gt;Crucially, you can export binaries or &lt;span class="caps"&gt;GUI&lt;/span&gt; apps from the container to the host. This means you can launch tools like &lt;span class="caps"&gt;VS&lt;/span&gt; Code or PyCharm directly from your system dock or application menu, just as if they were installed&amp;nbsp;natively.&lt;/p&gt;
&lt;h3 id="when-to-layer-packages"&gt;When to Layer&amp;nbsp;Packages&lt;/h3&gt;
&lt;p&gt;Some software legitimately needs system integration that Flatpak can&amp;#8217;t&amp;nbsp;provide:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Virtualization&lt;/strong&gt;: libvirt, &lt;span class="caps"&gt;QEMU&lt;/span&gt;,&amp;nbsp;virt-manager&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;VPNs&lt;/strong&gt;: OpenVPN, WireGuard with system&amp;nbsp;integration&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;System services&lt;/strong&gt;: Custom daemons that run at&amp;nbsp;boot&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Kernel modules&lt;/strong&gt;: Out-of-tree drivers (though check if the image already includes&amp;nbsp;them)&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For these cases, rpm-ostree layering is&amp;nbsp;appropriate:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;rpm-ostree&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;libvirt&lt;span class="w"&gt; &lt;/span&gt;virt-manager
systemctl&lt;span class="w"&gt; &lt;/span&gt;reboot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The general guidance: use Flatpak when possible, Distrobox for development, and rpm-ostree layering only when neither alternative&amp;nbsp;works.&lt;/p&gt;
&lt;h2 id="considerations-and-trade-offs"&gt;Considerations and&amp;nbsp;Trade-offs&lt;/h2&gt;
&lt;p&gt;Atomic desktops aren&amp;#8217;t without&amp;nbsp;trade-offs:&lt;/p&gt;
&lt;h3 id="adjustment-period"&gt;Adjustment&amp;nbsp;Period&lt;/h3&gt;
&lt;p&gt;Users accustomed&amp;nbsp;to &lt;code&gt;dnf install whatever&lt;/code&gt; need to adapt their mental model. The system isn&amp;#8217;t broken - it&amp;#8217;s intentionally designed to guide you toward better patterns. This adjustment takes a few days for most&amp;nbsp;users.&lt;/p&gt;
&lt;h3 id="reboot-requirements"&gt;Reboot&amp;nbsp;Requirements&lt;/h3&gt;
&lt;p&gt;System changes require a reboot to take effect. While this ensures atomicity, it can feel inconvenient compared to traditional package managers. In practice, most users update infrequently enough that this isn&amp;#8217;t&amp;nbsp;burdensome.&lt;/p&gt;
&lt;h3 id="layered-package-limits"&gt;Layered Package&amp;nbsp;Limits&lt;/h3&gt;
&lt;p&gt;While you &lt;em&gt;can&lt;/em&gt; layer packages with rpm-ostree, heavy layering defeats the purpose. If you&amp;#8217;re layering fifty packages, you&amp;#8217;re fighting the model. Consider whether those packages truly need system integration or whether Flatpak/Distrobox would&amp;nbsp;work.&lt;/p&gt;
&lt;h3 id="some-niche-software"&gt;Some Niche&amp;nbsp;Software&lt;/h3&gt;
&lt;p&gt;Certain applications expect traditional &lt;span class="caps"&gt;FHS&lt;/span&gt; layouts or assume they can modify system files. Most mainstream software works fine, but occasional edge cases&amp;nbsp;exist.&lt;/p&gt;
&lt;h2 id="getting-started"&gt;Getting&amp;nbsp;Started&lt;/h2&gt;
&lt;h3 id="choosing-an-image"&gt;Choosing an&amp;nbsp;Image&lt;/h3&gt;
&lt;p&gt;For general desktop use with &lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma: &lt;strong&gt;Aurora&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;For gaming or &lt;span class="caps"&gt;HTPC&lt;/span&gt;: &lt;strong&gt;Bazzite&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;For &lt;span class="caps"&gt;GNOME&lt;/span&gt; and developer workflows: &lt;strong&gt;Bluefin&lt;/strong&gt;&lt;/p&gt;
&lt;p&gt;For conservative, upstream-focused experience: &lt;strong&gt;Fedora&amp;nbsp;Silverblue/Kinoite&lt;/strong&gt;&lt;/p&gt;
&lt;h3 id="installation"&gt;Installation&lt;/h3&gt;
&lt;p&gt;Universal Blue images install like any other Linux distribution. Download the &lt;span class="caps"&gt;ISO&lt;/span&gt;, create a bootable &lt;span class="caps"&gt;USB&lt;/span&gt;, and run the installer. The Anaconda installer handles partitioning, user creation, and initial&amp;nbsp;setup.&lt;/p&gt;
&lt;p&gt;Post-installation, enable automatic updates if not already&amp;nbsp;enabled:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;rpm-ostreed-automatic.timer
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="first-steps"&gt;First&amp;nbsp;Steps&lt;/h3&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Explore Flatpak&lt;/strong&gt;: Visit &lt;a href="https://flathub.org/"&gt;Flathub&lt;/a&gt; and install your essential&amp;nbsp;applications&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Set up Distrobox&lt;/strong&gt;: Create a development container for your programming&amp;nbsp;environment&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Configure your desktop&lt;/strong&gt;: &lt;span class="caps"&gt;KDE&lt;/span&gt; Plasma or &lt;span class="caps"&gt;GNOME&lt;/span&gt; settings work exactly as&amp;nbsp;expected&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Verify automatic updates&lt;/strong&gt;:&amp;nbsp;Check &lt;code&gt;rpm-ostree status&lt;/code&gt; to see your current&amp;nbsp;deployment&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="the-future"&gt;The&amp;nbsp;Future&lt;/h2&gt;
&lt;p&gt;The atomic desktop model aligns with where computing is heading. Containers have proven their value for applications; applying the same principles to the operating system itself is the logical&amp;nbsp;evolution.&lt;/p&gt;
&lt;p&gt;bootc represents the next major step, unifying system management with container tooling. Universal Blue is already building toward this future while maintaining stability&amp;nbsp;today.&lt;/p&gt;
&lt;p&gt;For users tired of nursing aging installations, wondering what broke after the last update, or simply wanting a system that stays out of their way, atomic desktops offer a compelling alternative. They&amp;#8217;re not experimental technology - they&amp;#8217;re the result of years of refinement in the server space, now polished for desktop&amp;nbsp;use.&lt;/p&gt;
&lt;p&gt;The best part: if you don&amp;#8217;t like it, you can always rebase&amp;nbsp;back.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://universal-blue.org/"&gt;Universal Blue&lt;/a&gt; - Project&amp;nbsp;home&lt;/li&gt;
&lt;li&gt;&lt;a href="https://getaurora.dev/"&gt;Aurora&lt;/a&gt; - &lt;span class="caps"&gt;KDE&lt;/span&gt;-based desktop&amp;nbsp;image&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bazzite.gg/"&gt;Bazzite&lt;/a&gt; - Gaming-focused&amp;nbsp;image&lt;/li&gt;
&lt;li&gt;&lt;a href="https://ostreedev.github.io/ostree/"&gt;OSTree&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://coreos.github.io/rpm-ostree/"&gt;rpm-ostree&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://bootc-dev.github.io/bootc/"&gt;bootc&amp;nbsp;Project&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://fedoraproject.org/atomic-desktops/"&gt;Fedora Atomic&amp;nbsp;Desktops&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://flathub.org/"&gt;Flathub&lt;/a&gt; - Flatpak application&amp;nbsp;repository&lt;/li&gt;
&lt;li&gt;&lt;a href="https://distrobox.it/"&gt;Distrobox&lt;/a&gt; - Container integration for&amp;nbsp;development&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;Thanks to the Universal Blue community for building these excellent images and to the Fedora team for providing the solid foundation they&amp;#8217;re built&amp;nbsp;upon.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="fedora"/><category term="ostree"/><category term="rpm-ostree"/><category term="bootc"/><category term="universal-blue"/><category term="aurora"/><category term="bazzite"/><category term="atomic"/></entry><entry><title>Reviving Life is Strange: Before the Storm on Modern Linux with a glibc Shim</title><link href="https://hofstede.it/reviving-life-is-strange-before-the-storm-on-modern-linux-with-a-glibc-shim/" rel="alternate"/><published>2026-01-25T00:00:00+01:00</published><updated>2026-01-25T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-01-25:/reviving-life-is-strange-before-the-storm-on-modern-linux-with-a-glibc-shim/</id><summary type="html">&lt;p&gt;Fixing Life is Strange Before the Storm on modern Linux distributions by compiling a small shim library that restores deprecated glibc internal&amp;nbsp;functions.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;Life is Strange: Before the Storm shipped with native Linux support back in 2017. That was a different era - glibc 2.26 was current, and some developers made the unfortunate choice of linking against internal, undocumented glibc symbols. Fast forward to 2026, and the game refuses to start on modern distributions like Fedora 43. The symbols it depends on no longer&amp;nbsp;exist.&lt;/p&gt;
&lt;p&gt;The fix is a small shim library that brings back the missing functions. After implementing it, I played through the entire game without&amp;nbsp;issues.&lt;/p&gt;
&lt;p&gt;&lt;img alt="LiS-BTS" src="https://hofstede.it/images/2026-01-25-life-is-strange-before-the-storm-glibc-fix.png" title="Lige is Strange: Before the Storm on Linux"&gt;&lt;/p&gt;
&lt;h2 id="the-problem-missing-glibc-internals"&gt;The Problem: Missing glibc&amp;nbsp;Internals&lt;/h2&gt;
&lt;p&gt;When you try to launch the game on a modern system, it fails silently or crashes immediately. Running it from the terminal reveals the&amp;nbsp;culprit:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;./LifeIsStrange: symbol lookup error: ./LifeIsStrange: undefined symbol: __libc_dlopen_mode, version GLIBC_PRIVATE
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The game&amp;#8217;s binary was linked&amp;nbsp;against &lt;code&gt;__libc_dlopen_mode&lt;/code&gt; and &lt;code&gt;__libc_dlsym&lt;/code&gt; - internal glibc functions that were never part of the public &lt;span class="caps"&gt;API&lt;/span&gt;. These symbols lived in a special version namespace&amp;nbsp;called &lt;code&gt;GLIBC_PRIVATE&lt;/code&gt;, which is exactly what it sounds like: private implementation details that upstream reserves the right to change at&amp;nbsp;will.&lt;/p&gt;
&lt;p&gt;In glibc 2.34, the dynamic loader was merged into libc proper, and these internal symbols were restructured. The functions still exist internally, but they&amp;#8217;re no longer exported with the same symbol names and versions that older binaries&amp;nbsp;expect.&lt;/p&gt;
&lt;h2 id="the-solution-a-compatibility-shim"&gt;The Solution: A Compatibility&amp;nbsp;Shim&lt;/h2&gt;
&lt;p&gt;Since the internal functions are gone, we provide replacements. The game doesn&amp;#8217;t actually need the internal implementation - it just needs &lt;em&gt;something&lt;/em&gt; that loads shared libraries. The standard &lt;span class="caps"&gt;POSIX&lt;/span&gt;&amp;nbsp;functions &lt;code&gt;dlopen&lt;/code&gt; and &lt;code&gt;dlsym&lt;/code&gt; do exactly&amp;nbsp;that.&lt;/p&gt;
&lt;p&gt;The trick is making our replacement functions appear with the exact symbol names and versions the game expects. &lt;span class="caps"&gt;GCC&lt;/span&gt;&amp;#8217;s &lt;code&gt;__symver__&lt;/code&gt; attribute handles&amp;nbsp;this:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;dlfcn.h&amp;gt;&lt;/span&gt;

&lt;span class="n"&gt;__attribute__&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;__symver__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;__libc_dlopen_mode@GLIBC_PRIVATE&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;libc_dlopen_mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;__attribute__&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;__symver__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;__libc_dlsym@GLIBC_PRIVATE&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;libc_dlsym&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="kr"&gt;restrict&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="kr"&gt;restrict&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;symbol&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dlsym&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;symbol&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;__symver__&lt;/code&gt; attribute tells the linker to export these functions under specific versioned symbol names. When the game&amp;#8217;s dynamic linker searches&amp;nbsp;for &lt;code&gt;__libc_dlopen_mode@GLIBC_PRIVATE&lt;/code&gt;, it finds our shim instead of the missing glibc&amp;nbsp;symbol.&lt;/p&gt;
&lt;h2 id="building-the-shim-library"&gt;Building the Shim&amp;nbsp;Library&lt;/h2&gt;
&lt;p&gt;Create a working directory and add three&amp;nbsp;files:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;libc_dlopen_mode.c:&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;dlfcn.h&amp;gt;&lt;/span&gt;

&lt;span class="n"&gt;__attribute__&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;__symver__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;__libc_dlopen_mode@GLIBC_PRIVATE&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;libc_dlopen_mode&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;int&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dlopen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;flags&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="n"&gt;__attribute__&lt;/span&gt;&lt;span class="p"&gt;((&lt;/span&gt;&lt;span class="n"&gt;__symver__&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;__libc_dlsym@GLIBC_PRIVATE&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)))&lt;/span&gt;
&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;libc_dlsym&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="kt"&gt;void&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="kr"&gt;restrict&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="kr"&gt;restrict&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;symbol&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;dlsym&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;handle&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;symbol&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;version.map:&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;GLIBC_PRIVATE {
global:
    *;
};
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The version script defines our&amp;nbsp;custom &lt;code&gt;GLIBC_PRIVATE&lt;/code&gt; version tag. Without it, the linker wouldn&amp;#8217;t know how to apply the version specified&amp;nbsp;in &lt;code&gt;__symver__&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Makefile:&lt;/strong&gt;&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nf"&gt;.PHONY&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;all&lt;/span&gt; &lt;span class="n"&gt;clean&lt;/span&gt;

&lt;span class="nv"&gt;RES_LIB_NAME&lt;/span&gt;&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;liblibc_dlopen_mode
&lt;span class="nv"&gt;RES_LIB&lt;/span&gt;&lt;span class="w"&gt;         &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;RES_LIB_NAME&lt;span class="k"&gt;)&lt;/span&gt;.so
&lt;span class="nv"&gt;SRCS&lt;/span&gt;&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt;   &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;libc_dlopen_mode.c

&lt;span class="nv"&gt;CFLAGS&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;-fpic&lt;span class="w"&gt; &lt;/span&gt;-shared&lt;span class="w"&gt; &lt;/span&gt;-flto&lt;span class="w"&gt; &lt;/span&gt;-Wl,--version-script&lt;span class="w"&gt; &lt;/span&gt;-Wl,version.map&lt;span class="w"&gt; &lt;/span&gt;-Wl,--as-needed

&lt;span class="nf"&gt;all&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;RES_LIB&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;$(RES_LIB)&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;&lt;span class="nv"&gt;SRCS&lt;/span&gt;&lt;span class="k"&gt;)&lt;/span&gt; &lt;span class="n"&gt;version&lt;/span&gt;.&lt;span class="n"&gt;map&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;CC&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;CFLAGS&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nv"&gt;$@&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;SRCS&lt;span class="k"&gt;)&lt;/span&gt;

&lt;span class="nf"&gt;clean&lt;/span&gt;&lt;span class="o"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;RM&lt;span class="k"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;$(&lt;/span&gt;RES_LIB&lt;span class="k"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Build and&amp;nbsp;install:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;make
sudo&lt;span class="w"&gt; &lt;/span&gt;cp&lt;span class="w"&gt; &lt;/span&gt;liblibc_dlopen_mode.so&lt;span class="w"&gt; &lt;/span&gt;/usr/local/lib/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;You can verify the symbols are correctly&amp;nbsp;versioned:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;$&lt;span class="w"&gt; &lt;/span&gt;nm&lt;span class="w"&gt; &lt;/span&gt;-D&lt;span class="w"&gt; &lt;/span&gt;/usr/local/lib/liblibc_dlopen_mode.so
&lt;span class="w"&gt;                 &lt;/span&gt;w&lt;span class="w"&gt; &lt;/span&gt;__cxa_finalize@GLIBC_2.2.5
&lt;span class="w"&gt;                 &lt;/span&gt;U&lt;span class="w"&gt; &lt;/span&gt;dlopen@GLIBC_2.34
&lt;span class="w"&gt;                 &lt;/span&gt;U&lt;span class="w"&gt; &lt;/span&gt;dlsym@GLIBC_2.34
&lt;span class="m"&gt;0000000000001100&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;T&lt;span class="w"&gt; &lt;/span&gt;libc_dlopen_mode@GLIBC_PRIVATE
&lt;span class="m"&gt;0000000000001120&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;T&lt;span class="w"&gt; &lt;/span&gt;libc_dlsym@GLIBC_PRIVATE
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;T&lt;/code&gt; entries show our exported functions with the&amp;nbsp;correct &lt;code&gt;GLIBC_PRIVATE&lt;/code&gt; version&amp;nbsp;tag.&lt;/p&gt;
&lt;h2 id="configuring-steam"&gt;Configuring&amp;nbsp;Steam&lt;/h2&gt;
&lt;p&gt;Open Steam, right-click on Life is Strange: Before the Storm, and select Properties. In the Launch Options field,&amp;nbsp;add:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;LD_PRELOAD=/usr/local/lib/liblibc_dlopen_mode.so %command%
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;If you&amp;#8217;re using GameMode for performance&amp;nbsp;optimization:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;LD_PRELOAD=/usr/local/lib/liblibc_dlopen_mode.so gamemoderun %command%
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;LD_PRELOAD&lt;/code&gt; environment variable tells the dynamic linker to load our shim library before any others. When the game binary starts and&amp;nbsp;requests &lt;code&gt;__libc_dlopen_mode@GLIBC_PRIVATE&lt;/code&gt;, the linker finds it in our preloaded library instead of failing with an undefined symbol&amp;nbsp;error.&lt;/p&gt;
&lt;h2 id="why-this-works"&gt;Why This&amp;nbsp;Works&lt;/h2&gt;
&lt;p&gt;The game uses these internal functions to dynamically load additional libraries at runtime. The actual implementation details don&amp;#8217;t matter to the game - it just needs &lt;em&gt;a&lt;/em&gt; function that accepts a filename and flags, calls the kernel to map a shared object, and returns a handle. The&amp;nbsp;public &lt;code&gt;dlopen&lt;/code&gt; and &lt;code&gt;dlsym&lt;/code&gt; functions do exactly&amp;nbsp;this.&lt;/p&gt;
&lt;p&gt;Our shim acts as a translation layer: the game calls what it thinks are internal glibc functions, and we redirect those calls to the stable public &lt;span class="caps"&gt;API&lt;/span&gt;. The game can&amp;#8217;t tell the&amp;nbsp;difference.&lt;/p&gt;
&lt;h2 id="other-affected-games"&gt;Other Affected&amp;nbsp;Games&lt;/h2&gt;
&lt;p&gt;Life is Strange: Before the Storm isn&amp;#8217;t alone. Several games from the 2015-2018 era made the same mistake of depending on glibc internals. If you encounter similar &amp;#8220;undefined symbol&amp;#8221; errors&amp;nbsp;mentioning &lt;code&gt;GLIBC_PRIVATE&lt;/code&gt;, this same technique should work - just verify which specific symbols are missing and add corresponding wrapper&amp;nbsp;functions.&lt;/p&gt;
&lt;h2 id="caveats"&gt;Caveats&lt;/h2&gt;
&lt;p&gt;A few things to keep in&amp;nbsp;mind:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;This is a workaround, not a proper fix.&lt;/strong&gt; The correct solution would be for the game developers to rebuild against the public &lt;span class="caps"&gt;API&lt;/span&gt;. Unfortunately, many older Linux ports are&amp;nbsp;abandoned.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The shim intercepts calls globally&lt;/strong&gt; within the game process. This is fine for single-player games but be mindful when&amp;nbsp;using &lt;code&gt;LD_PRELOAD&lt;/code&gt; with any software that handles sensitive&amp;nbsp;data.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Future glibc changes&lt;/strong&gt; could theoretically break even the&amp;nbsp;public &lt;code&gt;dlopen&lt;/code&gt;/&lt;code&gt;dlsym&lt;/code&gt; &lt;span class="caps"&gt;API&lt;/span&gt;, though this is extremely unlikely given &lt;span class="caps"&gt;POSIX&lt;/span&gt; compatibility&amp;nbsp;requirements.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;A few lines of C and a Steam launch option are all it takes to revive this abandoned Linux port. The game runs flawlessly - I completed all three episodes and the bonus Farewell episode without a single&amp;nbsp;crash.&lt;/p&gt;
&lt;p&gt;It&amp;#8217;s a shame that native Linux ports from this era are becoming unplayable due to dependency rot, but at least the fix is straightforward. Sometimes the best compatibility layer is just a thin wrapper that pretends to be something it&amp;nbsp;isn&amp;#8217;t.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://sourceware.org/pipermail/libc-alpha/2021-August/129718.html"&gt;glibc 2.34 Release Notes&lt;/a&gt; - Details on the dynamic loader&amp;nbsp;merge&lt;/li&gt;
&lt;li&gt;&lt;a href="https://store.steampowered.com/app/554620/Life_is_Strange_Before_the_Storm/"&gt;Life is Strange: Before the Storm on&amp;nbsp;Steam&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://gcc.gnu.org/onlinedocs/gcc/Common-Function-Attributes.html"&gt;&lt;span class="caps"&gt;GCC&lt;/span&gt; Symbol Versioning&lt;/a&gt; - Documentation&amp;nbsp;for &lt;code&gt;__symver__&lt;/code&gt; attribute&lt;/li&gt;
&lt;li&gt;&lt;a href="https://man7.org/linux/man-pages/man8/ld.so.8.html"&gt;LD_PRELOAD Trick&lt;/a&gt; - Dynamic linker&amp;nbsp;documentation&lt;/li&gt;
&lt;/ul&gt;</content><category term="Linux"/><category term="linux"/><category term="gaming"/><category term="steam"/><category term="fedora"/><category term="glibc"/><category term="linuxgaming"/></entry><entry><title>Card Wars: Hiding Smartcard Readers from Eager Rust Agents with LD_PRELOAD</title><link href="https://hofstede.it/card-wars-hiding-smartcard-readers-from-eager-rust-agents-with-ld_preload/" rel="alternate"/><published>2026-01-17T00:00:00+01:00</published><updated>2026-01-17T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2026-01-17:/card-wars-hiding-smartcard-readers-from-eager-rust-agents-with-ld_preload/</id><summary type="html">&lt;p&gt;Using LD_PRELOAD to surgically hide a &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; smartcard reader from a Rust-based OpenPGP &lt;span class="caps"&gt;SSH&lt;/span&gt; agent, allowing two incompatible hardware security stacks to coexist peacefully on the same&amp;nbsp;system.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;Running multiple hardware security tokens simultaneously sounds straightforward until you try it. I recently built what I affectionately call a &amp;#8220;Workstation from Hell&amp;#8221;. A Fedora setup with two distinct smartcard ecosystems that absolutely refuse to get&amp;nbsp;along:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Enterprise Layer&lt;/strong&gt;: An &lt;strong&gt;Aventra MyEID&lt;/strong&gt; card (&lt;span class="caps"&gt;PKCS&lt;/span&gt;#15) in the internal laptop reader for Kerberos/&lt;span class="caps"&gt;PKINIT&lt;/span&gt;&amp;nbsp;authentication&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;GPG&lt;/span&gt; Layer&lt;/strong&gt;: A &lt;strong&gt;Nitrokey 3&lt;/strong&gt; (OpenPGP) for &lt;span class="caps"&gt;SSH&lt;/span&gt; and Git signing, managed by the modern Rust-based &lt;a href="https://codeberg.org/openpgp-card/ssh-agent"&gt;openpgp-card-ssh-agent&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The MyEID handles enterprise authentication. The Nitrokey handles developer workflows. In theory, they should coexist without issue. In practice, one of them throws a tantrum every time it sees the&amp;nbsp;other.&lt;/p&gt;
&lt;h2 id="the-problem-aggressive-slot-scanning"&gt;The Problem: Aggressive Slot&amp;nbsp;Scanning&lt;/h2&gt;
&lt;p&gt;The Rust OpenPGP stack is modern, fast, and memory-safe. It&amp;#8217;s also a bit &lt;em&gt;eager&lt;/em&gt;. When the &lt;span class="caps"&gt;SSH&lt;/span&gt; agent starts, it scans all available &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; slots looking for OpenPGP cards - a reasonable approach when you&amp;#8217;re the only smartcard on the&amp;nbsp;system.&lt;/p&gt;
&lt;p&gt;When it encounters my internal Alcor reader containing the MyEID card, things go sideways. The MyEID speaks strict &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15, not OpenPGP. The Rust stack receives unexpected response codes, interprets them as a fatal protocol failure, and&amp;nbsp;crashes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Jan 17 19:42:43 neochristop openpgp-card-ssh-agent[44822]: [2026-01-17T18:42:43Z ERROR ssh_agent_lib::agent] Error handling message: Proto(UnsupportedCommand { command: 27 })
Jan 17 19:42:43 neochristop openpgp-card-ssh-agent[44822]: [2026-01-17T18:42:43Z ERROR ssh_agent_lib::agent] Error handling message: Failure
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The agent doesn&amp;#8217;t offer a configuration option to whitelist specific readers. I needed a way to make the internal reader invisible to the Rust agent without affecting the rest of the system - the Kerberos stack still needs full access to the MyEID&amp;nbsp;card.&lt;/p&gt;
&lt;h2 id="the-solution-ld_preload-interception"&gt;The Solution: LD_PRELOAD&amp;nbsp;Interception&lt;/h2&gt;
&lt;p&gt;If the application won&amp;#8217;t filter readers, we filter them at the system &lt;span class="caps"&gt;API&lt;/span&gt; level. The &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt;&amp;nbsp;uses &lt;code&gt;SCardListReaders&lt;/code&gt; to enumerate available smartcard slots. By intercepting this function, we can surgically remove specific readers from the list before the application sees&amp;nbsp;them.&lt;/p&gt;
&lt;p&gt;The approach is&amp;nbsp;straightforward:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Create a shared library that&amp;nbsp;intercepts &lt;code&gt;SCardListReaders&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Call the real function to get the complete reader&amp;nbsp;list&lt;/li&gt;
&lt;li&gt;Remove any entries matching our filter&amp;nbsp;criteria&lt;/li&gt;
&lt;li&gt;Return the filtered list to the&amp;nbsp;application&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;This keeps the Kerberos stack happy (it sees both readers) while the OpenPGP agent only sees the reader it can actually work&amp;nbsp;with.&lt;/p&gt;
&lt;h2 id="the-shim-library"&gt;The Shim&amp;nbsp;Library&lt;/h2&gt;
&lt;p&gt;Here&amp;#8217;s the C code that makes this&amp;nbsp;work:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="cp"&gt;#define _GNU_SOURCE&lt;/span&gt;
&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;stdio.h&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;stdlib.h&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;string.h&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;dlfcn.h&amp;gt;&lt;/span&gt;
&lt;span class="cp"&gt;#include&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="cpf"&gt;&amp;lt;winscard.h&amp;gt;&lt;/span&gt;

&lt;span class="c1"&gt;// Function pointer matching the PC/SC prototype&lt;/span&gt;
&lt;span class="k"&gt;typedef&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LONG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;SCardListReaders_t&lt;/span&gt;&lt;span class="p"&gt;)(&lt;/span&gt;&lt;span class="n"&gt;SCARDCONTEXT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LPCSTR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LPSTR&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LPDWORD&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="n"&gt;LONG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nf"&gt;SCardListReaders&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SCARDCONTEXT&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;hContext&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LPCSTR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mszGroups&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;                      &lt;/span&gt;&lt;span class="n"&gt;LPSTR&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mszReaders&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;LPDWORD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pcchReaders&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Load and call the original function&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;SCardListReaders_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;original_fn&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;SCardListReaders_t&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="n"&gt;dlsym&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;RTLD_NEXT&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;SCardListReaders&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;LONG&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rv&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;original_fn&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;hContext&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mszGroups&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mszReaders&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pcchReaders&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// If the call failed or we&amp;#39;re just querying buffer size, return immediately&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;rv&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;SCARD_S_SUCCESS&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mszReaders&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;||&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;pcchReaders&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rv&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Filter the multi-string list&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// PC/SC returns readers as &amp;quot;Reader A\0Reader B\0\0&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mszReaders&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mszReaders&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;DWORD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;new_len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;DWORD&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;total_len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;pcchReaders&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;while&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;lt;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;mszReaders&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;total_len&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="kt"&gt;size_t&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;strlen&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;==&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;break&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="c1"&gt;// Double null indicates end of list&lt;/span&gt;

&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="c1"&gt;// FILTER LOGIC: If &amp;quot;Alcor&amp;quot; is in the name, skip it&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;strstr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;Alcor&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="c1"&gt;// Skip this reader (hide it from the application)&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="k"&gt;else&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="c1"&gt;// Keep this reader: shift bytes if necessary&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;                &lt;/span&gt;&lt;span class="n"&gt;memmove&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;            &lt;/span&gt;&lt;span class="n"&gt;new_len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;len&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;+&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Terminate the new list properly&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;writer&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="sc"&gt;&amp;#39;\0&amp;#39;&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="n"&gt;new_len&lt;/span&gt;&lt;span class="o"&gt;++&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Update the length returned to the caller&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;pcchReaders&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;new_len&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;

&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="k"&gt;return&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;rv&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The key insight is that &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; returns reader names as a multi-string buffer, a sequence of null-terminated strings followed by an extra null byte. The filter walks this structure, copying entries that don&amp;#8217;t match the filter pattern while skipping those that&amp;nbsp;do.&lt;/p&gt;
&lt;h2 id="compilation-and-installation"&gt;Compilation and&amp;nbsp;Installation&lt;/h2&gt;
&lt;p&gt;Building the shim requires gcc and the &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; headers. On&amp;nbsp;Fedora:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;sudo&lt;span class="w"&gt; &lt;/span&gt;dnf&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;pcsc-lite-devel&lt;span class="w"&gt; &lt;/span&gt;gcc
mkdir&lt;span class="w"&gt; &lt;/span&gt;-p&lt;span class="w"&gt; &lt;/span&gt;~/.local/lib

gcc&lt;span class="w"&gt; &lt;/span&gt;-shared&lt;span class="w"&gt; &lt;/span&gt;-fPIC&lt;span class="w"&gt; &lt;/span&gt;-o&lt;span class="w"&gt; &lt;/span&gt;~/.local/lib/libhackyreaderfix.so&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;hacky_reader_fix.c&lt;span class="w"&gt; &lt;/span&gt;-ldl&lt;span class="w"&gt; &lt;/span&gt;-I/usr/include/PCSC
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;-shared -fPIC&lt;/code&gt; flags create a position-independent shared library suitable&amp;nbsp;for &lt;code&gt;LD_PRELOAD&lt;/code&gt; injection.&amp;nbsp;The &lt;code&gt;-ldl&lt;/code&gt; links against the dynamic loader library for&amp;nbsp;the &lt;code&gt;dlsym&lt;/code&gt; call.&lt;/p&gt;
&lt;h2 id="injecting-the-shim"&gt;Injecting the&amp;nbsp;Shim&lt;/h2&gt;
&lt;p&gt;I&amp;nbsp;use &lt;code&gt;systemd --user&lt;/code&gt; to manage my &lt;span class="caps"&gt;SSH&lt;/span&gt; agent. To inject the library, create a drop-in&amp;nbsp;override:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;edit&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-ssh-agent.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Add the following&amp;nbsp;content:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;LD_PRELOAD=%h/.local/lib/libhackyreaderfix.so&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;code&gt;%h&lt;/code&gt; specifier expands to the user&amp;#8217;s home directory, keeping the configuration&amp;nbsp;portable.&lt;/p&gt;
&lt;p&gt;After saving, restart the&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;daemon-reload
systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;openpgp-card-ssh-agent
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="the-result"&gt;The&amp;nbsp;Result&lt;/h2&gt;
&lt;p&gt;With the shim in place, the two security stacks are perfectly&amp;nbsp;isolated:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;System/Kerberos&lt;/strong&gt;: Sees both readers, including the internal Alcor with the MyEID card. &lt;span class="caps"&gt;PKINIT&lt;/span&gt; authentication works&amp;nbsp;normally.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; Agent&lt;/strong&gt;: Completely unaware that the Alcor reader exists. Connects directly to the Nitrokey in the external Identiv reader. No more crashes, no more protocol&amp;nbsp;errors.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;You can verify the filtering is working using pcsc_scan (from the pcsc-tools package) to see exactly what the shim&amp;nbsp;exposes:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Without LD_PRELOAD - shows both readers&lt;/span&gt;
pcsc_scan
&lt;span class="c1"&gt;# Reader 0: Identiv uTrust 3512 SAM slot Token [CCID Interface] (55512030603915) 00 00&lt;/span&gt;
&lt;span class="c1"&gt;# Reader 1: Alcor Link AK9563 01 00&lt;/span&gt;
&lt;span class="c1"&gt;# Reader 2: Nitrokey Nitrokey 3 [CCID/ICCD Interface] 02 00&lt;/span&gt;

&lt;span class="c1"&gt;# With LD_PRELOAD - only shows the allowed reader&lt;/span&gt;
&lt;span class="nv"&gt;LD_PRELOAD&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;~/.local/lib/libhackyreaderfix.so&lt;span class="w"&gt; &lt;/span&gt;pcsc_scan
&lt;span class="c1"&gt;# Reader 0: Identiv uTrust 3512 SAM slot Token [CCID Interface] (55512030603915) 00 00&lt;/span&gt;
&lt;span class="c1"&gt;# Reader 1: Nitrokey Nitrokey 3 [CCID/ICCD Interface] 02 00&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="extending-the-filter"&gt;Extending the&amp;nbsp;Filter&lt;/h2&gt;
&lt;p&gt;The current implementation hardcodes &amp;#8220;Alcor&amp;#8221; as the filter pattern. For more flexibility, you could read the pattern from an environment&amp;nbsp;variable:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;const&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="kt"&gt;char&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;getenv&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;PCSC_HIDE_READER&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;span class="k"&gt;if&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;&amp;amp;&amp;amp;&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;strstr&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="n"&gt;filter&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="o"&gt;!=&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;NULL&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="c1"&gt;// Skip this reader&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Then configure it via&amp;nbsp;systemd:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;LD_PRELOAD=%h/.local/lib/libhackyreaderfix.so&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;PCSC_HIDE_READER=Alcor&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This makes the shim reusable across different reader combinations without&amp;nbsp;recompilation.&lt;/p&gt;
&lt;h2 id="why-not-upstream-a-fix"&gt;Why Not Upstream a&amp;nbsp;Fix?&lt;/h2&gt;
&lt;p&gt;The right solution would be for the Rust OpenPGP stack to handle non-OpenPGP cards gracefully. Either by ignoring readers that return unexpected responses or by offering a configuration option to whitelist readers. Both would be reasonable feature&amp;nbsp;requests.&lt;/p&gt;
&lt;p&gt;However, the LD_PRELOAD approach solves the immediate problem without waiting for upstream changes, and it works for any application that misbehaves when confronted with unexpected smartcard readers. It&amp;#8217;s a general-purpose tool for a specific class of&amp;nbsp;problems.&lt;/p&gt;
&lt;h2 id="caveats"&gt;Caveats&lt;/h2&gt;
&lt;p&gt;A few things to keep in&amp;nbsp;mind:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Security implications&lt;/strong&gt;: LD_PRELOAD modifies application behavior at runtime. Only use libraries you control and&amp;nbsp;trust.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Fragility&lt;/strong&gt;: If the &lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt; changes significantly, the shim may need updates. In practice, this &lt;span class="caps"&gt;API&lt;/span&gt; has been stable for&amp;nbsp;decades.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Debugging&lt;/strong&gt;: If something goes wrong, remember the shim is in the path. Temporarily removing the LD_PRELOAD can help isolate&amp;nbsp;issues.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Sometimes the cleanest solution is a dirty hack. LD_PRELOAD interception isn&amp;#8217;t elegant, but it&amp;#8217;s effective. When two pieces of software refuse to cooperate and neither offers the configuration knobs you need, a small shim library can bridge the&amp;nbsp;gap.&lt;/p&gt;
&lt;p&gt;The MyEID handles enterprise authentication. The Nitrokey handles my development workflow. And a few dozen lines of C keep them from stepping on each other&amp;#8217;s&amp;nbsp;toes.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://codeberg.org/openpgp-card/ssh-agent"&gt;openpgp-card-ssh-agent&lt;/a&gt; - Rust-based OpenPGP &lt;span class="caps"&gt;SSH&lt;/span&gt;&amp;nbsp;agent&lt;/li&gt;
&lt;li&gt;&lt;a href="https://pcsclite.apdu.fr/"&gt;&lt;span class="caps"&gt;PC&lt;/span&gt;/&lt;span class="caps"&gt;SC&lt;/span&gt; Lite&lt;/a&gt; - Middleware for smartcard access on Unix&amp;nbsp;systems&lt;/li&gt;
&lt;li&gt;&lt;a href="https://aventra.fi/products/myeid/"&gt;Aventra MyEID&lt;/a&gt; - &lt;span class="caps"&gt;PKCS&lt;/span&gt;#15 compliant&amp;nbsp;smartcards&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.nitrokey.com/products/nitrokeys"&gt;Nitrokey 3&lt;/a&gt; - Open-source hardware security&amp;nbsp;keys&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;The best kind of interoperability hack is one you can forget about. Set it up once, and your hardware security layers peacefully ignore each other&amp;nbsp;forever.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="smartcard"/><category term="security"/><category term="fedora"/></entry><entry><title>Interactive System Troubleshooting with AI: The Linux MCP Server</title><link href="https://hofstede.it/interactive-system-troubleshooting-with-ai-the-linux-mcp-server/" rel="alternate"/><published>2025-12-25T00:00:00+01:00</published><updated>2025-12-25T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-12-25:/interactive-system-troubleshooting-with-ai-the-linux-mcp-server/</id><summary type="html">&lt;p&gt;How the linux-mcp-server bridges &lt;span class="caps"&gt;AI&lt;/span&gt; assistants and Linux systems for interactive diagnostics, enabling natural language troubleshooting and system&amp;nbsp;analysis.&lt;/p&gt;</summary><content type="html">&lt;hr&gt;
&lt;p&gt;Troubleshooting Linux systems traditionally involves a familiar dance: run a command, copy the output, paste it into a chat with an &lt;span class="caps"&gt;AI&lt;/span&gt; assistant, wait for suggestions, repeat. The linux-mcp-server changes this paradigm entirely. Instead of being a passive advisor, your &lt;span class="caps"&gt;AI&lt;/span&gt; assistant becomes an active participant that can directly query your systems, correlate information across multiple data sources, and provide contextual analysis in&amp;nbsp;real-time.&lt;/p&gt;
&lt;p&gt;&lt;img alt="AI System Diagnostics" src="https://hofstede.it/images/2025-12-25-linux-mcp-server.png" title="AI-powered system diagnostics"&gt;&lt;/p&gt;
&lt;h2 id="what-is-mcp"&gt;What is &lt;span class="caps"&gt;MCP&lt;/span&gt;?&lt;/h2&gt;
&lt;p&gt;The Model Context Protocol (&lt;span class="caps"&gt;MCP&lt;/span&gt;) is an open standard introduced by Anthropic in November 2024. It defines how &lt;span class="caps"&gt;AI&lt;/span&gt; models can interact with external tools and data sources in a structured, secure way. Think of it as a standardized &lt;span class="caps"&gt;API&lt;/span&gt; that lets &lt;span class="caps"&gt;AI&lt;/span&gt; assistants reach beyond their training data to access live&amp;nbsp;information.&lt;/p&gt;
&lt;p&gt;&lt;span class="caps"&gt;MCP&lt;/span&gt; servers expose specific capabilities - tools that the &lt;span class="caps"&gt;AI&lt;/span&gt; can invoke when needed. The &lt;span class="caps"&gt;AI&lt;/span&gt; decides when and how to use these tools based on the conversation context, making interactions feel natural rather than&amp;nbsp;scripted.&lt;/p&gt;
&lt;h2 id="enter-linux-mcp-server"&gt;Enter&amp;nbsp;linux-mcp-server&lt;/h2&gt;
&lt;p&gt;The &lt;a href="https://github.com/rhel-lightspeed/linux-mcp-server"&gt;linux-mcp-server&lt;/a&gt; is a Red Hat project that implements &lt;span class="caps"&gt;MCP&lt;/span&gt; for Linux system administration. It provides read-only diagnostic tools&amp;nbsp;covering:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;System information&lt;/strong&gt;: &lt;span class="caps"&gt;OS&lt;/span&gt; details, &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, disk usage, hardware&amp;nbsp;specs&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Service management&lt;/strong&gt;: systemd service status, logs, and&amp;nbsp;configuration&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Process monitoring&lt;/strong&gt;: Running processes with resource&amp;nbsp;metrics&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Log analysis&lt;/strong&gt;: Journal queries, audit logs, application&amp;nbsp;logs&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Network diagnostics&lt;/strong&gt;: Interfaces, connections, listening&amp;nbsp;ports&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Storage analysis&lt;/strong&gt;: Block devices, mount points, directory&amp;nbsp;listings&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The key design principle is safety: all operations are read-only. The server can inspect your system but cannot modify it, making it safe to use even on production&amp;nbsp;systems.&lt;/p&gt;
&lt;h2 id="installation"&gt;Installation&lt;/h2&gt;
&lt;p&gt;The server requires Python 3.10+ and can be installed several&amp;nbsp;ways:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Via pip&lt;/span&gt;
pip&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;linux-mcp-server

&lt;span class="c1"&gt;# Via uv (recommended for isolation)&lt;/span&gt;
uv&lt;span class="w"&gt; &lt;/span&gt;tool&lt;span class="w"&gt; &lt;/span&gt;install&lt;span class="w"&gt; &lt;/span&gt;linux-mcp-server

&lt;span class="c1"&gt;# Or run directly without installing&lt;/span&gt;
uvx&lt;span class="w"&gt; &lt;/span&gt;linux-mcp-server
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;For Claude Desktop or Claude Code integration, add it to your &lt;span class="caps"&gt;MCP&lt;/span&gt;&amp;nbsp;configuration:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;mcpServers&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;linux-mcp-server&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;command&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;uvx&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;&amp;quot;args&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;linux-mcp-server&amp;quot;&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="local-and-remote-systems"&gt;Local and Remote&amp;nbsp;Systems&lt;/h2&gt;
&lt;p&gt;One of the most powerful features is transparent &lt;span class="caps"&gt;SSH&lt;/span&gt; support. Every tool accepts an&amp;nbsp;optional &lt;code&gt;host&lt;/code&gt; parameter. Without it, commands run locally. With it, they execute on the remote system via &lt;span class="caps"&gt;SSH&lt;/span&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;# Query local system
get_system_info()

# Query remote system
get_system_info(host=&amp;quot;webserver.example.com&amp;quot;)
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; authentication uses key-based auth exclusively (no passwords), reading keys&amp;nbsp;from &lt;code&gt;~/.ssh/&lt;/code&gt; or a path specified&amp;nbsp;via &lt;code&gt;LINUX_MCP_SSH_KEY_PATH&lt;/code&gt;. Your&amp;nbsp;existing &lt;code&gt;~/.ssh/config&lt;/code&gt; is respected, so per-host usernames, ports, and jump hosts work&amp;nbsp;automatically.&lt;/p&gt;
&lt;h2 id="real-world-example-diagnosing-a-remote-server"&gt;Real-World Example: Diagnosing a Remote&amp;nbsp;Server&lt;/h2&gt;
&lt;p&gt;Here&amp;#8217;s where things get interesting. Instead of explaining what the tools do, let me demonstrate by querying a real system. The following analysis was performed live on a server named&amp;nbsp;&amp;#8220;janeway&amp;#8221;:&lt;/p&gt;
&lt;h3 id="system-overview"&gt;System&amp;nbsp;Overview&lt;/h3&gt;
&lt;p&gt;When I ask the &lt;span class="caps"&gt;AI&lt;/span&gt; to check the system, it queries the server&amp;nbsp;directly:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;What can you tell me about the janeway&amp;nbsp;server?&amp;#8221;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;Hostname: janeway.home.hofstede.it
Operating System: Red Hat Enterprise Linux 10.1 (Coughlan)
Kernel Version: 6.12.0-124.21.1.el10_1.x86_64
Architecture: x86_64
Uptime: up 5 days, 3 hours, 59 minutes
Boot Time: 2025-12-20 12:54:29

CPU Model: Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz
CPU Cores: 2 logical (1 physical)
Load Average: 0.37, 0.22, 0.13
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This is a &lt;span class="caps"&gt;RHEL&lt;/span&gt; 10.1 &lt;span class="caps"&gt;VM&lt;/span&gt; running on a Proxmox host, with modest resources allocated for its workload. The &lt;span class="caps"&gt;AI&lt;/span&gt; can immediately see this is a stable system - low load averages and over 5 days of&amp;nbsp;uptime.&lt;/p&gt;
&lt;h3 id="service-health-check"&gt;Service Health&amp;nbsp;Check&lt;/h3&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;Is the Omada controller running&amp;nbsp;properly?&amp;#8221;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The server runs a &lt;span class="caps"&gt;TP&lt;/span&gt;-Link Omada network controller as a Podman container managed through systemd&amp;nbsp;Quadlets:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;● omada-controller.service - TP-Link Omada Controller (Podman Quadlet)
     Loaded: loaded (/etc/containers/systemd/omada-controller.container; generated)
     Active: active (running) since Sat 2025-12-20 12:54:56 CET; 5 days ago
   Main PID: 1398 (conmon)
      Tasks: 317 (limit: 22891)
     Memory: 2.1G (peak: 2.1G)
        CPU: 8h 6min 41.871s
     CGroup: /system.slice/omada-controller.service
             ├─libpod-payload-6cbbd984675b...
             │ ├─1413 java -server -Xms128m -Xmx1024m ...
             │ ├─1811 mongod --port 27217 --dbpath ../data/db ...
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;AI&lt;/span&gt; correlates multiple data points: the service is active, has been running since boot (matching the system uptime), and is using about 2.&lt;span class="caps"&gt;1GB&lt;/span&gt; of memory for its Java process and embedded MongoDB. It can also see recent log entries showing normal operation - firmware checks running on schedule, topology updates proceeding without&amp;nbsp;errors.&lt;/p&gt;
&lt;h3 id="resource-analysis"&gt;Resource&amp;nbsp;Analysis&lt;/h3&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;How are resources looking? Any&amp;nbsp;concerns?&amp;#8221;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;=== RAM Information ===
Total: 3.6GB
Available: 1.4GB
Used: 2.2GB (61.7%)
Free: 550.8MB

=== Swap Information ===
Total: 3.9GB
Used: 0.0B (0.0%)
Free: 3.9GB

=== Filesystem Usage ===
Filesystem             Size  Used Avail Use% Mounted on
/dev/mapper/rhel-root   35G  4.0G   31G  12% /
/dev/vda2              960M  448M  513M  47% /boot
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;AI&lt;/span&gt; provides contextual analysis: &amp;#8220;Memory usage at 62% is healthy for a server running a Java application with MongoDB. The fact that swap is completely unused indicates no memory pressure - the system has headroom. Disk usage at 12% on root is excellent, though /boot at 47% might warrant attention during future kernel&amp;nbsp;updates.&amp;#8221;&lt;/p&gt;
&lt;p&gt;This kind of interpretation is where the conversational approach shines. The raw numbers are available, but the &lt;span class="caps"&gt;AI&lt;/span&gt; adds the &amp;#8220;so what&amp;#8221; that helps you understand whether action is&amp;nbsp;needed.&lt;/p&gt;
&lt;h3 id="network-visibility"&gt;Network&amp;nbsp;Visibility&lt;/h3&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;span class="dquo"&gt;&amp;#8220;&lt;/span&gt;What services are exposed on this&amp;nbsp;machine?&amp;#8221;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;=== Listening Ports ===
tcp  LISTEN  0.0.0.0:22     SSH
tcp  LISTEN  0.0.0.0:111    rpcbind
tcp  LISTEN  0.0.0.0:8043   Omada Controller (HTTPS management)
tcp  LISTEN  0.0.0.0:8088   Omada Controller (HTTP redirect)
tcp  LISTEN  0.0.0.0:8843   Omada Controller (HTTPS portal)
tcp  LISTEN  0.0.0.0:29810-29814  Omada device communication
udp  UNCONN  0.0.0.0:29810  Omada device discovery
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;The &lt;span class="caps"&gt;AI&lt;/span&gt; maps the ports to their purposes: &amp;#8220;This is a dedicated Omada controller. &lt;span class="caps"&gt;SSH&lt;/span&gt; is available for administration. The Omada-specific ports (8043, 8088, 8843, 29810-29814) are for the controller&amp;#8217;s web interface and communication with managed network devices. I also notice port 111 (rpcbind) is listening - this is typically for &lt;span class="caps"&gt;NFS&lt;/span&gt; support. If &lt;span class="caps"&gt;NFS&lt;/span&gt; isn&amp;#8217;t actively used, consider whether this service needs to be&amp;nbsp;running.&amp;#8221;&lt;/p&gt;
&lt;h2 id="the-conversational-advantage"&gt;The Conversational&amp;nbsp;Advantage&lt;/h2&gt;
&lt;p&gt;The real power isn&amp;#8217;t in individual commands - it&amp;#8217;s in the conversation. Consider this&amp;nbsp;exchange:&lt;/p&gt;
&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;User&lt;/strong&gt;: &amp;#8220;My web server seems slow. Can you check what&amp;#8217;s going&amp;nbsp;on?&amp;#8221;&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;The &lt;span class="caps"&gt;AI&lt;/span&gt; assistant can&amp;nbsp;now:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Query system load and memory&amp;nbsp;pressure&lt;/li&gt;
&lt;li&gt;Check if the web server service is running&amp;nbsp;properly&lt;/li&gt;
&lt;li&gt;Examine recent logs for&amp;nbsp;errors&lt;/li&gt;
&lt;li&gt;Look at network connections and listening&amp;nbsp;ports&lt;/li&gt;
&lt;li&gt;Correlate all this information and provide a coherent&amp;nbsp;diagnosis&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;All without the user needing to know which specific commands to run or how to interpret the raw output. The assistant contextualizes numbers that might be meaningless to a casual user: &amp;#8220;Memory usage is at 89%, which is high but not critical. However, I notice significant swap activity which could explain the perceived&amp;nbsp;slowness.&amp;#8221;&lt;/p&gt;
&lt;h2 id="security-considerations"&gt;Security&amp;nbsp;Considerations&lt;/h2&gt;
&lt;p&gt;The linux-mcp-server takes a defense-in-depth&amp;nbsp;approach:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Read-only operations&lt;/strong&gt;: No tool can modify system&amp;nbsp;state&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Predefined tools only&lt;/strong&gt;: Arbitrary command execution is not&amp;nbsp;possible&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Log path allowlisting&lt;/strong&gt;: Only explicitly permitted log files can be read (controlled&amp;nbsp;via &lt;code&gt;LINUX_MCP_ALLOWED_LOG_PATHS&lt;/code&gt;)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Parameter validation&lt;/strong&gt;: Inputs are sanitized to prevent injection&amp;nbsp;attacks&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Audit logging&lt;/strong&gt;: All operations are logged for&amp;nbsp;accountability&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;For remote access, standard &lt;span class="caps"&gt;SSH&lt;/span&gt; security applies. The server uses your existing &lt;span class="caps"&gt;SSH&lt;/span&gt; configuration and key-based authentication, inheriting whatever access controls you&amp;#8217;ve already&amp;nbsp;established.&lt;/p&gt;
&lt;h2 id="configuration-options"&gt;Configuration&amp;nbsp;Options&lt;/h2&gt;
&lt;p&gt;Environment variables control server&amp;nbsp;behavior:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Variable&lt;/th&gt;
&lt;th&gt;Purpose&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;LINUX_MCP_ALLOWED_LOG_PATHS&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Comma-separated list of accessible log file paths&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;LINUX_MCP_LOG_LEVEL&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Logging verbosity (&lt;span class="caps"&gt;DEBUG&lt;/span&gt;, &lt;span class="caps"&gt;INFO&lt;/span&gt;, &lt;span class="caps"&gt;WARNING&lt;/span&gt;, &lt;span class="caps"&gt;ERROR&lt;/span&gt;)&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;LINUX_MCP_SSH_KEY_PATH&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Path to &lt;span class="caps"&gt;SSH&lt;/span&gt; private key for remote connections&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;LINUX_MCP_USER&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Default &lt;span class="caps"&gt;SSH&lt;/span&gt; username for remote connections&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="available-tools"&gt;Available&amp;nbsp;Tools&lt;/h2&gt;
&lt;p&gt;The server exposes these diagnostic&amp;nbsp;capabilities:&lt;/p&gt;
&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Tool&lt;/th&gt;
&lt;th&gt;Description&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_system_information&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;OS&lt;/span&gt; version, kernel, hostname, uptime&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_cpu_information&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;CPU&lt;/span&gt; model, cores, current load&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_memory_information&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;RAM&lt;/span&gt; usage, swap, available memory&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_disk_usage&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Filesystem usage, mount points&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_hardware_information&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;&lt;span class="caps"&gt;PCI&lt;/span&gt;/&lt;span class="caps"&gt;USB&lt;/span&gt; devices, &lt;span class="caps"&gt;DMI&lt;/span&gt; data&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;list_services&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;All systemd services and their states&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_service_status&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Detailed status of a specific service&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_service_logs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Journal entries for a service&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;list_processes&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Running processes with resource usage&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_process_info&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Detailed info about a specific &lt;span class="caps"&gt;PID&lt;/span&gt;&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_network_interfaces&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Network interfaces and addresses&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_network_connections&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Active network connections&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_listening_ports&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Services listening on ports&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_journal_logs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Systemd journal queries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;get_audit_logs&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Security audit log entries&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;read_log_file&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Read from allowlisted log files&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;list_block_devices&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Disk and partition information&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;&lt;code&gt;list_files&lt;/code&gt; / &lt;code&gt;list_directories&lt;/code&gt;&lt;/td&gt;
&lt;td&gt;Directory contents with metadata&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;
&lt;h2 id="use-cases"&gt;Use&amp;nbsp;Cases&lt;/h2&gt;
&lt;p&gt;Beyond ad-hoc troubleshooting, the linux-mcp-server&amp;nbsp;enables:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Pre-upgrade assessment&lt;/strong&gt;: &amp;#8220;Check if this server is ready for a major &lt;span class="caps"&gt;OS&lt;/span&gt; upgrade&amp;#8221; - the &lt;span class="caps"&gt;AI&lt;/span&gt; can examine repositories, disk space, running services, and potential compatibility&amp;nbsp;issues.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Capacity planning&lt;/strong&gt;: &amp;#8220;Analyze resource usage trends&amp;#8221; - correlate &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, and disk metrics to identify bottlenecks before they become&amp;nbsp;problems.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Security auditing&lt;/strong&gt;: &amp;#8220;Review what services are exposed to the network&amp;#8221; - examine listening ports, running services, and their&amp;nbsp;configurations.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Documentation&lt;/strong&gt;: &amp;#8220;Summarize this server&amp;#8217;s configuration&amp;#8221; - generate human-readable descriptions of system setup for documentation&amp;nbsp;purposes.&lt;/p&gt;
&lt;h2 id="limitations"&gt;Limitations&lt;/h2&gt;
&lt;p&gt;The current implementation has intentional&amp;nbsp;constraints:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;No write operations&lt;/strong&gt;: You can&amp;#8217;t restart services, edit files, or make changes through the &lt;span class="caps"&gt;MCP&lt;/span&gt;&amp;nbsp;server&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt;/systemd focus&lt;/strong&gt;: Optimized for Red Hat-based distributions; some tools may not work on non-systemd&amp;nbsp;systems&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Smaller models need verification&lt;/strong&gt;: While Claude and other large models provide reliable guidance, smaller open-source models may produce suggestions that require&amp;nbsp;verification&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="getting-started"&gt;Getting&amp;nbsp;Started&lt;/h2&gt;
&lt;ol&gt;
&lt;li&gt;Install the&amp;nbsp;server: &lt;code&gt;pip install linux-mcp-server&lt;/code&gt; or &lt;code&gt;uv tool install linux-mcp-server&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;Configure your &lt;span class="caps"&gt;AI&lt;/span&gt; client (Claude Desktop, Claude Code, or another &lt;span class="caps"&gt;MCP&lt;/span&gt;-compatible&amp;nbsp;client)&lt;/li&gt;
&lt;li&gt;Start asking questions about your&amp;nbsp;systems&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The barrier to entry is remarkably low. If you can &lt;span class="caps"&gt;SSH&lt;/span&gt; to a machine, the linux-mcp-server can query&amp;nbsp;it.&lt;/p&gt;
&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;The linux-mcp-server represents a meaningful shift in how we interact with systems. Instead of the &lt;span class="caps"&gt;AI&lt;/span&gt; being a sophisticated search engine that helps you find the right commands, it becomes a collaborative partner that can actually see your system&amp;#8217;s&amp;nbsp;state.&lt;/p&gt;
&lt;p&gt;This isn&amp;#8217;t about replacing sysadmin knowledge - you still need to understand what the &lt;span class="caps"&gt;AI&lt;/span&gt; is telling you and make informed decisions. But it dramatically lowers the friction of system diagnostics, especially&amp;nbsp;for:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Developers who occasionally need to debug production&amp;nbsp;issues&lt;/li&gt;
&lt;li&gt;Junior administrators learning the&amp;nbsp;ropes&lt;/li&gt;
&lt;li&gt;Anyone managing systems outside their primary&amp;nbsp;expertise&lt;/li&gt;
&lt;li&gt;Experienced admins who want faster initial&amp;nbsp;triage&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The project is open source and actively developed. Contributions for additional diagnostic tools, support for other distributions, and integration with documentation via &lt;span class="caps"&gt;RAG&lt;/span&gt; are&amp;nbsp;welcome.&lt;/p&gt;
&lt;hr&gt;
&lt;h2 id="references"&gt;References&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://github.com/rhel-lightspeed/linux-mcp-server"&gt;linux-mcp-server on&amp;nbsp;GitHub&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://fedoramagazine.org/find-out-how-your-fedora-system-really-feels-with-the-linux-mcp-server/"&gt;Fedora Magazine: Find out how your Fedora system really&amp;nbsp;feels&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://modelcontextprotocol.io/"&gt;Model Context Protocol&amp;nbsp;specification&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.anthropic.com/news/model-context-protocol"&gt;Anthropic&amp;#8217;s &lt;span class="caps"&gt;MCP&lt;/span&gt;&amp;nbsp;announcement&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;
&lt;p&gt;Thanks to the Red Hat and Fedora teams for developing linux-mcp-server and making &lt;span class="caps"&gt;AI&lt;/span&gt;-assisted system administration a reality. The future of troubleshooting is&amp;nbsp;conversational.&lt;/p&gt;</content><category term="Linux"/><category term="linux"/><category term="ai"/><category term="mcp"/><category term="troubleshooting"/><category term="fedora"/><category term="rhel"/><category term="claude"/></entry><entry><title>Production-Grade Container Deployment with Podman Quadlets</title><link href="https://hofstede.it/production-grade-container-deployment-with-podman-quadlets/" rel="alternate"/><published>2025-11-16T00:00:00+01:00</published><updated>2025-11-16T00:00:00+01:00</updated><author><name>Larvitz</name></author><id>tag:blog.hofstede.it,2025-11-16:/production-grade-container-deployment-with-podman-quadlets/</id><summary type="html">&lt;p&gt;A Practical Guide to build secure, maintainable multi-container applications with&amp;nbsp;Podman&lt;/p&gt;</summary><content type="html">&lt;p&gt;&lt;em&gt;Update, 2026-04-27: an earlier version of this article described Podman&amp;#8217;s secret store as providing encrypted storage. With the&amp;nbsp;default &lt;code&gt;file&lt;/code&gt; driver it does not: secrets are base64-encoded with filesystem permissions as the only protection. The relevant section has been corrected. See &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/#secrets-management"&gt;Podman in Production: Secrets Management&lt;/a&gt; for what the default driver actually buys&amp;nbsp;you.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="introduction"&gt;Introduction&lt;/h2&gt;
&lt;p&gt;Containers have become the de facto standard for application deployment, but the conversation often jumps straight to Kubernetes when discussing production workloads. While K8s excels at large-scale orchestration, many production services don&amp;#8217;t require that level of complexity. For single-host or small-scale deployments, a well-architected Podman setup with systemd integration can provide robust, secure, and maintainable&amp;nbsp;infrastructure.&lt;/p&gt;
&lt;p&gt;This article demonstrates a production-grade container deployment using &lt;strong&gt;Red Hat Enterprise Linux 10&lt;/strong&gt;, &lt;strong&gt;Podman Quadlets&lt;/strong&gt;, and &lt;strong&gt;Traefik&lt;/strong&gt; as a reverse proxy. We&amp;#8217;ll walk through deploying Forgejo (a self-hosted Git service) as a practical example, covering the technical implementation and the architectural decisions behind&amp;nbsp;it.&lt;/p&gt;
&lt;p&gt;&lt;img alt="Some shipping containers" src="https://hofstede.it/images/2025-11-16-container_forgejo_podman_quadlet.webp" title="Container Image"&gt;&lt;/p&gt;
&lt;h2 id="why-this-approach"&gt;Why This&amp;nbsp;Approach?&lt;/h2&gt;
&lt;p&gt;Before diving into the implementation, let&amp;#8217;s address the fundamental design&amp;nbsp;decisions:&lt;/p&gt;
&lt;h3 id="podman-over-docker"&gt;Podman over&amp;nbsp;Docker&lt;/h3&gt;
&lt;p&gt;Red Hat has made Podman the standard container runtime in &lt;span class="caps"&gt;RHEL&lt;/span&gt; for compelling technical and security reasons. This isn&amp;#8217;t just vendor preference - it represents fundamental architectural&amp;nbsp;improvements:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Daemonless architecture&lt;/strong&gt;: No privileged daemon running as root, reducing the attack surface significantly. Each container runs as a direct child of systemd or the user&amp;nbsp;session.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rootless containers&lt;/strong&gt;: Native support for running containers as unprivileged users - a first-class feature, not a&amp;nbsp;bolt-on&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;systemd integration&lt;/strong&gt;: First-class integration with the init system that already manages your services. This is particularly powerful in &lt;span class="caps"&gt;RHEL&lt;/span&gt; environments where systemd&amp;#8217;s maturity and tooling are&amp;nbsp;well-established.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;OCI&lt;/span&gt; compliance&lt;/strong&gt;: Full compatibility with Docker images and registries - your existing container images work without&amp;nbsp;modification&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Pod support&lt;/strong&gt;: Kubernetes-style pod concepts for grouping containers, making the transition to K8s smoother if&amp;nbsp;needed&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Fork/exec model&lt;/strong&gt;: Unlike Docker&amp;#8217;s client-server architecture, Podman uses traditional Unix fork/exec, making it more auditable and debuggable with standard&amp;nbsp;tools&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;From an enterprise perspective, Podman aligns with &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s security-first philosophy. SELinux, user namespaces, and cgroups v2 integration are not afterthoughts but core design&amp;nbsp;elements.&lt;/p&gt;
&lt;h3 id="quadlets-over-compose"&gt;Quadlets over&amp;nbsp;Compose&lt;/h3&gt;
&lt;p&gt;Traditional Docker Compose files are imperative and require a separate daemon. Podman Quadlets leverage systemd&amp;#8217;s unit file format,&amp;nbsp;providing:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Declarative configuration&lt;/strong&gt;: Define the desired state, let systemd handle the&amp;nbsp;lifecycle&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Native service management&lt;/strong&gt;: Use&amp;nbsp;familiar &lt;code&gt;systemctl&lt;/code&gt; commands - the same tooling administrators already&amp;nbsp;know&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Dependency management&lt;/strong&gt;: Leverage systemd&amp;#8217;s robust dependency graph&amp;nbsp;(&lt;code&gt;After=&lt;/code&gt;, &lt;code&gt;Requires=&lt;/code&gt;,&amp;nbsp;etc.)&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automatic updates&lt;/strong&gt;: Built-in support for container image updates&amp;nbsp;via &lt;code&gt;podman-auto-update.timer&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Resource control&lt;/strong&gt;: Direct access to systemd&amp;#8217;s cgroup integration for &lt;span class="caps"&gt;CPU&lt;/span&gt;, memory, and I/O&amp;nbsp;limits&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Journal integration&lt;/strong&gt;: All container logs automatically flow to journald, integrating with existing log&amp;nbsp;infrastructure&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="network-segmentation-by-design"&gt;Network Segmentation by&amp;nbsp;Design&lt;/h3&gt;
&lt;p&gt;Proper network isolation is crucial for&amp;nbsp;security:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Frontend network&lt;/strong&gt;: IPv6-enabled network for Traefik and application&amp;nbsp;containers&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Backend networks&lt;/strong&gt;: Isolated networks for database&amp;nbsp;communication&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No unnecessary exposure&lt;/strong&gt;: Database containers never touch the frontend&amp;nbsp;network&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="the-architecture"&gt;The&amp;nbsp;Architecture&lt;/h2&gt;
&lt;p&gt;Our example deployment consists of three&amp;nbsp;components:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Forgejo&lt;/strong&gt; - The Git service&amp;nbsp;application&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;PostgreSQL&lt;/strong&gt; - The database&amp;nbsp;backend&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Traefik&lt;/strong&gt; - The reverse proxy handling &lt;span class="caps"&gt;TLS&lt;/span&gt; and&amp;nbsp;routing&lt;/li&gt;
&lt;/ol&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;                    Internet
                       │
                       ▼
                   Traefik (Port 443)
                       │
          ┌────────────┴────────────┐
          │    frontend network     │
          │    (10.89.0.0/24)       │
          └────────────┬────────────┘
                       │
                  Forgejo Container
                       │
          ┌────────────┴────────────┐
          │ forgejo-backend.network │
          │   (isolated)            │
          └────────────┬────────────┘
                       │
                PostgreSQL Container
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="practical-implementation-deploying-forgejo"&gt;Practical Implementation: Deploying&amp;nbsp;Forgejo&lt;/h2&gt;
&lt;p&gt;Let&amp;#8217;s walk through deploying a complete Git hosting service with database backend and &lt;span class="caps"&gt;TLS&lt;/span&gt;&amp;nbsp;termination.&lt;/p&gt;
&lt;h3 id="step-1-enable-podman-socket-for-traefik"&gt;Step 1: Enable Podman Socket for&amp;nbsp;Traefik&lt;/h3&gt;
&lt;p&gt;Traefik&amp;#8217;s Docker provider expects a Docker-compatible &lt;span class="caps"&gt;API&lt;/span&gt; socket. Podman provides this through a systemd-managed&amp;nbsp;socket:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Enable and start the Podman socket&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;podman.socket

&lt;span class="c1"&gt;# Verify the socket is active&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;podman.socket

&lt;span class="c1"&gt;# Check socket location&lt;/span&gt;
ls&lt;span class="w"&gt; &lt;/span&gt;-la&lt;span class="w"&gt; &lt;/span&gt;/run/podman/podman.sock
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Technical detail&lt;/strong&gt;: The Podman socket&amp;nbsp;(&lt;code&gt;/run/podman/podman.sock&lt;/code&gt;) provides a Docker-compatible &lt;span class="caps"&gt;REST&lt;/span&gt; &lt;span class="caps"&gt;API&lt;/span&gt;. Traefik connects to this socket to discover containers and read their labels for dynamic configuration. This is mounted into the Traefik container&amp;nbsp;as &lt;code&gt;/var/run/docker.sock&lt;/code&gt;, maintaining compatibility with Traefik&amp;#8217;s Docker provider&amp;nbsp;configuration.&lt;/p&gt;
&lt;p&gt;Without this socket enabled, Traefik cannot discover containers or process their routing labels - the declarative configuration simply won&amp;#8217;t&amp;nbsp;work.&lt;/p&gt;
&lt;h3 id="step-2-network-configuration"&gt;Step 2: Network&amp;nbsp;Configuration&lt;/h3&gt;
&lt;p&gt;First, create the networks. The frontend network enables IPv6 and connects to&amp;nbsp;Traefik:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Create the IPv6-enabled frontend network&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;network&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--ipv6&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--subnet&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.89.0.0/24&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;--gateway&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;10&lt;/span&gt;.89.0.1&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="se"&gt;\&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;ipv6

&lt;span class="c1"&gt;# Create the isolated backend network for database communication&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;network&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;forgejo-backend.network
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Design Decision&lt;/strong&gt;: Why two networks? The frontend network (despite its name &amp;#8220;ipv6&amp;#8221;) handles all external-facing traffic. The backend network ensures PostgreSQL is never directly accessible from the frontend, implementing&amp;nbsp;defense-in-depth.&lt;/p&gt;
&lt;h3 id="step-3-secrets-management"&gt;Step 3: Secrets&amp;nbsp;Management&lt;/h3&gt;
&lt;p&gt;Never hardcode passwords in configuration files. Podman secrets integrate with systemd and provide secure credential&amp;nbsp;storage:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Generate a strong database password&lt;/span&gt;
pwgen&lt;span class="w"&gt; &lt;/span&gt;-s&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;32&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;1&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="p"&gt;|&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;podman&lt;span class="w"&gt; &lt;/span&gt;secret&lt;span class="w"&gt; &lt;/span&gt;create&lt;span class="w"&gt; &lt;/span&gt;forgejo_db_password&lt;span class="w"&gt; &lt;/span&gt;-
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Design Decision&lt;/strong&gt;:&amp;nbsp;Using &lt;code&gt;podman secret&lt;/code&gt; over environment variables in files provides:
- Storage outside the unit file and outside any image layers you might build
- Access control via filesystem permissions on the secret store
- No secrets in process lists, environment dumps, or systemd&amp;#8217;s journal
- Easy rotation without rebuilding&amp;nbsp;containers&lt;/p&gt;
&lt;p&gt;Note that the&amp;nbsp;default &lt;code&gt;file&lt;/code&gt; driver does not encrypt secrets at rest: it stores them base64-encoded, protected by filesystem permissions. That is still a meaningful improvement over baking passwords into unit files or images, but if you need encryption at rest with a separate trust boundary, see the &lt;a href="https://hofstede.it/podman-in-production-quadlets-secrets-auto-updates-and-docker-compatibility/#secrets-management"&gt;Secrets Management section of the Podman deep-dive&lt;/a&gt; for when an external system like Vault or OpenBAO is the right&amp;nbsp;answer.&lt;/p&gt;
&lt;h3 id="step-4-database-container-quadlet"&gt;Step 4: Database Container&amp;nbsp;(Quadlet)&lt;/h3&gt;
&lt;p&gt;Create &lt;code&gt;/etc/containers/systemd/forgejo-db.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-db&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/postgres:16-alpine&lt;/span&gt;

&lt;span class="c1"&gt;# Network isolation - only on backend network&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-backend.network&lt;/span&gt;

&lt;span class="c1"&gt;# PostgreSQL configuration&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;POSTGRES_USER=forgejo&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;POSTGRES_DB=forgejo&lt;/span&gt;

&lt;span class="c1"&gt;# Secret injection as environment variable&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo_db_password,type=env,target=POSTGRES_PASSWORD&lt;/span&gt;

&lt;span class="c1"&gt;# Persistent storage with SELinux context&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/forgejo/postgres:/var/lib/postgresql/data:z&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;default.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Key technical points&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;AutoUpdate=registry&lt;/code&gt;: Enables automatic image updates&amp;nbsp;via &lt;code&gt;podman auto-update&lt;/code&gt;&lt;/li&gt;
&lt;li&gt;&lt;code&gt;Volume&lt;/code&gt; flag &lt;code&gt;:z&lt;/code&gt;: Automatically relabels SELinux contexts for container&amp;nbsp;access&lt;/li&gt;
&lt;li&gt;&lt;code&gt;Secret&lt;/code&gt; directive: Injects the secret as an environment variable at&amp;nbsp;runtime&lt;/li&gt;
&lt;li&gt;No frontend network: Database is completely isolated from external&amp;nbsp;access&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="step-5-application-container-quadlet"&gt;Step 5: Application Container&amp;nbsp;(Quadlet)&lt;/h3&gt;
&lt;p&gt;Create &lt;code&gt;/etc/containers/systemd/forgejo-server.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-server&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;codeberg.org/forgejo/forgejo:13&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;

&lt;span class="c1"&gt;# Dual network attachment&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-backend.network&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;ipv6&lt;/span&gt;

&lt;span class="c1"&gt;# Application configuration&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;USER_UID=1000&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;USER_GID=1000&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__database__DB_TYPE=postgres&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__database__HOST=forgejo-db:5432&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__database__NAME=forgejo&lt;/span&gt;
&lt;span class="na"&gt;Environment&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;FORGEJO__database__USER=forgejo&lt;/span&gt;

&lt;span class="c1"&gt;# Database password from secret&lt;/span&gt;
&lt;span class="na"&gt;Secret&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo_db_password,type=env,target=FORGEJO__database__PASSWD&lt;/span&gt;

&lt;span class="c1"&gt;# Persistent storage&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/forgejo/forgejo:/data:z&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/timezone:/etc/timezone:ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/localtime:/etc/localtime:ro&lt;/span&gt;

&lt;span class="c1"&gt;# Traefik labels for routing&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.enable=true&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.docker.network=ipv6&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.rule=Host(`git.example.com`)&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.entrypoints=https&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.service=forgejo-http&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.tls.certresolver=traefiktls&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.routers.forgejo.middlewares=secure-headers@file&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.http.services.forgejo-http.loadbalancer.server.port=3000&amp;quot;&lt;/span&gt;

&lt;span class="c1"&gt;# SSH Git access via Traefik TCP routing&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.tcp.routers.forgejo-ssh.rule=HostSNI(`*`)&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.tcp.routers.forgejo-ssh.entrypoints=ssh&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.tcp.routers.forgejo-ssh.service=forgejo-ssh&amp;quot;&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;&amp;quot;traefik.tcp.services.forgejo-ssh.loadbalancer.server.port=22&amp;quot;&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;default.target&lt;/span&gt;

&lt;span class="k"&gt;[Unit]&lt;/span&gt;
&lt;span class="na"&gt;After&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;forgejo-db.service&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Design decisions explained&lt;/strong&gt;:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Dual network attachment&lt;/strong&gt;: The container needs backend network for PostgreSQL and frontend for&amp;nbsp;Traefik&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Traefik labels&lt;/strong&gt;: Declarative routing configuration - no manual Traefik config files&amp;nbsp;needed&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;SSH&lt;/span&gt; routing&lt;/strong&gt;: Traefik handles both &lt;span class="caps"&gt;HTTP&lt;/span&gt; and &lt;span class="caps"&gt;TCP&lt;/span&gt; (Git &lt;span class="caps"&gt;SSH&lt;/span&gt;) on different&amp;nbsp;ports&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Systemd dependency&lt;/strong&gt;: &lt;code&gt;After=forgejo-db.service&lt;/code&gt; ensures database starts&amp;nbsp;first&lt;/li&gt;
&lt;/ol&gt;
&lt;h3 id="step-6-reverse-proxy-configuration"&gt;Step 6: Reverse Proxy&amp;nbsp;Configuration&lt;/h3&gt;
&lt;p&gt;Create &lt;code&gt;/etc/containers/systemd/traefik.container&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;ContainerName&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik&lt;/span&gt;
&lt;span class="na"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;docker.io/traefik:latest&lt;/span&gt;
&lt;span class="na"&gt;AutoUpdate&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;registry&lt;/span&gt;

&lt;span class="c1"&gt;# Required for binding to privileged ports&lt;/span&gt;
&lt;span class="na"&gt;AddCapability&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;CAP_NET_BIND_SERVICE&lt;/span&gt;

&lt;span class="c1"&gt;# Frontend network only&lt;/span&gt;
&lt;span class="na"&gt;Network&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;ipv6&lt;/span&gt;

&lt;span class="c1"&gt;# Port exposure&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;80:80&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;443:443&lt;/span&gt;
&lt;span class="na"&gt;PublishPort&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;2222:2222&lt;/span&gt;

&lt;span class="c1"&gt;# Security hardening&lt;/span&gt;
&lt;span class="na"&gt;NoNewPrivileges&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;true&lt;/span&gt;
&lt;span class="na"&gt;SecurityLabelType&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;container_runtime_t&lt;/span&gt;

&lt;span class="c1"&gt;# Configuration and state&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/localtime:/etc/localtime:ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/run/podman/podman.sock:/var/run/docker.sock:ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/traefik/traefik.yml:/etc/traefik/traefik.yml:z,ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/traefik/config.yml:/etc/traefik/config.yml:z,ro&lt;/span&gt;
&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/traefik/letsencrypt:/letsencrypt:z&lt;/span&gt;

&lt;span class="c1"&gt;# Self-configuration for dashboard&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.enable=true&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.rule=Host(`traefik.example.com`)&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.entrypoints=https&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.service=api@internal&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.tls=true&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.tls.certresolver=traefiktls&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.routers.dashboard.middlewares=dashboard-auth,secure-headers@file&lt;/span&gt;
&lt;span class="na"&gt;Label&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;traefik.http.middlewares.dashboard-auth.basicauth.users=admin:$$2y$$05$$...&lt;/span&gt;

&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;Restart&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;always&lt;/span&gt;

&lt;span class="k"&gt;[Install]&lt;/span&gt;
&lt;span class="na"&gt;WantedBy&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;default.target&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Create &lt;code&gt;/opt/traefik/traefik.yml&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;global&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;checkNewVersion&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;sendAnonymousUsage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;

&lt;span class="nt"&gt;api&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;dashboard&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;insecure&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;

&lt;span class="nt"&gt;entryPoints&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;http&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;address&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;:80&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;http&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;redirections&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;entryPoint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;to&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;https&lt;/span&gt;
&lt;span class="w"&gt;          &lt;/span&gt;&lt;span class="nt"&gt;scheme&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;https&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;https&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;address&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;:443&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;ssh&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;address&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;:2222&amp;quot;&lt;/span&gt;

&lt;span class="nt"&gt;providers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;docker&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;endpoint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;unix:///var/run/docker.sock&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;exposedByDefault&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;false&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;network&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;ipv6&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;file&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;filename&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/etc/traefik/config.yml&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;watch&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;

&lt;span class="nt"&gt;certificatesResolvers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;traefiktls&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;acme&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;email&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;admin@example.com&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;storage&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;/letsencrypt/acme.json&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;httpChallenge&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;entryPoint&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;http&lt;/span&gt;

&lt;span class="nt"&gt;log&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;level&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;INFO&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Create &lt;code&gt;/opt/traefik/config.yml&lt;/code&gt; for shared&amp;nbsp;middlewares:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="nt"&gt;http&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;  &lt;/span&gt;&lt;span class="nt"&gt;middlewares&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;    &lt;/span&gt;&lt;span class="nt"&gt;secure-headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;      &lt;/span&gt;&lt;span class="nt"&gt;headers&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;stsSeconds&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;31536000&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;stsIncludeSubdomains&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;stsPreload&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;forceSTSHeader&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;customFrameOptionsValue&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;SAMEORIGIN&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;contentTypeNosniff&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;browserXssFilter&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="l l-Scalar l-Scalar-Plain"&gt;true&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;referrerPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;strict-origin-when-cross-origin&amp;quot;&lt;/span&gt;
&lt;span class="w"&gt;        &lt;/span&gt;&lt;span class="nt"&gt;permissionsPolicy&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s"&gt;&amp;quot;geolocation=(),&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;microphone=(),&lt;/span&gt;&lt;span class="nv"&gt; &lt;/span&gt;&lt;span class="s"&gt;camera=()&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;Security highlights&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;NoNewPrivileges&lt;/code&gt;: Prevents privilege&amp;nbsp;escalation&lt;/li&gt;
&lt;li&gt;&lt;code&gt;SecurityLabelType&lt;/code&gt;: SELinux type&amp;nbsp;enforcement&lt;/li&gt;
&lt;li&gt;Automatic &lt;span class="caps"&gt;HTTP&lt;/span&gt; to &lt;span class="caps"&gt;HTTPS&lt;/span&gt;&amp;nbsp;redirect&lt;/li&gt;
&lt;li&gt;&lt;span class="caps"&gt;HSTS&lt;/span&gt; headers for &lt;span class="caps"&gt;HTTPS&lt;/span&gt;&amp;nbsp;enforcement&lt;/li&gt;
&lt;li&gt;Let&amp;#8217;s Encrypt automation for &lt;span class="caps"&gt;TLS&lt;/span&gt;&amp;nbsp;certificates&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="step-7-deployment"&gt;Step 7:&amp;nbsp;Deployment&lt;/h3&gt;
&lt;p&gt;Reload systemd to recognize the new&amp;nbsp;units:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;systemctl&lt;span class="w"&gt; &lt;/span&gt;daemon-reload
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Enable and start the&amp;nbsp;services:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Enable automatic startup&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;forgejo-db.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;traefik.service

&lt;span class="c1"&gt;# Start the stack&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;forgejo-db.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service
systemctl&lt;span class="w"&gt; &lt;/span&gt;start&lt;span class="w"&gt; &lt;/span&gt;traefik.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;The beauty of systemd integration&lt;/strong&gt;: You can now manage containers like any other&amp;nbsp;service:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check status&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;status&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service

&lt;span class="c1"&gt;# View logs&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service&lt;span class="w"&gt; &lt;/span&gt;-f

&lt;span class="c1"&gt;# Restart&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;restart&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service

&lt;span class="c1"&gt;# View dependency tree&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;list-dependencies&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="step-8-automated-updates"&gt;Step 8: Automated&amp;nbsp;Updates&lt;/h3&gt;
&lt;p&gt;Enable automatic container image&amp;nbsp;updates:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Enable the timer&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;podman-auto-update.timer

&lt;span class="c1"&gt;# Check update status&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;auto-update&lt;span class="w"&gt; &lt;/span&gt;--dry-run
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;With &lt;code&gt;AutoUpdate=registry&lt;/code&gt; in the Quadlet files, Podman&amp;nbsp;will:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Check for new images&amp;nbsp;daily&lt;/li&gt;
&lt;li&gt;Pull updates if&amp;nbsp;available&lt;/li&gt;
&lt;li&gt;Recreate containers with new&amp;nbsp;images&lt;/li&gt;
&lt;li&gt;Preserve all volumes and&amp;nbsp;configuration&lt;/li&gt;
&lt;/ol&gt;
&lt;h2 id="advanced-topics"&gt;Advanced&amp;nbsp;Topics&lt;/h2&gt;
&lt;h3 id="working-with-red-hat-registries"&gt;Working with Red Hat&amp;nbsp;Registries&lt;/h3&gt;
&lt;p&gt;While this example uses public container registries, production &lt;span class="caps"&gt;RHEL&lt;/span&gt; deployments often leverage Red Hat&amp;#8217;s container&amp;nbsp;catalog:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Authenticate to Red Hat registry (requires active subscription)&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;login&lt;span class="w"&gt; &lt;/span&gt;registry.redhat.io

&lt;span class="c1"&gt;# Pull RHEL-based images&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;pull&lt;span class="w"&gt; &lt;/span&gt;registry.redhat.io/rhel9/postgresql-15

&lt;span class="c1"&gt;# Use in Quadlet&lt;/span&gt;
&lt;span class="nv"&gt;Image&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;registry.redhat.io/rhel9/postgresql-15
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Red Hat certified container images include:
- Support lifecycle matching &lt;span class="caps"&gt;RHEL&lt;/span&gt; versions
- Security errata and &lt;span class="caps"&gt;CVE&lt;/span&gt; fixes
- Compliance with enterprise requirements
- Verified compatibility with &lt;span class="caps"&gt;RHEL&lt;/span&gt; container&amp;nbsp;hosts&lt;/p&gt;
&lt;h3 id="selinux-integration"&gt;SELinux&amp;nbsp;Integration&lt;/h3&gt;
&lt;p&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s mandatory access control is a feature, not a bug. While many Docker tutorials suggest disabling SELinux, Podman embraces it as a critical security layer.&amp;nbsp;The &lt;code&gt;:z&lt;/code&gt; flag in volume mounts automatically handles SELinux&amp;nbsp;labeling:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/opt/forgejo/postgres:/var/lib/postgresql/data:z&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;This relabels the host directory with the correct SELinux context&amp;nbsp;(&lt;code&gt;container_file_t&lt;/code&gt;) for container access. For read-only mounts,&amp;nbsp;use &lt;code&gt;:ro&lt;/code&gt; without &lt;code&gt;:z&lt;/code&gt;:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="na"&gt;Volume&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/etc/localtime:/etc/localtime:ro&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;&lt;strong&gt;&lt;span class="caps"&gt;RHEL&lt;/span&gt; best practice&lt;/strong&gt;: Never disable SELinux. If you encounter permission issues, investigate the context&amp;nbsp;(&lt;code&gt;ls -Z&lt;/code&gt;) rather than setting permissive mode. Podman&amp;#8217;s integration makes this dramatically easier than it was with&amp;nbsp;Docker.&lt;/p&gt;
&lt;p&gt;For advanced scenarios, you can use&amp;nbsp;uppercase &lt;code&gt;:Z&lt;/code&gt; to make the volume private to that specific container, or manually manage&amp;nbsp;contexts:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Check SELinux context&lt;/span&gt;
ls&lt;span class="w"&gt; &lt;/span&gt;-Z&lt;span class="w"&gt; &lt;/span&gt;/opt/forgejo/

&lt;span class="c1"&gt;# Manually set context if needed&lt;/span&gt;
semanage&lt;span class="w"&gt; &lt;/span&gt;fcontext&lt;span class="w"&gt; &lt;/span&gt;-a&lt;span class="w"&gt; &lt;/span&gt;-t&lt;span class="w"&gt; &lt;/span&gt;container_file_t&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;/opt/forgejo(/.*)?&amp;quot;&lt;/span&gt;
restorecon&lt;span class="w"&gt; &lt;/span&gt;-Rv&lt;span class="w"&gt; &lt;/span&gt;/opt/forgejo/
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="resource-limits"&gt;Resource&amp;nbsp;Limits&lt;/h3&gt;
&lt;p&gt;Systemd provides granular resource control through&amp;nbsp;cgroups:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Service]&lt;/span&gt;
&lt;span class="na"&gt;MemoryMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;2G&lt;/span&gt;
&lt;span class="na"&gt;CPUQuota&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;200%&lt;/span&gt;
&lt;span class="na"&gt;TasksMax&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;1024&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;These limits are enforced by the kernel and prevent resource&amp;nbsp;exhaustion.&lt;/p&gt;
&lt;h3 id="health-checks"&gt;Health&amp;nbsp;Checks&lt;/h3&gt;
&lt;p&gt;Podman supports container health&amp;nbsp;checks:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="k"&gt;[Container]&lt;/span&gt;
&lt;span class="na"&gt;HealthCmd&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;/usr/bin/curl -f http://localhost:3000/ || exit 1&lt;/span&gt;
&lt;span class="na"&gt;HealthInterval&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;30s&lt;/span&gt;
&lt;span class="na"&gt;HealthTimeout&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;5s&lt;/span&gt;
&lt;span class="na"&gt;HealthRetries&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="s"&gt;3&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;p&gt;Systemd can react to failed health checks and restart containers&amp;nbsp;automatically.&lt;/p&gt;
&lt;h2 id="monitoring-and-observability"&gt;Monitoring and&amp;nbsp;Observability&lt;/h2&gt;
&lt;h3 id="viewing-logs"&gt;Viewing&amp;nbsp;Logs&lt;/h3&gt;
&lt;p&gt;All container output goes to&amp;nbsp;journald:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Real-time logs&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service&lt;span class="w"&gt; &lt;/span&gt;-f

&lt;span class="c1"&gt;# Last 100 lines&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service&lt;span class="w"&gt; &lt;/span&gt;-n&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="m"&gt;100&lt;/span&gt;

&lt;span class="c1"&gt;# Logs from specific time&lt;/span&gt;
journalctl&lt;span class="w"&gt; &lt;/span&gt;-u&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service&lt;span class="w"&gt; &lt;/span&gt;--since&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="s2"&gt;&amp;quot;2024-01-01 12:00:00&amp;quot;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h3 id="container-inspection"&gt;Container&amp;nbsp;Inspection&lt;/h3&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# Container details&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;forgejo-server

&lt;span class="c1"&gt;# Resource usage&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;stats&lt;span class="w"&gt; &lt;/span&gt;forgejo-server

&lt;span class="c1"&gt;# Network information&lt;/span&gt;
podman&lt;span class="w"&gt; &lt;/span&gt;network&lt;span class="w"&gt; &lt;/span&gt;inspect&lt;span class="w"&gt; &lt;/span&gt;ipv6
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="comparison-with-kubernetes"&gt;Comparison with&amp;nbsp;Kubernetes&lt;/h2&gt;
&lt;p&gt;This approach is not meant to replace Kubernetes for large-scale deployments, but it offers distinct advantages for single-host or small-scale&amp;nbsp;scenarios:&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Advantages&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Significantly lower resource&amp;nbsp;overhead&lt;/li&gt;
&lt;li&gt;Simpler mental model and&amp;nbsp;troubleshooting&lt;/li&gt;
&lt;li&gt;Direct integration with &lt;span class="caps"&gt;OS&lt;/span&gt;-level&amp;nbsp;tools&lt;/li&gt;
&lt;li&gt;No additional control plane&amp;nbsp;components&lt;/li&gt;
&lt;li&gt;Easier to audit and&amp;nbsp;secure&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;&lt;strong&gt;When to choose K8s instead&lt;/strong&gt;:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Multi-host orchestration&amp;nbsp;requirements&lt;/li&gt;
&lt;li&gt;Advanced scheduling&amp;nbsp;needs&lt;/li&gt;
&lt;li&gt;Built-in service mesh&amp;nbsp;requirements&lt;/li&gt;
&lt;li&gt;Teams already invested in K8s&amp;nbsp;ecosystem&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id="security-considerations"&gt;Security&amp;nbsp;Considerations&lt;/h2&gt;
&lt;p&gt;This setup implements several security&amp;nbsp;layers:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Network segmentation&lt;/strong&gt;: Databases isolated from&amp;nbsp;frontend&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Rootless option&lt;/strong&gt;: All containers can run as unprivileged&amp;nbsp;users&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;SELinux enforcement&lt;/strong&gt;: Mandatory access&amp;nbsp;control&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Secret management&lt;/strong&gt;: No credentials in configuration&amp;nbsp;files&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automatic updates&lt;/strong&gt;: Regular security&amp;nbsp;patches&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;&lt;span class="caps"&gt;TLS&lt;/span&gt; termination&lt;/strong&gt;: Encrypted transport with Let&amp;#8217;s&amp;nbsp;Encrypt&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Security headers&lt;/strong&gt;: &lt;span class="caps"&gt;HSTS&lt;/span&gt;, &lt;span class="caps"&gt;CSP&lt;/span&gt;, and other&amp;nbsp;protections&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;For even stricter security, run containers in rootless&amp;nbsp;mode:&lt;/p&gt;
&lt;div class="highlight"&gt;&lt;pre&gt;&lt;span&gt;&lt;/span&gt;&lt;code&gt;&lt;span class="c1"&gt;# As regular user (not root)&lt;/span&gt;
systemctl&lt;span class="w"&gt; &lt;/span&gt;--user&lt;span class="w"&gt; &lt;/span&gt;&lt;span class="nb"&gt;enable&lt;/span&gt;&lt;span class="w"&gt; &lt;/span&gt;--now&lt;span class="w"&gt; &lt;/span&gt;forgejo-server.service
&lt;/code&gt;&lt;/pre&gt;&lt;/div&gt;

&lt;h2 id="conclusion"&gt;Conclusion&lt;/h2&gt;
&lt;p&gt;Modern container deployment doesn&amp;#8217;t require Kubernetes for every use case. With &lt;span class="caps"&gt;RHEL&lt;/span&gt; Quadlets, Podman, and proper architectural patterns, you can build production-grade container infrastructure that&amp;nbsp;is:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Secure&lt;/strong&gt;: Multiple layers of isolation and access control, leveraging &lt;span class="caps"&gt;RHEL&lt;/span&gt;&amp;#8217;s security-first&amp;nbsp;design&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Maintainable&lt;/strong&gt;: Declarative configuration with systemd&amp;nbsp;integration&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Observable&lt;/strong&gt;: Native integration with journald and systemd&amp;nbsp;tools&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Automated&lt;/strong&gt;: Built-in update mechanisms via systemd&amp;nbsp;timers&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Resilient&lt;/strong&gt;: Systemd&amp;#8217;s proven service&amp;nbsp;management&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Enterprise-ready&lt;/strong&gt;: Backed by Red Hat&amp;#8217;s support lifecycle and security&amp;nbsp;practices&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;This approach proves particularly valuable&amp;nbsp;for:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Self-hosted services and edge&amp;nbsp;deployments&lt;/li&gt;
&lt;li&gt;Development environments matching&amp;nbsp;production&lt;/li&gt;
&lt;li&gt;Organizations preferring simpler operational&amp;nbsp;models&lt;/li&gt;
&lt;li&gt;Hybrid scenarios where some services don&amp;#8217;t warrant K8s&amp;nbsp;overhead&lt;/li&gt;
&lt;/ul&gt;
&lt;h3 id="next-steps-for-red-hat-practitioners"&gt;Next Steps for Red Hat&amp;nbsp;Practitioners&lt;/h3&gt;
&lt;p&gt;This foundation scales naturally into the broader Red Hat container&amp;nbsp;ecosystem:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;&lt;strong&gt;Fedora CoreOS&lt;/strong&gt;: Apply these Quadlet patterns to immutable, auto-updating&amp;nbsp;infrastructure&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;OpenShift&lt;/strong&gt;: Recognize how systemd-managed containers relate to K8s&amp;nbsp;pods&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Ansible automation&lt;/strong&gt;: Codify Quadlet deployment&amp;nbsp;with &lt;code&gt;containers.podman&lt;/code&gt; collection&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Image building&lt;/strong&gt;: Explore Buildah and Skopeo for &lt;span class="caps"&gt;OCI&lt;/span&gt; image&amp;nbsp;workflows&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;The combination of Podman&amp;#8217;s security-first design and systemd&amp;#8217;s battle-tested service management creates a robust foundation for containerized applications without the operational overhead of full orchestration platforms - and provides essential knowledge for working across Red Hat&amp;#8217;s entire container&amp;nbsp;portfolio.&lt;/p&gt;
&lt;h2 id="further-reading"&gt;Further&amp;nbsp;Reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;a href="https://docs.podman.io/en/latest/markdown/podman-systemd.unit.5.html"&gt;Podman Quadlet&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/10/html/building_running_and_managing_containers/"&gt;Red Hat Enterprise Linux Container&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://doc.traefik.io/traefik/providers/docker/"&gt;Traefik Podman&amp;nbsp;Provider&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://www.freedesktop.org/software/systemd/man/systemd.resource-control.html"&gt;systemd.resource-control&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://connect.redhat.com/en/partner-with-us/red-hat-container-certification"&gt;Red Hat Container&amp;nbsp;Certification&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;&lt;a href="https://docs.fedoraproject.org/en-US/fedora-coreos/"&gt;Fedora CoreOS&amp;nbsp;Documentation&lt;/a&gt;&lt;/li&gt;
&lt;/ul&gt;
&lt;hr&gt;</content><category term="Linux"/><category term="linux"/><category term="rhel"/><category term="podman"/><category term="containers"/><category term="forgejo"/></entry></feed>