Podman #7 – Enough, Just Give Me the Files

By now, we’ve gone through rootful/rootless containers, networking, pods, and even WordPress. Sometimes you just want the files, put them in the right place, and start things. Here’s a “all-in-one” reference for your Quadlets.

0. Enable User Linger

Before rootless services can run when you’re not logged in, enable lingering for your user:

sudo loginctl enable-linger $UID

0b. Create Required Directories

Create the folders for Quadlets and persistent container storage:

mkdir -p ~/.config/containers/systemd
mkdir -p ~/.local/share/containers/storage/volumes/wp-mariadb
mkdir -p ~/.local/share/containers/storage/volumes/wp-html

0c. Create Secrets

Before starting the containers, create the secrets required for MariaDB and WordPress:

# MariaDB root password
echo -n "example-root-pw" | podman secret create blog_db_rootpassword -

# Database name
echo -n "wordpress" | podman secret create blog_db_name -

# Database user
echo -n "wpuser" | podman secret create blog_db_user -

# User password
echo -n "example-wp-pw" | podman secret create blog_db_password -

Verify the secrets:

podman secret ls

1. Network Quadlet

nano ~/.config/containers/systemd/intisostrictnet.network

[Unit]
Description=Isolated internal network

[Network]
Driver=bridge
Internal=true
Options=isolate=strict

[Install]
WantedBy=default.target

2. Pod Quadlet

nano ~/.config/containers/systemd/podracer.pod

[Unit]
Description=It goes brrrrr

[Pod]
PublishPort=8080:80
Network=intisostrictnet.network

[Install]
WantedBy=default.target

3. MariaDB Container

nano ~/.config/containers/systemd/mariadb.container

[Unit]
Description=MariaDB container for WordPress
PartOf=podracer.pod

[Container]
Image=docker.io/library/mariadb:11
Pod=podracer.pod
AutoUpdate=registry
StartWithPod=true
NoNewPrivileges=true
ContainerName=mariadb
Volume=%h/.local/share/containers/storage/volumes/wp-mariadb:/var/lib/mysql:z
Secret=blog_db_name,type=env,target=MYSQL_DATABASE
Secret=blog_db_user,type=env,target=MYSQL_USER
Secret=blog_db_password,type=env,target=MYSQL_PASSWORD
Secret=blog_db_rootpassword,type=env,target=MARIADB_ROOT_PASSWORD

[Service]
Restart=always

4. WordPress Container

nano ~/.config/containers/systemd/wordpress.container

[Unit]
Description=WordPress container
PartOf=podracer.pod
After=mariadb.service

[Container]
Image=docker.io/library/wordpress:latest
Pod=podracer.pod
AutoUpdate=registry
StartWithPod=true
NoNewPrivileges=true
ContainerName=wordpress
Volume=%h/.local/share/containers/storage/volumes/wp-html:/var/www/html:z
Secret=blog_db_name,type=env,target=WORDPRESS_DB_NAME
Secret=blog_db_user,type=env,target=WORDPRESS_DB_USER
Secret=blog_db_password,type=env,target=WORDPRESS_DB_PASSWORD
Environment=WORDPRESS_DB_HOST=127.0.0.1

[Service]
Restart=always

5. Start Everything

Reload systemd and start the services:

systemctl --user daemon-reload

Check if the service was created:

ls /run/user/$UID/systemd/generator/

systemctl --user start intisostrictnet-network.service
systemctl --user start podracer-pod.service

The containers will start automatically with the pod as specified in the Quadlet.

systemctl --user start mariadb.service
systemctl --user start wordpress.service

6. Troubleshoot

If something doesn’t start, check the generator:

/usr/lib/systemd/user-generators/podman-user-generator -dryrun

Follow logs with:

journalctl -fe

Or check container logs directly:

podman logs <container_name>

7. Access WordPress

If everything started correctly, open your browser and go to http://localhost:8080 to configure WordPress.

Podman #4 - Networking - Layer 2 Dragons

Finally, networking. I’ve been doing it for over 20 years, so let’s set some boundaries and enforce a few paradigms.

podman network create test

Yes, this creates a network just like in Docker. But I don’t want my containers talking to each other, and I definitely don’t want them talking to the internet.

The documentation I used to figure this out is here (again), but there are two key concepts that stood out: internal and isolate. And they can be combined!

Internal

Internal=true

This restricts external access for the network. In other words, containers won’t have a default gateway and can’t connect to the internet. This even improves startup time for some containers, like OpenSearch.

Isolate

Options=isolate=true

This option blocks traffic between networks that have it enabled. Networks with isolate=true can’t talk to each other, but they can talk to networks without isolation.

Strict Isolation

It turns out:

--opt=isolate=strict

…is also a thing. I stumbled across it on Reddit or GitHub (not well documented). With strict, the network can’t talk to anything unless that other network also has isolation enabled. Unlike true, it won’t communicate with “normal” networks.

Perfect. This means if we combine internal and isolate, we can build a network where nothing leaks out. That will be the default setup from here on.

Examples

podman network create --internal internalNet
podman network create --opt=isolate=true isoTrueNet
podman network create --opt=isolate=strict isoStrictNet
podman network create --internal --opt=isolate=true intIsoTrueNet
podman network create --internal --opt=isolate=strict intIsoStrictNet

Now let’s run some containers and see if they can ping each other:

podman run --rm -it --network=isoStrictNet alpine sh
podman run --rm -it --network=intIsoTrueNet alpine sh

And yes — it behaves exactly as described.

Rootless Quadlet

Since we’re not using the CLI to deploy services, we need to define this network in a rootless Quadlet file:

nano ~/.config/containers/systemd/intisostrictnet.network

[Unit]
Description=Isolated internal network

[Network]
Driver=bridge
Internal=true
Options=isolate=strict

[Install]
WantedBy=default.target

Then generate the systemd file:

systemctl --user daemon-reload

You can check if it was created with:

ls /run/user/$UID/systemd/generator/

If it isn’t there, troubleshoot with:

/usr/lib/systemd/user-generators/podman-user-generator -dryrun

If all looks good, start the service:

systemctl --user start intisostrictnet-network.service

Now podman network ls will show your shiny new network.

We can put a container on it to verify that we have no network connectivity:

 podman run --rm -it --network=systemd-intisostrictnet alpine sh

That’s it for Podman #4 – Networking.

https://docs.podman.io/en/latest/markdown/podman-network-create.1.html

What 'Network Stability' Implies

My thoughts on what to do and not do if you have a goal of operating a stable network.

Human Infrastructure Magazine 71 – What Network Stability Implies by Ethan Banks

Several weeks ago, I tweeted...

As a network engineer, perhaps your most important responsibility is network stability. Much is implied by the word "stability" @ecbanks

What was I getting at when I said that, “Much is implied by the word stability”?

No science experiments

I’ve worked on several networks where it seemed like every feature some previous engineer read about in a certification book was enabled. A production network is not your lab. Don’t turn on (or off) a feature unless you have a specific business reason to do so.

Fix bad designs

Most networks have problems. Those problems will, if ignored, lead to unscheduled downtime. One of your jobs as an engineer is to anticipate the weaknesses in a network and, making the most of your allotted budget and available equipment, engineer those weaknesses out.

Standard, simple, replicable design

Networks that are built the same from pod to pod, closet to closet, and site to site tend to be more stable. Standardized designs have been standardized because they work. They tend to be as simple as possible, but no simpler. Standardized, simple designs lend themselves to being copied. The documenting of standard designs means that other engineers have the opportunity to enforce consistency across the network landscape.

No cowboys

As an engineer working on a production network, you are not a special snowflake. Fall in line. Build the network according to the standard. Stop making it up as you go like a gunslinging anti-hero. Networks don’t stay up if you make it up. Stable networks are the result of thoughtful design pondered over time and carefully considered in the context of the business and the rest of the IT stack.

A minimum of changes

Stop changing things whenever you’re in the mood. Every time you’re about to commit a change, ask yourself if it’s necessary. If yes, is this the appropriate time? Change windows exist for a reason. Do the right thing at the right time to mitigate the risk inherent in change.

Capacity monitoring and planning

A stable network is one that carries all of its traffic with consistent end-to-end latency and very little packet loss. Congested links are a form of network instability, because they result in undelivered traffic. Keeping ahead of bottlenecks is key to long-term network stability.

Wizard-like knowledge of traffic impact

While companies like Veriflow and Forward Networks get up to speed verifying changes against network models, the best network modeler is often the engineer making the change. You should understand the protocols you’re shooting from your keyboard laser pistol so thoroughly that you can predict what’s going to happen when you start making “pew pew” noises. If every new command results in a surprise, you’re going to cause downtime without meaning to.

Obvious security holes filled

Read a hardening guide and apply the bits you can. Change default credentials. Turn off unused services. Avoid silly things like “public” and “private” SNMP community strings, remnants from a bygone era. If you leave doors open, you’re going to get owned. Once the doors are shut, make sure the windows are latched. Stable networks make an attacker work to compromise the infrastructure.

Bug free code

Don’t run the latest release just because it’s out. New code can, and often will, introduce bugs. If the code you run now does what you need and has no known serious security vulnerabilities, stick with it. When possible, prefer maintenance trains and vendor recommended releases over code still dripping with developer sweat.

If I Had To Pick One Thing...

I’d pick “a minimum of changes” as the single greatest factor in network stability. Leave the network alone if it isn’t broken. When the network design does need to change, plan the change with excruciating caution, and schedule the change for a time that the business agrees to.

Never make changes on the sly, hoping no one notices. That can be hard when you’re an introverted type like I am, especially when you just want to get something done.

However, I’ve learned that no matter how capable you are or how unlikely the change is to cause a disruption, accidents happen. A mid-day outage is an introvert’s worst nightmare, especially if you’re the cause.