Earlier this month, I woke up and tried to start working on my previous post, but quickly hit a snag. I was unable to start the toolbox container I use while working on my website. In fact, none of my podman containers would start.
Was Podman Broken?
Specifically, when I tried to start a container I encountered this error message:
podman start website Error: unable to start container "f8ab31d42b9d04d051b23c65604e19748a9496f17bd3baab8e6f947eee8f3692": creating cgroup directory `/email@example.com/user.slice/libpod-f8ab31d42b9d04d051b23c65604e19748a9496f17bd3baab8e6f947eee8f3692.scope/container`: No such file or directory: OCI runtime command not found error
So, I attempted to use podman on a different computer. It worked fine. I compared version numbers and noticed that the second computer had a newer version of podman installed. I figured that I had hit a bug that must now be fixed, so I waited for the update to reach my desktop (it wasn’t available on that machine yet for some reason).
A day later when I ran my updates, the new version of
podman was installed, which
I thought would surely fix my problem. It didn’t. (ಠ_ಠ)
I started to scour the internet again to look for answers. Eventually, I found this reddit post. While reading it, the poster’s experience sounded very similar to my own. After reading some of the comments that connected the private internet access client to the original poster’s issues, I suddenly remembered… I had installed the PIA client on my machine earlier that week!
Sure enough, when I checked the ownership of my
net_cls files (as suggested
in the thread), it looked like
piavpn was claiming group ownership of the
➜ ~ ll /sys/fs/cgroup/net_cls total 0 -rw-r--r--. 1 root piavpn 0 Aug 4 21:21 cgroup.clone_children -rw-r--r--. 1 root piavpn 0 Aug 4 21:21 cgroup.procs -r--r--r--. 1 root piavpn 0 Aug 4 21:21 cgroup.sane_behavior drwxr-xr-x. 6 root root 0 Aug 4 21:21 machine.slice -rw-r--r--. 1 root piavpn 0 Aug 4 21:21 net_cls.classid -rw-r--r--. 1 root piavpn 0 Aug 4 21:21 notify_on_release -rw-r--r--. 1 root piavpn 0 Aug 4 21:21 release_agent -rw-r--r--. 1 root piavpn 0 Aug 4 21:21 tasks
Some commenters in the thread stated that the conflict went away after they removed the PIA client.
Removing the PIA Client
As a result, I decided to un-install my PIA client. It wasn’t a major loss for me, as I hadn’t used it in months. I only installed it to double check if it was a service I wanted, or if I should cancel my subscription before it auto-renewed later that month.
At first, I couldn’t find an un-install option, but eventually found it deep
in the settings. After removing the client, the
piavpn group went away… sort
of. It still had a
1004 gid, which I’m guessing was the previous
➜ ~ ll /sys/fs/cgroup/net_cls total 0 -rw-r--r--. 1 root 1004 0 Aug 4 21:21 cgroup.clone_children -rw-r--r--. 1 root 1004 0 Aug 4 21:21 cgroup.procs -r--r--r--. 1 root 1004 0 Aug 4 21:21 cgroup.sane_behavior drwxr-xr-x. 6 root root 0 Aug 4 21:21 machine.slice -rw-r--r--. 1 root 1004 0 Aug 4 21:21 net_cls.classid -rw-r--r--. 1 root 1004 0 Aug 4 21:21 notify_on_release -rw-r--r--. 1 root 1004 0 Aug 4 21:21 release_agent -rw-r--r--. 1 root 1004 0 Aug 4 21:21 tasks
Whatever the case… podman still didn’t work.
… Don’t forget to Reboot!
I was furious. After calming down, I reasoned it probably still wasn’t working
due to cruft from the client lingering on my system (like the
1004 group for
example), so I rebooted my desktop… and it worked!
In conclusion… why did I write this post? This complication was a huge pain to troubleshoot. It was only by chance that I stumbled on that reddit post, and would have had an even harder time without it. I assume having at least one more page on the internet stating that podman and the PIA client don’t play nice, might help others find the solution quicker. Hence, this post.
Before I end, it is worth nothing that some users reported that configuring the PIA openvpn profiles and using them to connect to the VPN works without issue. It is just the client that breaks containers. So if you want to still use PIA (I let mine expire. I don’t use it enough), give it a try!
Running VMs with VirtIO 3D Acceleration Running my Website Tests in Parallel with Pytest-Parallel