While recovering several OKD clusters after a power outage, I found images that CRI-O listed as present but that the node could not use. On one node, pulling an image again by digest brought OVN and Multus back. That left me with a question for the next recovery: what happens when several nodes need to pull the same images at once?
Read more: After the Power Outage: An Image Cache for OpenShift with Zot
While testing HTTP requests with bodies in the RealWorld server, I ran into a problem in an unexpected place. The Worker could read the body from its TCP connection, but passing those Bytes to the Actor that ran the application exposed a bug in the Protos runtime. Fixing it became part of building the server. This was exactly the kind of problem I wanted to find by taking the language beyond small examples.
Protos is an experimental prototype-based language I am developing on GraalVM and Truffle. I chose RealWorld, also known as Conduit, to put it to the test. RealWorld is an article-publishing application with a common API that allows different frontends and backends to work together. My implementation covers all 19 API routes, including registration, login, profiles, articles, the feed, comments, follows and favourites. RealWorld project · Protos backend source
Read more: Building RealWorld in Protos: HTTP, Actors and authentication in an experimental language
When you build container images frequently, container storage can fill the system disk without making the cause obvious. In my Protos development environment, built around GraalVM and Truffle, I use rootless Podman for several VS Code devcontainers and for building images with different GraalVM versions. Eventually / reached 97% usage, while a second disk mounted at /datos still had 386 GB free.
The goal was to move Podman's data to that second disk without losing images, containers, or volumes. The copy itself was straightforward. The interesting part came when Podman rejected the new graphroot because its database still remembered the old path. Here is the actual procedure, including the error and measured results.
Read more: Moving rootless Podman storage to another disk without losing containers
A real incident on OKD: blocked processes, orphaned agents, cgroups, pipelines that must not be resumed, and an unexpected problem when removing the recovery safeguards.
Read more: When Jenkins Almost Brought Down a Kubernetes Worker
Migrating Portainer Agent to Edge Agent Standard across several OpenShift/OKD clusters exposed a chain of independent issues: filesystem permissions, immutable Kubernetes Service fields, DNS resolution, incomplete credentials, and deployment automation. This technical report documents the observed symptoms, confirmed causes, and corrective actions.
Note: this is an anonymized real-world technical case. Organization, cluster, node, domain, IP address, environment ID, and credential details have been removed. All examples use generic names and must be adapted before running commands.