Journal · GAM

After the Power Outage: An Image Cache for OpenShift with Zot

Details: Written by: Guille; Category: Journal; Published: 12 November 2026; Hits: 19

While recovering several OKD clusters after a power outage, I found images that CRI-O listed as present but that the node could not use. On one node, pulling an image again by digest brought OVN and Multus back. That left me with a question for the next recovery: what happens when several nodes need to pull the same images at once?

Building RealWorld in Protos: HTTP, Actors and authentication in an experimental language

Details: Written by: Guille; Category: Journal; Published: 10 November 2026; Hits: 17

While testing HTTP requests with bodies in the RealWorld server, I ran into a problem in an unexpected place. The Worker could read the body from its TCP connection, but passing those Bytes to the Actor that ran the application exposed a bug in the Protos runtime. Fixing it became part of building the server. This was exactly the kind of problem I wanted to find by taking the language beyond small examples.

Protos is an experimental prototype-based language I am developing on GraalVM and Truffle. I chose RealWorld, also known as Conduit, to put it to the test. RealWorld is an article-publishing application with a common API that allows different frontends and backends to work together. My implementation covers all 19 API routes, including registration, login, profiles, articles, the feed, comments, follows and favourites. RealWorld project · Protos backend source

Moving rootless Podman storage to another disk without losing containers

Details: Written by: Guille; Category: Journal; Published: 09 November 2026; Hits: 17

When you build container images frequently, container storage can fill the system disk without making the cause obvious. In my Protos development environment, built around GraalVM and Truffle, I use rootless Podman for several VS Code devcontainers and for building images with different GraalVM versions. Eventually / reached 97% usage, while a second disk mounted at /datos still had 386 GB free.

The goal was to move Podman's data to that second disk without losing images, containers, or volumes. The copy itself was straightforward. The interesting part came when Podman rejected the new graphroot because its database still remembered the old path. Here is the actual procedure, including the error and measured results.

When Jenkins Almost Brought Down a Kubernetes Worker

Details: Written by: Guille; Category: Journal; Published: 30 October 2026; Hits: 47

A real incident on OKD: blocked processes, orphaned agents, cgroups, pipelines that must not be resumed, and an unexpected problem when removing the recovery safeguards.

Migrating Portainer Agent to Edge Agent on OpenShift: Troubleshooting and Resolving Multiple Deployment Issues

Details: Written by: Guille; Category: Journal; Published: 28 October 2026; Hits: 58

Migrating Portainer Agent to Edge Agent Standard across several OpenShift/OKD clusters exposed a chain of independent issues: filesystem permissions, immutable Kubernetes Service fields, DNS resolution, incomplete credentials, and deployment automation. This technical report documents the observed symptoms, confirmed causes, and corrective actions.

Note: this is an anonymized real-world technical case. Organization, cluster, node, domain, IP address, environment ID, and credential details have been removed. All examples use generic names and must be adapted before running commands.

Recovering CRI-O Image Storage After a Power Outage

Details: Written by: Guille; Category: Journal; Published: 27 October 2026; Hits: 49

A power outage left several OpenShift/OKD clusters in a puzzling state: nodes appeared available, pods were stuck, and critical components could not create containers. The root cause was not the remote image registry. It was inconsistent local image references and layers in CRI-O's storage. This guide records the commands used, the observed results, and a controlled recovery procedure.

Note: this is an anonymized real incident. Hostnames, networks, internal domains, image identifiers, and deployment-specific details have been removed. Replace all <...> placeholders before running commands. The text distinguishes observed actions from general operational recommendations.

For the broader story, read OpenShift After a Power Outage: The Images Were There, but They Weren’t.

OpenShift After a Power Outage: The Images Were There, but They Weren’t

Details: Written by: Guille; Category: Journal; Published: 26 October 2026; Hits: 57

After a power outage, several OpenShift clusters began coming back online. The nodes were up, but critical parts of the platform would not start. The strangest clue was that some images appeared in the local inventory while the runtime insisted they did not exist.

Some incidents begin with one symptom and end up teaching you that the real problem lives two layers below it. This was one of those incidents.

When power returned, several clusters had to recover at almost the same time. Some nodes became available, but essential pods were unable to start. At first it looked like a major network failure. The most useful evidence, however, was in the nodes' local image storage.

Looking for the commands? The detailed diagnosis, CRI-O repair procedure, and verification steps are in the technical recovery guide.

I Set Out to Fix a Security Issue and Ended Up Rebuilding My Website

Details: Written by: Guille; Category: Journal; Published: 25 October 2026; Hits: 63

From a critical PHP alert to several days of updates, security reviews, migrations and a redesign I'd been putting off for years.

It all started with a PHP security alert.

On 21 October 2026, fixes were released for two critical vulnerabilities that could allow remote code execution. The sort of news that, when you work with systems, makes you stop what you're doing and ask which of your services might be affected.

At first, the job seemed fairly straightforward: check the installed versions, work out what needed updating, apply the fixes and make sure everything still worked.

Just a bit of maintenance, really.