Our builds failed with a full disk. Retrying was the wrong fix

On 25 September a KantaHub deploy failed with "no space left on device". We retried, and it passed. That is the trap: the retry worked, so it looked like a fluke.

What was filling the disk

Our builds on Railway keep caches between runs so they stay fast: the Next.js build cache, node_modules caches and the package manager's store. Nothing prunes those caches automatically, so every build leaves a little more behind until the build machine runs out of room.

The failure moves earlier

The first sign was a failure at the very last step, when the finished image is exported. A retry can pass here, because there is just enough room left for one more build.

The second sign is the one to watch. The same service later failed much earlier, while installing dependencies, before any of our own code had compiled.

A failure that moves earlier is not a second fluke. It means the disk pressure has grown, and a retry will not hold.

What we do now

• Never retry in a loop on a disk-space error. First check whether it failed earlier than last time.
• If it did, set the NO_CACHE=1 variable on the service to force one clean build that throws the old caches away.
• Expect it to be slow. Ours took about 5 minutes against about 90 seconds with the cache. The slowness is how you know it really rebuilt from scratch.
• Remove NO_CACHE=1 straight afterwards so later builds use the cache again. Removing the variable does not need a deploy of its own.

What we are still checking

One of our monorepo services was keeping a Next.js cache for all three of its apps while building only one of them. That looks like wasted space. We have flagged it, but we have not yet confirmed it or trimmed it.

If you run builds with a cache

A build that fails for lack of space and then passes on retry has not been fixed. Read where it failed, compare that with last time, and clear the cache before the problem moves to a step you cannot ignore.