Solaris 8 plant recovery

What this is

An Ammann AS2000 asphalt plant runs on a control system hosted on Solaris 8, which is to say SunOS 5.8, a platform that shipped in 2000. The plant is current. The computer telling it what to do is a quarter of a century old, and that is entirely normal in industrial settings, where the machinery outlives several generations of the equipment controlling it.

The control system had stopped coming up cleanly. My job was to get it back and leave it stable.

Why this is different from ordinary troubleshooting

Three things make legacy industrial recovery its own discipline.

There is no vendor path and no search result. The platform is long out of support. The application is a vertical product with a small installed base. Nobody has posted your error message, because roughly eleven people in the world have seen it.

You cannot experiment freely. On a normal server you try something and reboot. Here the machine is attached to a production asphalt plant, and every restart is a production decision that somebody has to authorise. You get few attempts, so each one has to be reasoned rather than guessed.

There is no rollback. No snapshots, no image, no rebuild path. The install on that disk is the only copy of a configuration assembled over years by people who have moved on.

The startup chain

The core failure was a corrupted startup chain. On System V init the boot is a sequence of scripts run in order, and the application came up at the end of it through S99uag in rc2.d. The chain from init through the run level directory to that script had been broken, so the machine reached a running state with no application on it.

Repairing this means reading the chain rather than restarting it. Which run level the system is entering, which scripts that level executes, in what order, and which link in that sequence is absent or pointing at nothing. It is unglamorous work and it is the whole job: once you can describe the intended sequence accurately, the break is usually obvious.

Alongside it, the environment the application expected had drifted. UAG, PATH and APP_ROOT all had to be re-established. Environment problems on legacy systems are particularly deceptive because the failure surfaces far from the cause: the application starts, cannot find something it assumed, and reports a symptom that has nothing to do with a missing variable.

The respawn loop

A failure mode worth naming, because it looks like the system is doing something.

init is configured to keep certain processes alive. When such a process fails immediately at startup, init restarts it, it fails again, and the cycle continues. The machine is busy, the console produces output, and nothing is progressing. It presents as a hang while actually being an infinite loop of correct behaviour responding to a broken component.

Breaking that loop means finding what the respawned process needs and cannot get, which in this case ran through the next set of problems.

Shared memory, permissions, and the watchdog

Three related faults sat underneath the startup failures.

Shared memory. The application uses System V IPC to communicate between its components. When shmget fails, either because a segment is stale from a previous unclean shutdown or because the kernel parameters no longer permit what is being requested, the components cannot talk and each one reports its own unrelated-looking error.

Executable permissions. Mundane, and genuinely was part of it. Files that had to be executable were not, so the startup chain reached them and stopped.

The watchdog. Industrial control systems run a watchdog that restarts the application when it stops responding. When the application is failing for another reason, the watchdog turns a diagnosable fault into a moving target: every time you get close to reading the state, the thing you are inspecting is restarted underneath you.

Autologin, deliberately

The console needed to reach the operating application automatically after a power event, without waiting for someone to type a password into an industrial console that has no keyboard in normal use.

Written down plainly that sounds like a security failure, and in an office context it would be. In this one it is a considered decision: the machine is physically inside plant controls, its role is single purpose, and the alternative is a plant that will not resume after a power cut until a specific person drives to the site. The security control here is physical access, not a login prompt, and pretending otherwise would have produced a worse outcome with better documentation.

The rest

The communication layer to the Phoenix controllers was unstable and needed stabilising, which is where the control system meets the plant and therefore where instability is least acceptable. The crash partitions had filled with accumulated dumps and were cleaned and resized, because on a system of this vintage a full partition produces failures that look like anything except a full partition.

Result

The system recovers cleanly, comes up on its own after a power event, and holds its controller communication. Zero production downtime after the recovery.

Legacy industrial systems are not badly built. They are correctly built for constraints nobody wrote down, by people you cannot ask.

What I took from it

Reading a startup chain end to end, in order, without assuming any link works, is a skill that transfers directly to systems built this decade. Most of what I do on modern estates is the same activity with better tooling and worse documentation.