INTERBUS ring fault diagnosis

The fault

A 2008 asphalt plant had been stopping at random for weeks. The control system, a SCADA still running on Solaris 8, would throw a bus error and take the whole plant down with it. No pattern anyone could see, no warning, and production lost every time it happened.

The conditions were the ones that make this kind of work slow. No spare parts on site. Limited manufacturer support for equipment this old. And documentation that turned out to be three different drawing revisions mixed into one folder, so any two pages might describe different plants.

Reading the log as a timeline

The obvious move is to work the alarms. There were ten of them on every event, and ten alarms feels like ten things to investigate.

Laying the fault logs out as a timeline instead made the shape obvious. Every single event had the same signature: one gateway error first, then ten separate alarms behind it. The ten were not ten problems. They were ten subsystems each correctly reporting that they had lost the thing they depended on.

One physical event, ten consequences. Everything after the first line was noise generated by the first line.

Rebuilding the topology from paper

The error message named a segment and a module number, which is only useful if you can say which physical device those numbers point at. Nobody could, because no current drawing of the ring existed.

So the electrical drawings got read page by page and the INTERBUS ring was reconstructed: nine cabinets, each carrying a bus terminal and its I/O modules, in the order the ring actually visits them. That reconstruction is what turns Seg./Mod. in an error string into a specific card on a specific rail in a specific cabinet.

Half a day of reading drawings. It felt like the slow option at the time.

The test that settled it

With a candidate location, the question was whether the fault sat in the cable between cabinets or in a module on the rail. Both produce a bus error, and both were plausible.

Rather than argue about it, I broke the system deliberately and watched how the error changed. Unplugging one inter-cabinet cable on purpose made the system report Mod.=0. The real fault, every time it had occurred, reported Mod.=4.

Different number, different failure. The cable between cabinets was not it. The problem was the fourth module on that rail, and now it had an address.

What was actually wrong

An eight channel analog input card carrying the signals of three weigh scales: aggregate, filler and mixer. Its local bus socket had a poor contact.

Cleaned and reseated properly. Fault closed.

Weeks of unpredictable plant stoppages, and the cause was a connector that was not quite home. That is not unusual on plant of this age; what is unusual is how expensive it is to find without a map.

What it left behind

By the end, the plant's INTERBUS topology and its complete I/O lists existed in one document for the first time. Previously that knowledge was distributed across three drawing revisions and several people's memories.

The next fault on this plant starts with a map. That is worth more than the repair.

Three things I took from it

• The first line in the log is the cause. The rest are consequences. Ten alarms invite you to investigate ten things. Sorting by time and reading only the first entry is faster and more often correct.
• Going to the panel without reading the drawings is expensive. Half a day on drawings, then half an hour in the field. Reversing that order costs days, because without the topology you are testing at random.
• On legacy systems, breaking it on purpose is the most reliable diagnostic you have. There is no simulator, no vendor lab and no test rig. Inducing a known fault and comparing how the error message changes turns guesswork into a measurement.

The same plant's control system needed a separate recovery at the operating system layer, which is written up in Solaris 8 plant recovery.