Archive · 14 Oct 2026

One line held a lock too long: a decoder deadlock, 13 October 2026

A review of our own code found that a file failing to open at a programme change, while a playlist load was waiting to write, would stop a channel's decoder for good. It had been in our first unit's builds for eleven days. We proved it with a deliberately slow open, fixed it with one line, and put the fix on air as a restart of its own.

Written once, on the date above. Corrections are added at the end as dated addenda, and nothing above them is changed silently.

What we found

On the night of 13 October 2026 a read-only review of the engine’s newest code, one of the probe-and-invert passes we run on our own work, turned up an older fault outside the code it was looking at. At every change from one programme item to the next, the decoder reads the next item’s Media ID from the schedule and opens its file. The read sat inside the expression the open’s result was matched on. In Rust (the 2021 edition the engine is built with), the temporaries of that expression live until the whole match ends, and the schedule’s read lock was one of them. So the decoder held that lock through the open, and through the branch that handles an open failing, which reads the schedule again on the same thread.

Rust’s standard reader-writer lock does not grant a new read while a writer is waiting. If something asked to write the schedule while the open ran, such as an operator loading a playlist or the background pass that measures files’ lengths, the writer waited for the decoder’s lock. If the open then failed, the decoder’s second read waited for the writer. Neither could go on.

What it would have done on air

Both halves had to coincide: a file that fails to open at the moment its item is due (missing, unreadable, or slow to fail, for instance on a stalled network path), and a schedule write arriving during that open. A missing local file fails in microseconds, so the window is narrow. When both happen the consequence is total for that channel. The decoder stops; nothing covers the gap, not even our safety net, which airs emergency material before black; nothing after it airs; and the playlist load that asked to write never returns, so every load after it waits too.

The fault was in every build on our first unit since 2 October 2026. We have not seen it happen on air, and our logs hold no sign that it did. We are not counting that as evidence that it could not.

How we proved it

We made the narrow window wide on purpose. In a cardless test of the engine, without the SDI card, the item due next was a named pipe rather than a file, so its open blocked until something wrote to the pipe. With the open held, the test loaded a new playlist, then closed the pipe so the open failed.

On the code then on air: the decoder thread sat in a lock wait, nothing aired after the item before, and the playlist load never returned.
With the fix: the load returned at once, the failed item was recorded as such, the safety net covered the gap, and the next item aired.
Two staged builds checked: we ran the same test on the two builds we had already staged for later restarts. Both deadlocked. Each was rebuilt with the fix before going any further, and the scripts that install them now refuse any build that does not carry what is already on air.

The fix, and getting it on air

The fix is one line moved: the Media ID is read into its own variable before the open, so the lock is released before the file is touched. It went on air on our first unit on 14 October 2026 at 00:00:28 UTC, as a restart of its own carrying nothing else. The channel was off air for 3.4 seconds. Every acceptance check after the restart passed, including the first items on their exact frames, no frame lost across three minutes of programme changes, and both channels’ compliance logs verifying across the restart. The night’s rotation of those logs passed too.