Automating TVs that no driver can attach to

This is the testing half of the TV lab. The other half — getting a hand on a television at all, from anywhere — is its own story, and everything here assumes it.

Two platforms, two answers

Vidio ships to six TV operating systems, and they split cleanly into two groups that needed opposite treatment.

Android TV had something to adopt: Maestro, which also covers our Android and iOS phone apps.

The web-engine family — Tizen, webOS, VIDAA, Whale, Coolita — had nothing that worked. That’s where RETINA (React TV New Automation) came from, built in-house with the TV app team.

Why it has to be real televisions

The honest answer is that identical code does not behave identically on these devices. A flow works perfectly on one television and breaks on the next, and device-specific bugs like that were frequent enough to stop being surprising.

For a streaming service the exposure is concentrated in one place: the player. And the player’s test matrix is wide. Live versus VOD. Free versus paid. DRM versus non-DRM. With ads and without. Multiply that by six platforms and a dozen vendor firmwares and you have a surface no emulator will honestly cover.

Before RETINA, all of it was manual. A full pass took two to three days, depending on how busy the team was — which meant in practice it was often the thing that got shortened.

Why we couldn’t use an existing driver

We tried a commercial device farm first. It was less reliable than we expected, and the pricing is time-based, which is a bad fit for automation — the more you run, the more it costs, exactly backwards from what you want a test suite to encourage.

The deeper problem is simpler. There is no WebDriver on these televisions, and no remote debugging port. Playwright, Selenium and WebDriver all assume they can reach into a browser from the outside and take control. On Tizen, webOS, VIDAA, Whale and Coolita there is no such door. Appium doesn’t help either — it wants a device automation layer these platforms don’t expose.

Every standard tool assumes the same thing, and the assumption is false here.

Turning the connection around

If you can’t attach to the device from outside, let the device call you.

RETINA is a small JavaScript agent bundled into the TV app itself. When the app starts in debug mode it loads the agent, and the agent opens a socket.io connection outward to a broker running on a mini PC in the office. It is present only in debug builds — it never ships to production.

The test suite connects to that same broker as a peer. Now both ends are clients of a message relay in the middle, and the broker can address one specific television or broadcast to every one of them at once.

Through that channel the agent exposes three primitives, which turns out to be enough:

• press a key — synthesised keyboard events inside the page, so D-pad navigation works exactly as the app sees it
• evaluate an expression — read anything the page knows about itself
• wait for an XPath — poll until an element appears, or time out

Everything else is composed from those three.

There was no long detour to get here, and I think that’s worth saying plainly rather than dressing it up. Once you accept that nothing can attach from the outside, let the device dial out is close to the only shape left. The constraint narrowed the design far more than any decision I made — which is the opposite of how the control plane went, where I picked the wrong thing first and had to back out of it.

The part that was actually hard

A message relay that can broadcast is easy to write and awkward to test against, because replies are not naturally tied to requests. When one controller is driving several televisions at once, every response arrives on the same channel, and a result meant for the Samsung will happily be read as the answer for the LG.

That correlation problem is where my own contribution mostly sits. I wrote the Robot Framework driver that made RETINA usable as a test framework rather than a protocol: keeping per-device response state, matching each reply back to the request and the television that produced it, discarding stale answers left over from a previous step, and retrying when a response came back from the wrong device. None of that is glamorous, and all of it is the difference between a demo and a suite you can leave running.

Built with the app team

RETINA was not a solo project, and the split is worth being precise about. The in-page agent was written by a TV software engineer — my JavaScript at the time was not strong enough to build it well. What I brought was the idea and the design of how the pieces should talk to each other, the requirements the agent had to satisfy, and the driver on the other end that made it drivable from a test.

The useful lesson was narrow and practical: I did not need to be the person who could write the agent. I needed to know exactly what it had to do.

Later the web suite moved from Robot Framework to Playwright — done by colleagues, not me. RETINA itself didn’t change. Only the framework talking to it did, which is a reasonable sign the protocol boundary was drawn in the right place.

Sharing devices with humans

Automation and people compete for the same physical televisions, so runs take a lock on a device before they touch it, and a person can deliberately occupy a set for a bounded window when they need it for debugging. It sounds like an administrative detail. It is the reason the two uses of the lab coexist instead of quietly corrupting each other’s results.

Maestro, and giving something back

On the Android side, being a heavy Maestro user meant finding its rough edges. When the team hit something, I looked into it and sent a fix upstream:

• Fix Android parallel/shards race condition
• Fix exit code for a single test

Both are small. Both were things that would otherwise have needed a workaround in our own pipeline forever, and an exit code that lies to CI is the kind of bug that quietly erodes trust in every green build.

What changed

11 web-engine televisions now run automated tests across brands, on top of the Maestro suites for Android and iOS.

A full pass went from two to three days of manual work to about one day. Coverage rose roughly 10% — a modest-sounding number that undersells it, because the added coverage is concentrated on the player, which is where a streaming company actually lives or dies.

The change that mattered most is not in either figure: we can now deploy to production daily. Player regressions get caught before release rather than after, and the release cadence stopped being gated by how much manual checking anyone had time for.

Where it stops

RETINA asserts on the DOM. It can tell you the player element reports that it is playing. It cannot tell you that the picture is actually moving on the panel.

For most flows that distinction is academic. For a streaming app it eventually isn’t — a black frame behind a perfectly healthy player state is precisely the bug you least want to ship, and it is invisible to every assertion described above.

The interesting part is that the missing piece is already in the room. Every television in the lab has a camera pointed at it for remote access. Closing that loop — asserting on what the camera sees, and not only on what the page says about itself — is the obvious direction the day the DOM stops being enough.