Best use of Artificial Intelligence

How We Built a Real-Time AI Commentary System for Visually Impaired NBA Fans

TL;DR: FCB New York and Michelob Ultra challenged WAV to build a system that would let visually impaired fans at Madison Square Garden follow live Knicks game action through real-time audio commentary — with no more than a 5-second delay from play to announcement. The system was delivered, deployed at MSG, and certified by the American Foundation for the Blind.

The Fan First Experience

Visually impaired fans don’t go to basketball games for the box score. They go for the same reason everyone does — 19,000 people in a room reacting to the same moment at the same time. The roar when a three drops. The groan when it rims out. That collective experience doesn’t require sight.

What it does require is a thread connecting the sound of the crowd to what just happened on the court. Without it, you’re surrounded by a reaction you can’t decode.

FCB New York and Michelob Ultra came to WAV with a precise brief: build that thread. A system that listens to the live game feed and speaks — calmly, in real time, into a fan’s ear — before the crowd noise has faded. On a standard mobile browser. No app required. No special hardware. And certified as genuinely accessible by the American Foundation for the Blind.

The 5-second requirement wasn’t a preference. It was the brief. Miss it and the commentary arrives after the crowd has already reacted, which tells a visually impaired fan nothing useful about what just happened.

What a Fan Actually Hears

A Knicks possession ends in a made three-pointer. Within 5 seconds, a listener hears:

That’s it. Clean, fast, specific. No delay long enough to break the connection between what happened and the crowd noise surrounding it.

During timeouts, the system fills the air with relevant stats — team shooting percentages, the point leader, recent run differentials. Between plays, when the game clock is running but the ball isn’t moving, color commentary bridges the silence. The hard rule built into the system: nothing longer than 15 seconds of dead air, ever.

The commentary library covers every scoreable play type in the NBA: two-point and three-point makes, layups and jump shots, assisted and unassisted. Each clip runs under 2.5 seconds. Each play type has two delivery modes — standard and “agitated” — so a go-ahead basket in the fourth quarter sounds different from a routine mid-range jumper in the first.

No App, Nothing to Install, Just a mobile phone, Wi-Fi or Data and standard Browser.

The AFB Certification

The American Foundation for the Blind certification requirement shaped every surface-level decision. The player had to work on any mobile browser — not a dedicated app, not a Bluetooth-paired device, not anything requiring setup at the seat. Screen-reader compatibility and assistive technology interoperability were non-negotiable.

This meant the audio had to be delivered over HLS — the same protocol that powers every major streaming service — and the interface had to be operable without visual reference. A fan should be able to open a URL, press play, and hear the game. That’s the full interaction model.

Certification isn’t a badge in this context. It’s evidence that the system was tested against actual use by actual visually impaired fans and held up.

The Build: For Those Who Want to Know How It Works

The 5-second clock starts the moment SportRadar’s live game feed registers a play. It ends when commentary reaches a listener’s ear.
Everything in between is a pipeline that had to be engineered to run faster than it naturally wanted to.

The Data Problem

Play data from SportRadar doesn’t arrive as a single complete event. A made basket comes through in sequential packets — the play type first, then the shot classification, then the player identity. Waiting for all of it adds 3–4 seconds the budget doesn’t have. The filtering engine — a Node.js service sitting between the SportRadar feed and the rest of the system — was built to act on the minimum viable information: play confirmed, event type confirmed, primary player resolved. Everything after that is discarded for latency-sensitive paths.

Dead air problem

SportRadar sends a heartbeat signal every 5 seconds when no plays are coming through. The system uses this as a clock: 5 seconds of silence that doesn’t coincide with a timeout or period end triggers color commentary automatically. Timeouts and period ends trigger pre-loaded stat scripts. This is how the system maintains a constant audio presence throughout the game without requiring a human operator.

Audio Assembly

Once a play is classified, a pre-rendered audio library does the heavy lifting. Generating voice from a language model in real time would eat the entire latency budget. Instead, hundreds of scripts covering every play type, delivery mode, and score-call variation were produced and indexed in advance. At runtime, FFMPEG stitches the relevant clips into a single track, LiquidSoap manages the stream switch, and SRS distributes the audio across HLS, WebRTC, and SRT simultaneously.

The numbers.

End-to-end target: 5 seconds. Achieved: 4.8 seconds. The 0.2-second margin is real and accounted for — it comes from the filter-to-audio handoff running at 200 milliseconds, which leaves headroom for network variance in a stadium Wi-Fi environment.

What This Was, Actually

The Dreamcaster franchise was already proof that accessibility and spectacle aren’t in conflict. Courtside extended that proof into real-time infrastructure — the kind of build where the experience only works if the engineering is invisible.

A visually impaired fan at MSG shouldn’t be aware they’re listening to a system built on SportRadar data, a custom filtering engine, LiquidSoap, and SRS. They should just know the game. That’s the brief. That’s what shipped.

Frequently Asked Questions

What does the fan experience actually feel like?

A visually impaired fan opens a URL on their phone, presses play, and hears live commentary within seconds of each play. No app download, no pairing, no setup at the seat. The commentary arrives before the crowd noise has fully settled — which means it functions as context, not recap.

Does it work on bad Wi-Fi?

It was designed specifically for stadium Wi-Fi, which is one of the worst network environments for streaming. The audio format — AAC at 16kHz, 32kbps — was chosen for voice intelligibility at low bitrates, not audio fidelity. HLS delivery breaks the stream into small segments that handle buffering and reconnection gracefully. It holds up.

What does AFB certification mean in practice?

The American Foundation for the Blind reviewed the system against accessibility standards for visually impaired users — including screen-reader compatibility, interface operability without visual reference, and audio quality standards for speech intelligibility. Certification means it passed that review, not just internal testing.

What happens when nothing is happening on the court?

The system monitors the live data feed for silence. When 5 seconds pass without a play, it determines whether the game is in a timeout, a period end, or simply a live-ball pause. Each condition triggers a different audio response: stat scripts on structured breaks, color commentary during live-game pauses. Nothing longer than 15 seconds of silence is permitted.

Why not generate commentary with AI in real time?

The 5-second latency target doesn’t leave room for language model inference plus voice synthesis on the critical path. The pre-rendered library approach was chosen deliberately — scripts and audio are produced in advance, the system routes and assembles them at runtime. The result is consistent, predictable latency regardless of system load.

What sport or league could this be built for next?

The architecture is sport-agnostic. The SportRadar integration covers multiple leagues and sports. The commentary library structure — event type, shot type, player, assist — maps directly to most professional sports play-by-play taxonomies. Baseball, football, and soccer each have different event vocabularies but the same pipeline logic applies.

The Bottom Line

The challenge FCB New York brought to WAV was real: build a system that reliably connects a visually impaired fan to live game action, every possession, in under 5 seconds, at Madison Square Garden. The brief didn’t leave room for “close enough.”

We built to 4.8 seconds because we knew exactly where every tenth of a second in the pipeline was going. That’s what accessible production infrastructure looks like when it’s done right — invisible to the fan, and nothing left to chance in the build.

See more work