Add initial support for frame processor usage directly on tracks

[1egoman]

Updates the node sdk so that FrameProcessor-based noise cancellation providers can be used directly on AudioStream, without having to go through the agent's RoomIO to be able to initialize itself with credentials.

For example, with this change, something like the below becomes possible:

const stream = new AudioStream(track, {
  sampleRate: SAMPLE_RATE,
  numChannels: CHANNELS,
  noiseCancellation: aicAudioEnhancement({ model: 'quailVfL' }),
});

The way this works - Tracks now keep track of which room they are part of (holding a WeakRef value). When the room a track is in changes, it computes new frame processor options and sends these to any AudioStreams which are associated with the track.

The noiseCancellationLeaveOpen parameter allows the agents sdk to construct an AudioStream with a frame processor which remains open across the whole session, and won't be auto-closed when the track is closed.

Todo

• Test throughly (just ran a happy path test so far)

[changeset-bot]

🦋 Changeset detected

Latest commit: c1e6f43

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package

Name	Type
@livekit/rtc-node	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[1egoman]

Is this an ok addition to be making here? Or does there need to be a WeakRef polyfill which falls back to some sort of a no-op class wrapper?

(in case it's useful, WeakRef was introduced in node v16.4: mdn link)

[devin-ai-integration]

Devin Review found 1 potential issue.

View 4 additional findings in Devin Review.

[1egoman]

Going to wait for some of the conversation on the python version of this to shake out before continuing to push this forward.

[devin-ai-integration]

Devin Review found 1 new potential issue.

View 5 additional findings in Devin Review.

[devin-ai-integration]

🟡 Missing processor metadata update in localTrackRepublished handler causes stale/empty metadata after reconnection

The PR adds setRoom(this) calls in localTrackPublished, localTrackUnpublished, trackSubscribed, and trackUnsubscribed handlers to push processor metadata. However, the localTrackRepublished handler (fired during full reconnect when the SDK auto-republishes tracks) has no corresponding setRoom call or metadata push. After republish, the publication receives a new SID via publication.updateInfo(newInfo) at room.ts:567, but the track's info.sid (used by pushProcessorMetadataToStream at track.ts:125) is never updated. This means any subsequent call to pushProcessorMetadataToStream (e.g., when a new AudioStream is registered on the track after reconnection) will fail the lookup publication.sid === trackSid at track.ts:141 because the publication now has the new SID while the track still has the old one. The processor receives empty participantIdentity and publicationSid.

(Refers to lines 566-570)

Prompt for agents

In the localTrackRepublished handler in room.ts (around line 561-573), after the publication info is updated and re-keyed in the map, the track's processors are never notified of the new publication SID. Two things need to happen:

1. The track's own info.sid needs to be updated to match the new publication SID, because pushProcessorMetadataToStream in track.ts uses this.sid (which reads this.info?.sid) to look up the publication in the trackPublications map.

2. A metadata push should be triggered so that processors attached to existing AudioStreams on this track receive the updated stream info (new publicationSid).

A fix would be to add something like this inside the `if (publication)` block in the localTrackRepublished handler, after the publication is re-keyed:

  if (publication.track) {
    publication.track.info = { ...publication.track.info, sid: publication.sid }; // or however TrackInfo is updated
    publication.track.setRoom(this); // re-pushes metadata to all registered audio streams
  }

Note: setRoom(this) when oldRoom === room will still properly re-register the listener and push metadata to streams. However, be careful that the track's info structure is updated correctly (it's a protobuf TrackInfo object).

---

Was this helpful? React with 👍 or 👎 to provide feedback.