Sound scheduling issue when playing OPUS from websocket

I'm trying to use the library https://github.com/AnthumChris/opus-stream-decoder/

I have a stream of OPUS encoded sound (2ch, 48kHz) from a high quality microphone (but I play a music in loop on it to test this). I know it works because I can hear it if I use:

websocat --binary ws://third-i.local/api/sound - | mpv -

(It's opening the websocket and streaming its output to mpv (mplayer)).

But when I play in the browser all I hear is very small parts of the sound every second or so. But the sound itself sounds good (I believe it is a very small part of the music).

Here is the JS code I wrote to listen in the browser:

let audioWorker: any;
let exampleSocket;
let opusDecoder: any;
let audioCtx: any;
let startTime = 0;
let counter = 0;

function startAudio() {
  /*
  const host = document.location.hostname;
  const scheme = document.location.protocol.startsWith("https") ? "wss" : "ws";
  const uri = `${scheme}://${host}/api/sound`;
  */
  const uri = "ws://third-i.local/api/sound";
  audioCtx = new AudioContext();
  startTime = 100 / 1000;
  exampleSocket = new WebSocket(uri);
  exampleSocket.binaryType = "arraybuffer";
  opusDecoder = new OpusStreamDecoder({onDecode});
  exampleSocket.onmessage = (event) => opusDecoder.ready.then(
    () => opusDecoder.decode(new Uint8Array(event.data))
  );
  exampleSocket.onclose = () => console.log("socket is closed!!");
}

function onDecode({left, right, samplesDecoded, sampleRate}: any) {
  const source = audioCtx.createBufferSource();
  const buffer = audioCtx.createBuffer(2, samplesDecoded, sampleRate);
  buffer.copyToChannel(left, 0);
  buffer.copyToChannel(right, 1);
  source.buffer = buffer;
  source.connect(audioCtx.destination);
  source.start(startTime);
  startTime += buffer.duration;
}

https://github.com/BigBoySystems/third-i-frontend/blob/play-audio/src/App.tsx#L54-L88

from Sound scheduling issue when playing OPUS from websocket