FFmpeg - RTMP streaming from Node, stream is faster than realtime

My goal is to render a canvas in Node, and stream that canvas to an RTMP server (Twitch ultimately, but testing on a local RTMP server). The standard way to stream to RTMP seems to be ffmpeg, so I'm using that, spawned as a child process from within node. I've tried a bunch of different combinations of techniques and ffmpeg params to get a consistent framerate and a stream at "realtime" speed, but can't figure it out. Here's the paths I've gone down so far

Render canvas and send input in continuous interval

import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  '-re',
  '-framerate', String(.fps),
  '-r', String(fps),

  '-i', '-',
  
  '-vcodec', 'libx264',
  '-r', String(fps),
  '-s', '1920x1080',
  '-g:v', String(2*fps),
  '-c:a', 'aac',
  '-f', 'flv', 'rtmp://127.0.0.1/live'
]);
ffmpeg.stdout.pipe(process.stdout)
ffmpeg.stderr.pipe(process.stderr)


const send = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  ffmpeg.stdin.write(canvas.toBuffer())
  setImmediate(() => send())
}
send()

Observations

• Took about 35 seconds for the stream to actually start (I think because of ffmpeg needing some amount of time to analyze the input?)
• Frame rate extremely below what I set it to, and "speed" also very low, although I'm not 100% sure what this means. example log Frame= 906 fps=3.9 q=29.0 size= 311kB time=00:00:27.83 bitrate= 91.5kbits/s speed=0.119x
• Stream behavior
  • Takes about a minute to load once opened in VLC
  • Timer on the stream starts about 1 minute behind real time, stays stuck on a single second for 30+ seconds, then shoots up a few seconds quickly, and gets stuck again

I had a hunch here that at least some of the reason for the strange behavior was that rendering the canvas in the same loop that I send input to ffmpeg in was too slow to achieve 30 FPS.

Render canvas in separate interval from ffmpeg input interval

Only render canvas FPS-times per second

Continue sending input to ffmpeg as fast as possible

import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ctx.fillStyle = 'red'
  ctx.fillRect(0, 0, 1920, 1080);
  ctx.font = '100px Arial';
  ctx.fillStyle = 'black'
  ctx.fillText(new Date().toLocaleString(), 500, 500);
  buffer = canvas.toBuffer();
  setTimeout(() => render(), 1/fps)
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setImmediate(() => send())
}
send()

Observations

• ffmpeg starts streaming almost immediately
• fps starts out around 16, takes a couple seconds to hit 28, and then ~30 more seconds to hit 30fps. speed much closer to 1x, but not quite all the way. example log frame=15421 fps= 30 q=29.0 size= 4502kB time=00:08:31.66 bitrate= 72.1kbits/s speed=0.994x
• Stream behavior
  • Takes about 5 seconds to load once opened in VLC
  • Timer stays stuck on the same second for multiple minutes

My hunch here for the stream being stuck on 1 timestamp is that while ffmpeg is sending frames out at 30 frames per second, I'm sending it frames much quicker than that. So in the first 1st of a second of streaming

1. Canvas renders with timestamp T 30 times
2. send runs N times where N is likely way higher than 30, sending ffmpeg N frames with the current timestamp
3. ffmpeg now has N frames with timestamp T on them, but can only send them out 30 per second, so it takes more than 1 second for the timestamp on the screen to change

Only send ffmpeg a frame every 1/FPS second

Same as before, but instead of sending ffmpeg frames as quickly as possible, only send it FPS frames every second.

import { createCanvas } from 'canvas';

const canvas = createCanvas(1920, 1080);
const ctx = canvas.getContext('2d');

let buffer = canvas.toBuffer();

const fps = 30;
const ffmpeg = spawn('ffmpeg', [
  ...same as before
]);

const render = () => {
  ...same as before
}
render();

const send = () => {
  ffmpeg.stdin.write(buffer)
  setTimeout(() => send(), 1/fps)
}
send()

Observations

• ffmpeg takes a few seconds to start streaming
• fps starts out high, around 28, and over the next minute or so drops down to 16. Speed drops along with it. example log frame= 1329 fps= 16 q=29.0 size= 463kB time=00:00:41.93 bitrate= 90.5kbits/s speed= 0.5x
• Stream behavior
  • Takes about 10 seconds to load once opened in VLC
  • Timer increases about twice as fast as expected, then gets hung on one second for a bit, and then starts increasing again at same rate

---

I'll stop there, but tl;dr I've tried a whole bunch of different combinations of -re, -framerate, -fps_mode, -r ffmpeg args, and some other techniques in the code like continuing to use setImmediate to send frames, but use a date comparison to actually send a frame at an FPS rate. I'm sure there's probably some fundamental video streaming knowledge I'm missing, so I'm just looking for any sort of guidance on how I can get my canvas to stream at a "realtime" speed, or whatever I could be missing here.

from FFmpeg - RTMP streaming from Node, stream is faster than realtime