Tuesday, 13 September 2022

Phaser3: Shader with microphone input (Web)

I want to do an audio input vizualizer with Phaser 3, I’m trying to get the mic input to a shader but I can’t find a way to make it work.

I have a basic understanding on shaders and I can work with textures that are images but I really don’t understand how to provide a sound. I checked a working example made in three.js: three.js webaudio - visualizer and I already managed to get the sound input from the microphone as an Uint8Array of 1024 numbers.

Here’s the shader I’m using:

// simplesound.gsl.js
#ifdef GL_ES
precision highp float;
#endif

precision mediump float;
uniform vec2 resolution;
uniform sampler2D iChannel0;

varying vec2 fragCoord;

void main() {
  vec2 uv = fragCoord.xy / resolution.xy;
  vec2 mu = texture2D(iChannel0, uv).rg;

  float y = uv.y - mu.x;
  y = smoothstep(0., 0.02, abs(y - 0.1));

  gl_FragColor = vec4(y);
}

And here's my scene code, trying to make it work:

import Phaser from 'phaser';
// This will provide the array mentioned above with code that will use `navigator.getUserMedia`.
import { setupAudioContext } from '../audiostream';

export default class MainScene2 extends Phaser.Scene {
  constructor() {
    super({ key: 'MainScene2' });
  }

  preload() {
    this.load.glsl('simplesound', '/static/simplesound.glsl.js');
  }

  create() {
    this.shader = this.add.shader('simplesound', 400, 300, 800, 600);

    // When the user presses the 'g' key we will start listening for mic input
    const GKey = this.input.keyboard.addKey('G');

    GKey.on('down', () => {
      setupAudioContext((array) => {
        // this array is the array mentioned above, in the three.js example they do something like creating
        // a texture from this input and providing that texture to the shader uniform. I tried different things but
        // nothing worked :(
        //
        // I tried using this.shader.setChannel0 and this.shader.setUniform but nothing seems to work as well.
      });
    });
  }
}

I've been trying to make this work for a while, but couldn't get anything :(



from Phaser3: Shader with microphone input (Web)

No comments:

Post a Comment