Friday, 30 April 2021

Multipeer connection onicecandidate event won't fire

I'm having problems with the logic to build behind the webRTC multi peer connections handling. Basically I'm trying to make a Room full of people in a videoconference call. I'm using the basic WebSocket library provided by js, and React for the frontend and Java (spring boot) for the backend.

As of my understanding right now this is what I managed to write down (filtered based on what I "think" is relevant)

This is my web socket init method (adding listeners)

let webSocketConnection = new WebSocket(webSocketUrl);
webSocketConnection.onmessage = (msg) => {
    const message = JSON.parse(msg.data);
    switch (message.type) {
    case "offer":
        handleOfferMessage(message);
        break;
    case "text":
        handleReceivedTextMessage(message);
        break;
    case "answer":
        handleAnswerMessage(message);
        break;
    case "ice":
        handleNewICECandidateMessage(message);
        break;
    case "join":
        initFirstUserMedia(message);
        break;
    case "room":
        setRoomID(message.data);
        break;
    case "peer-init":
        handlePeerConnection(message);
        break;
    default:
        console.error("Wrong type message received from server");
}

Plus of course the 'on error', 'on close' and 'on open' listeners This is the method handling the incoming offer

 const handleOfferMessage = (message) => {
    console.log("Accepting Offer Message");
    console.log(message);
    let desc = new RTCSessionDescription(message.sdp);
    let newPeerConnection = new RTCPeerConnection(peerConnectionConfig);
    newPeerConnection.onicecandidate = handleICECandidateEvent;
    newPeerConnection.ontrack = handleTrackEvent;
    if (desc != null && message.sdp != null) {
      console.log("RTC Signalling state: " + newPeerConnection.signalingState);
      newPeerConnection
        .setRemoteDescription(desc)
        .then(function () {
          console.log("Set up local media stream");
          return navigator.mediaDevices.getUserMedia(mediaConstraints);
        })
        .then(function (stream) {
          console.log("-- Local video stream obtained");
          localStream = stream;
          try {
            videoSelf.current.srcObject = localStream;
          } catch (error) {
            videoSelf.current.src = window.URL.createObjectURL(stream);
          }

          console.log("-- Adding stream to the RTCPeerConnection");
          localStream
            .getTracks()
            .forEach((track) => newPeerConnection.addTrack(track, localStream));
        })
        .then(function () {
          console.log("-- Creating answer");
          return newPeerConnection.createAnswer();
        })
        .then(function (answer) {
          console.log("-- Setting local description after creating answer");
          return newPeerConnection.setLocalDescription(answer);
        })
        .then(function () {
          console.log("Sending answer packet back to other peer");
          webSocketConnection.send(
            JSON.stringify({
              from: user,
              type: "answer",
              sdp: newPeerConnection.localDescription,
              destination: message.from
            })
          );
        })
        .catch(handleErrorMessage);
    }
    peerConnections[message.from.id] = newPeerConnection;
    console.log("Peer connections updated now ", peerConnections);
  };

SN: I got the peer connections defined as an array of RTCPeerConnection indexed by the user unique id

let [peerConnections, setPeerConnections] = useState([]);

And here comes the part that I think I got wrong and on which I'm having trouble understanding

  const handleAnswerMessage = (message) => {
    console.log("The peer has accepted request");
    let currentPeerConnection = peerConnections[message.from.id];
    if (currentPeerConnection) {
      currentPeerConnection.setRemoteDescription(message.sdp).catch(handleErrorMessage);
      peerConnections[message.from.id] = currentPeerConnection;
    } else {
      console.error("No user was found with id ", message.from.id);
    }
    console.log("Peer connections updated now ", peerConnections);

  };
    currentPeerConnection.setRemoteDescription(message.sdp).catch(handleErrorMessage);
    peerConnections[message.from.id] = currentPeerConnection;
    console.log("Peer connections updated now ", peerConnections);

  };

The answer and the offer work perfectly, I can clearly see the two peers communicating one by sending the offer and the other one responding with an answer. The only problem is that after that nothing happens, but from what I read about webRTC it should actually start gathering ice candidates as soon as a local description has been set.

I can understand why the peer handling the answer (caller) actually does not fire up iceecandidate and that's probably because I do not set a local description on the answer message (I don't know if it would be correct). the callee on the other hand, handling the offer message should actually start gathering iceecandidates tho, I'm setting the local description on there.

This some additional code that might help

function getMedia(constraints, peerCnnct, initiator) {
    if (localStream) {
      localStream.getTracks().forEach((track) => {
        track.stop();
      });
    }
    navigator.mediaDevices
      .getUserMedia(constraints)
      .then(stream => {
        return getLocalMediaStream(stream, peerCnnct, initiator);
      })
      .catch(handleGetUserMediaError);
  }

  function getLocalMediaStream(mediaStream, peerConnection, initiator) {
    localStream = mediaStream;
    const video = videoSelf.current;
    if (video) {
      video.srcObject = mediaStream;
      video.play();
    }
    //localVideo.srcObject = mediaStream;
    console.log("Adding stream tracks to the peer connection: ", peerConnection);

    if (!initiator) {
      localStream
        .getTracks()
        .forEach((track) => peerConnection.addTrack(track, localStream));
    }
  }

  const handlePeerConnection = (message) => {
    console.info("Creating new peer connection for user ", message.from);

    let newPeerConnection = new RTCPeerConnection(peerConnectionConfig);
    // event handlers for the ICE negotiation process
    newPeerConnection.ontrack = handleTrackEvent;
    newPeerConnection.onicecandidate = handleICECandidateEvent;
    getMedia(mediaConstraints, newPeerConnection, false);
    newPeerConnection.onnegotiationneeded = handleNegotiationNeededEvent(newPeerConnection, webSocketConnection, user, message.from);
    peerConnections[message.from.id] = newPeerConnection;
  };

Here you can clearly see my desperate attempt in finding a solution and creating a peer connection just for the sake of sending the offer. I cannot index a peer connection that has no end user because I would need his id, that I receive only after I received an answer from him when I first join the room.

(The backend should work but either way putting a debugger on the ice candidate handler method I could clearly see that it's just not fired)

What am I doing wrong?

EDIT: Now the WebSocketMessage Server side has also a destination user. This way the the new peer that connects to the room receives as many peer-init messages as the already connected peers are. Then proceeds to make one offer per peer setting it as a destination.

The problem still persists though



from Multipeer connection onicecandidate event won't fire

No comments:

Post a Comment