THREE.JS & Reality Capture - Rotation issue photogrammetry reference camera's in a 3D space

Thanks for taking the time to review my post. I hope that this post will not only yield results for myself but perhaps helps others too!

Introduction

Currently I am working on a project involving pointclouds generated with photogrammetry. It consists of photos combined with laser scans. The software used in making the pointcloud is Reality Capture. Besides the pointcloud export one can export "Internal/External camera parameters" providing the ability of retrieving photos that are used to make up a certain 3D point in the pointcloud. Reality Capture isn't that well documented online and I have also posted in their forum regarding camera variables, perhaps it can be of use in solving the issue at hand?

Only a few variables listed in the camera parameters file are relevant (for now) in referencing camera positioning such as filename, x,y,alt for location, heading, pitch and roll as its rotation.

Currently the generated pointcloud is loaded into the browser compatible THREE.JS viewer after which the camera parameters .csv file is loaded and for each known photo a 'PerspectiveCamera' is spawned with a green cube. An example is shown below:

The challenge

As a matter of fact you might already know what the issue might be based on the previous image (or the title of this post of course ;P) Just in case you might not have spotted it, the direction of the cameras is all wrong. Let me visualize it for you with shabby self-drawn vectors that rudimentary show in what direction it should be facing (Marked in red) and how it is currently vectored (green).

Things tried so far

Something we discovered was that the exported model was mirrored from reality however this did not affect the placement of the camera references as they aligned perfectly. We attempted to mirror the referenced cameras, pointcloud and viewport camera but this did not seem to fix the issue at hand. (hence the camera.applyMatrix4(new THREE.Matrix4().makeScale(-1, 1, 1));)

So far we attempted to load Euler angles, set angles directly or convert and apply a Quaternion sadly without any good results. The camera reference file is being parsed with the following logic:

// Await the .csv file being parsed from the server
    await new Promise((resolve) => {
      (file as Blob).text().then((csvStr) => {
        const rows = csvStr.split('\n');
        for (const row of rows) {
          const col = row.split(',');
          if (col.length > 1) {
            const suffixes = col[0].split('.');
            const extension = suffixes[suffixes.length - 1].toLowerCase();
            const validExtensions = ['jpeg', 'jpg', 'png'];
            if (!validExtensions.includes(extension)) {
              continue;
            }
            // == Parameter index by .csv column names ==
            // 0: #name; 1: x; 2: y; 3: alt; 4: heading; 5: pitch; 6: roll; 7:f (focal);
            // == Non .csv param ==
            // 8: bool isRadianFormat default false
            this.createCamera(col[0], parseFloat(col[1]), parseFloat(col[2]), parseFloat(col[3]), parseFloat(col[4]), parseFloat(col[5]), parseFloat(col[6]), parseFloat(col[7]));
          }
        }
        resolve(true);
      });
    });
  }

Below you will find the code snippet for instantiating a camera with its position and rotation. I left some additional comments to elaborate it somewhat more. I left the commented code lines in as well to see what else we have been trying:

private createCamera(fileName: string, xPos: number, yPos: number, zPos: number, xDeg: number, yDeg: number, zDeg: number, f: number, isRadianFormat = false) : void {
    // Set radials as THREE.JS explicitly only works in radians
    const xRad = isRadianFormat ? xDeg : THREE.MathUtils.degToRad(xDeg);
    const yRad = isRadianFormat ? yDeg : THREE.MathUtils.degToRad(yDeg)
    const zRad = isRadianFormat ? zDeg : THREE.MathUtils.degToRad(zDeg)

    // Create camera reference and extract frustum
    // Statically set the FOV and aspectratio; Near is set to 0,1 by default and Far is dynamically set whenever a point is clicked in a 3D space.
    const camera = new THREE.PerspectiveCamera(67, 5280 / 2970, 0.1, 1); 
    const pos = new THREE.Vector3(xPos, yPos, zPos); // Reality capture z = up; THREE y = up;

    /* ===
    In order to set an Euler angle one must provide the heading (x), pitch (y) and roll(z) as well as the order (variable four 'XYZ') in which the rotations will be applied 
    As a last resort we even tried switching the x,y and zRad variables as well as switching the orientation orders.
    Possible orders:
     XYZ 
     XZY
     YZX
     YXZ
     ZYX
     ZXY
       === */
    const rot = new THREE.Euler(xRad, yRad, zRad, 'XYZ');
    //camera.setRotationFromAxisAngle(new THREE.Vector3(0,))

    //camera.applyMatrix4(new THREE.Matrix4().makeScale(-1, 1, 1));
    // const rot = new THREE.Quaternion();
    // rot.setFromAxisAngle(new THREE.Vector3(1, 0, 0), zRad);
    // rot.setFromAxisAngle(new THREE.Vector3(0, 1, 0), xRad);
    // rot.setFromAxisAngle(new THREE.Vector3(0, 0, 1), yRad);
    // XYZ

    // === Update camera frustum ===
    camera.position.copy(pos);
    // camera.applyQuaternion(rot);
    camera.rotation.copy(rot);
    camera.setRotationFromEuler(rot);
    camera.updateProjectionMatrix(); // TODO: Assert whether projection update is required here
    /* ===
    The camera.applyMatrix listed below was an attempt in rotating several aspects of the 3D viewer.
    An attempt was made to rotate each individual photo camera position, the pointcloud itself aswell as the viewport camera both separately
    as well as solo. It made no difference however.
       === */
    //camera.applyMatrix4(new THREE.Matrix4().makeScale(-1, 1, 1));

    // Instantiate CameraPosition instance and push to array
    const photo: PhotoPosition = {
      file: fileName,
      camera,
      position: pos,
      rotation: rot,
      focal: f,
      width: 5120,  // Statically set for now
      height: 5120, // Statically set for now
    };

    this.photos.push(photo);
  }

The cameras created in the snippet above are then grabbed by the next piece of code which passes the cameras to the camera manager and draws a CameraHelper (displayed in both 3D viewer pictures above). It is written within an async function awaiting the csv file to be loaded before proceeding to initialize the cameras.

private initializeCameraPoses(url: string, csvLoader: CSVLoader) {
    const absoluteUrl = url + '\\references.csv';

    (async (scene, csvLoader, url, renderer) => {
      await csvLoader.init(url);
      const photos = csvLoader.getPhotos(); // The cameras created by the createCamera() method
      this.inspectionRenderer = new InspectionRenderer(scene);  // InspectionRenderer manages all further camera operations
      this.inspectionRenderer.populateCameras(photos);
      for (const photoData of photos) {
        // Draw the green cube
        const geometry = new THREE.BoxGeometry(0.5, 0.5, 0.5);
        const material = new THREE.MeshBasicMaterial({ color: 0x00ff00 });
        const cube = new THREE.Mesh(geometry, material);
        scene.add(cube);

        cube.position.copy(photoData.position);
        photoData.camera.updateProjectionMatrix();
        
        // Draws the yellow camera viewport to the scene
        const helper = new CameraHelper(photoData.camera);
        renderer.render(scene, photoData.camera);    
        scene.add(helper);
      }
    })(this.scene, csvLoader, absoluteUrl, this.renderer);
  }

Somehow I think that a solution is within reach and that in the end it'll come down to some very small detail that has been overlooked. I'm really looking forward into seeing a reply. If something is still unclear please say so and I'll provide the necessary details if required ^^

Thanks for reading this post so far!

from THREE.JS & Reality Capture - Rotation issue photogrammetry reference camera's in a 3D space