Efficiently updating ARSCNFaceGeometry from a set of blend shapes

I am using ARSCNFaceGeometry and need to update the face model's blend shapes as part of my game loop. My current solution is to call ARSCNFaceGeometry.update with a new ARFaceGeometry:

class Face {
    let geometry: ARSCNFaceGeometry
    
    init(device: MTLDevice) {
        guard let geometry = ARSCNFaceGeometry(device: device, fillMesh: true) else {
            fatalError("Could not create ARSCNFaceGeometry")
        }
        self.geometry = geometry
    }
    
    func update(blendShapes: [ARFaceAnchor.BlendShapeLocation: NSNumber]) {
        let faceGeometry = ARFaceGeometry(blendShapes: blendShapes)!
        geometry.update(from: faceGeometry)
    }
}

However this is not suitable for realtime usage since the ARFaceGeometry line alone takes around 0.01 seconds (just for reference, at 60fps we have a total frame budget of 0.0166 seconds).

After a few hundred updates or so, we seem to run to some sort of bug that takes the entire game loop to 10-20fps. Here's a 6 second sample from the profiler that sees ARFaceGeometry taking 2.3 seconds!:

Is there a more efficient way to update an existing ARSCNFaceGeometry from a set of blend shapes?

With custom 3D models for example, I can just update the blend shape values on SCNNode.morpher. Is there an equivalent for ARSCNFaceGeometry?

from Efficiently updating ARSCNFaceGeometry from a set of blend shapes