SwiftUI custom Facebook Login with FB app installed

I'm trying to build a custom fb login in my iOS app, which is built in SwiftUI. The basic flow seems to work, I get a response when login gets cancelled. I even get a token when logging into the FB web interface, everything works fine for that.

The problem exists only when a user has the FB app on their phone (which is fairly common). My app initially opens the web interface, which then opens the native FB app. I log into the FB app, and it goes back to my app without my app seeing anything new.

The code below is the main login call. There is other code of course, but not sure what else is relevant. Remember, the login does work perfectly with the FB web interface, it only breaks with the FB app. Any help would be appreciated!

let permissions = ["email", "public_profile"]
loginManager.logIn(permissions: permissions, from: nil) { (result, error) in
    if error != nil {
        print("Login failed")
        // deal with failure
    }
    else if result?.isCancelled != true {
        let token = result!.token
        // store token
    } else {
        print("Facebook login cancelled")
        // handle cancel
    }
}

from SwiftUI custom Facebook Login with FB app installed