AVAudioSession plays sound on receiver OR mic don't tap

I have an app taps the microphone and also play sounds depending on mic input. This code below works. But one problem is the output plays on the small top speaker and not the bottom real loud speakers. I could solve this problem strangely by putting the 3 lines below just before the player starts, Then I can hear the sound on speakers. But then the microphone stops listening! Even after the player stops playing. Basically mic does not like when it is

.defaultToSpeaker

Any idea?

Here also documented what I am trying to do is correct:

https://developer.apple.com/documentation/avfoundation/avaudiosession/categoryoptions/1616462-defaulttospeaker

    try AVAudioSession.sharedInstance().setActive(false)
    try? AVAudioSession.sharedInstance().setCategory(.playAndRecord, options: [.defaultToSpeaker])
    try AVAudioSession.sharedInstance().setActive(true)

And this the tap:

    let sampleRate = Double(11025)
    let outputFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: sampleRate, channels: 1, interleaved: true)!
    let converter = AVAudioConverter(from: inputFormat, to: outputFormat)!
    let inputBufferSize = 44100 //  100ms of 44.1K = 4410 samples.
    let sampleRateRatio = 44100 / sampleRate
 input.installTap(onBus: 0, bufferSize: AVAudioFrameCount(inputBufferSize), format: inputFormat) {
        buffer, time in
        var error: NSError? = nil
        let capacity = Int(Double(buffer.frameCapacity) / sampleRateRatio)
        let bufferPCM16 = AVAudioPCMBuffer(pcmFormat: outputFormat, frameCapacity: AVAudioFrameCount(capacity))!
        ...
}

from AVAudioSession plays sound on receiver OR mic don't tap