Issues using custom testInstrumentationRunner

I've not found any post on Stack Overflow that helped, so I'm posting my own question.

I have a custom testInstrumentationRunner that when set doesn't run any tests. The logs says:

App restart successful without requiring a re-install.
Running tests
adb shell am instrument -w -m  --no-window-animation  -e debug false -e class 'com.my.app.MyFragmentTest' com.my.app.debug.test/com.my.app.CustomTestRunner
Connected to process on device

When running the normal androidx.test.runner.AndroidJUnitRunner the tests run, but fail, because I need the custom runner.

The runner is set up like the androidx one, in:

defaultConfig {
// testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner
   testInstrumentationRunner "com.my.app.CustomTestRunner"
}

It's also located in the com.my.app package in androidTest-folder

The code for the runner is:

package com.my.app

import ...

class CustomTestRunner: AndroidJUnitRunner() {
    override fun newApplication(cl: ClassLoader?, className: String?, context: Context?): Application {
        DexOpener.install(this)
        Timber.i("This should print hopefully")
        return super.newApplication(cl, ApplicationTest::class.java.canonicalName, context)
    }
}

Not sure if it matters, but here's the code for the application class, which extends the application class used for the app itself:

package com.my.app

class ApplicationTest: MyApplication() {
    override fun onCreate() {
        super.onCreate()
        appContext = applicationContext
        initializeDependencyInjection()
    }

    override fun initializeDependencyInjection() {}
}

I have a test-case I'm trying to run, which contains 7 tests. The androidx runner fails on 7/7 tests. But when I run with the custom runner, I get duration 0ms and 0/0 tests.

So it seems like Android isn't detecting this runner. I also get the same results using a random name in the testInstrumentationRunner, like testInstrumentationRunner "not.a.TestRunner"

from Issues using custom testInstrumentationRunner