Work Manager not scheduling Work with setInitialDelay

So I have a Worker that has to run the next day from when it's scheduled. So if the work is activated at today night 8PM, then I need the work to be executed at next day 9AM. So I am using OneTimeWorkRequest with a setInitialDelay().

Here is the code

val currentTime = System.currentTimeMillis()
// calculate the timestamp for next dat 9AM 
val calendar = Calendar.getInstance()

calendar.set(Calendar.HOUR_OF_DAY, 9) 
calendar.set(Calendar.MINUTE, 0)
calendar.set(Calendar.SECOND, 0)
// next day
calendar.add(Calendar.DAY_OF_MONTH, 1)

val tomorrowTime = calendar.timeInMillis
val timeDiffBetweenNowAndTomorrow = tomorrowTime - currentTime

Timber.i("Tomorrow date is ${calendar.timeInMillis}")
Timber.i("Difference between now and tomorrow ${timeDiffBetweenNowAndTomorrow}")

val randomWorkRequest = OneTimeWorkRequestBuilder<RandomWallpaperWorker>()
                    .setInitialDelay(timeDiffBetweenNowAndTomorrow, TimeUnit.MILLISECONDS)
                    .build()

WorkManager.getInstance().enqueue(randomWorkRequest)

But I checked and the work wasn't executed when I woke up the next day. Why isn't it being scheduled? Is there something wrong the way I calculate the timestamp of the next day?

from Work Manager not scheduling Work with setInitialDelay