Formatting numbers in compose TextField

I am trying to create a reusable NumberField component:

@Composable
fun NumberField(
  value: Number?,
  onNumberChange: (Number) -> Unit,
) {
  TextField(
    value = value?.toString() ?: "",
    onValueChange = {
      it.toDoubleOrNull()?.let { value ->
        if (value % 1.0 == 0.0) {
          onNumberChange(value.toInt())
        } else {
          onNumberChange(value)
        }
      }
    },
    singleLine = true,
    keyboardOptions = KeyboardOptions(keyboardType = KeyboardType.Number)
  )
}

To be used as:

@Composable
fun NumberContent() {
  val number = remember { mutableStateOf<Number?>(null) }

  NumberField(value = number.value) {
    number.value = it
  }
}

I would like the number to be an Int or Double depending on what the user is typing. What I have above works until you try to enter a decimal number, as it seems "5.", does not parse as double. I want to allow the user to type 5. and then fill in rest. As such I don't want to add a zero after decimal automatically because that might not be the next number they want to enter. Is this the best way to go about it? I know that I can just accept any text and then try to format the text they entered later as an int or double and make them fix it then, just thought it would be nice to bundle it all in the composable.

from Formatting numbers in compose TextField