Can I use scalable pixels for layout dimensions?

Is anyone aware of any problems with using sp units for layout dimensions? So, say I have a 16sp text, is there any problem with setting TextView height to 20sp for example, so it scales with the text?

<TextView
    android:textSize="16sp"
    android:layout_width="match_parent"
    android:layout_height="20sp"
    ...

This way the padding around the text stays proportional to text size after scaling. Also, this is useful with auto sizing, where using wrap_content is not recommended.

However, the official docs state that sp should never be used for text size (without any explanation though):

When defining text sizes, however, you should instead use scalable pixels (sp) as your units (but never use sp for layout sizes).

What is the reasoning behind this? Both dps and sps are converted to a certain device-specific pixel amount in the end, are they not? What's the core difference then?

from Can I use scalable pixels for layout dimensions?