Viewpager2 with fragments and Jetpack navigation: Restore fragments instead of recreating them

I have a Viewpager2 inside a Fragment (lets call it HomeFragment). That Viewpager itself also contains Fragments. When I navigate away from the HomeFragment its view will be destroyed and when I navigate back the view will be recreated. Now I set the adapter of the Viewpager2 in the HomeFragment during onViewCreated(). Therefore the adapter will be recreated when I navigate back to the HomeFragment, which also recreates all Fragments in the Viewpager2 and the current item is reset to 0. If i try to re-use the adapter that I instantiated on the first creation of the HomeFragmenti get an exception, because of this check inside of the FragmentStateAdapter:

public void onAttachedToRecyclerView(@NonNull RecyclerView recyclerView) {
        checkArgument(mFragmentMaxLifecycleEnforcer == null);

Does anybody have an idea how I can prevent recreating everything when navigating back? Otherwise this is a pretty big performance overhead and hinders my UX.

from Viewpager2 with fragments and Jetpack navigation: Restore fragments instead of recreating them