Disable the automatic scroll AppBarLayout is doing when expanded

I have an AppBarLayout that expands if the user scrolls down the list a little bit and it comes back up after 2 seconds if the user is not scrolling anymore. However, the annoying thing is that AppBarLayout is also scrolling the RecyclerView for the amount of AppBarLayout's height. That way it keeps the list at the same position where the user stopped in the list so the list doesn't move, however, the viewport does and I would like to disable that if possible.

The only way it kind of works is when I placed ViewPager (in it is later on Fragment that contains RecyclerView) above the AppBarLayout and removed app:layout_behavior.

<androidx.coordinatorlayout.widget.CoordinatorLayout
    android:layout_width="0dp"
    android:layout_height="0dp"
    app:layout_constraintBottom_toBottomOf="parent"
    app:layout_constraintEnd_toEndOf="parent"
    app:layout_constraintStart_toStartOf="parent"
    app:layout_constraintTop_toBottomOf="@id/divider_logo">

   <androidx.viewpager.widget.ViewPager
        android:id="@+id/view_pager"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        app:layout_behavior="@string/appbar_scrolling_view_behavior" />

    <com.google.android.material.appbar.AppBarLayout
        android:id="@+id/app_bar"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:background="@android:color/white"
        app:elevation="0dp"
        app:layout_behavior="com.my.app.sava.main.AppBarLayoutBehavior">

        <androidx.constraintlayout.widget.ConstraintLayout
            android:layout_width="match_parent"
            android:layout_height="wrap_content"
            app:layout_scrollFlags="scroll|enterAlways">

            <View
                android:id="@+id/tv_location_bg"
                android:layout_width="0dp"
                android:layout_height="0dp"
                android:background="@color/colorPrimaryLight"
                app:layout_constraintBottom_toBottomOf="@id/tv_location"
                app:layout_constraintEnd_toEndOf="parent"
                app:layout_constraintStart_toStartOf="parent"
                app:layout_constraintTop_toTopOf="@id/tv_location" />
               .
               .
               .

However there is a delay when wanting to scroll then because AppBarLayout is scrolled first without the list moving, then the RecyclerView is starting to scroll.

from Disable the automatic scroll AppBarLayout is doing when expanded