RecyclerView Jumps in middle of the list when paginatining

I have this page which I've setUp with mvvm and pagination using recyclerView and Pull to refresh. Clicking on some items in this recyclerView will navigate to another fragment.

My problem is whenever I load the page for the first time and scroll all the way down in works perfectly. Pull to refresh will work as well.

But when I navigate to the other fragment and get back, scroll all the way top, swipe to refresh : The recyclerView will jump to middle of the page( Right on the item where I clicked to navigate to the other fragment)

My Fragment

override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)
        val adapter = GroupAdapter<ViewHolder>().apply { add(mainSection) }
        val gridLayoutManager = GridLayoutManager(activity, adapter.spanCount)
        gridLayoutManager.spanSizeLookup = adapter.spanSizeLookup
        with(recyclerView) {
            layoutManager = gridLayoutManager
            setHasFixedSize(true)
            addOnScrollListener(PaginationScrollListener {
                viewModel.onEndOfList()
            })
        }
        recyclerView.adapter = adapter


        pullToRefresh.apply {
            setProgressBackgroundColorSchemeColor(
                ContextCompat.getColor(context, R.color.window_level_2)
            )
            setColorSchemeColors(ContextCompat.getColor(context, R.color.brand_primary))
            setOnRefreshListener { viewModel.onRefresh() }
        }
    }

My ViewModel

    fun onRefresh() {
        page = 0
        widgetItems.clear()
        _widgetListObservable.postValue(widgetItems)
        finishedLoading = false
        isFirstFetch = true
        getItems()
    }

private fun getItems() {
        isLoading = true
        dataSource.getPage(page)
            .subscribeOn(backgroundThread.getScheduler())
            .observeOn(mainThread.getScheduler())
            .flatMap {
                Flowable.fromIterable(it)
                    .toList()
                    .toFlowable()
            }
            .doAfterTerminate {
                isLoading = false
            }
            .subscribe(Consumer {
                finishedLoading = it.isEmpty()

                if (isFirstFetch && finishedLoading) {
                    _isMyPaymentsEmptyObservable.postValue(true)
                }
                widgetItems.addAll(it)
                _widgetListObservable.postValue(widgetItems)
                page++
                isFirstFetch = false
            }, {
              println(it)
            })

EDIT when I remove the onRefreshListener on the swipeToRefreshin the first fragment it works. I have no idea why this happens?

Thanks for reading.

from RecyclerView Jumps in middle of the list when paginatining