ItemTouchHelper not swiping after item removed from adapter data

have an issue with removing items from adapter using ItemTouchHelper. When first item is swiped and removed via updateItemRemoved() any other cannot be swiped. Looks like swiping for remaining items is "ignored" after first removal of any item.

Adapter:

class Adapter: RecyclerView.Adapter<BaseUserVH>() {

    private var data: List<User> = listOf()
    private var activeUserPosition = 0


    override fun onCreateViewHolder(parent: ViewGroup, viewType: Int) = when (viewType) {
        R.layout.active -> ActiveVH(parent.inflateView(R.layout.active))
        else -> InactiveVH(parent.inflateView(R.layout.inactive))
    }

    override fun getItemCount(): Int = data.size

    override fun onBindViewHolder(holder: BaseUserVH, position: Int) = holder.bind(data[position])

    override fun getItemViewType(position: Int) =
        if (position == activeUserPosition ) R.layout.active
        else R.layout.inactive

    fun selectUser(position: Int) {
        activeUserPosition = position
        notifyItemChanged(position)
    }

    fun setData(users: List<User>) {
        data = users
        notifyDataSetChanged()
    }

    fun updateItemRemoved(position: Int) {
        data.removeAt(position)
        notifyItemRemoved(position)
        notifyItemRangeChanged(position, itemCount)
    }
}

ItemTouchHelper & Fragment:

fun RecyclerView.swipeToDelete(
    onSwiped: (Int) -> Unit
) {
    val itemTouchHelperCallback = object : ItemTouchHelper.SimpleCallback(0, ItemTouchHelper.LEFT) {

        override fun onSwiped(viewHolder: RecyclerView.ViewHolder, direction: Int) {
            onSwiped(viewHolder.layoutPosition)
        }

        override fun onMove(
            recyclerView: RecyclerView,
            viewHolder: RecyclerView.ViewHolder,
            target: RecyclerView.ViewHolder
        ) = false

        override fun getMovementFlags(
            recyclerView: RecyclerView,
            viewHolder: RecyclerView.ViewHolder
        ) = if (adapter?.itemCount ?: 0 == 1 && viewHolder.layoutPosition == 0) 0
        else super.getMovementFlags(recyclerView, viewHolder)

    val itemTouchHelper = ItemTouchHelper(itemTouchHelperCallback)
    itemTouchHelper.attachToRecyclerView(this)
}

class Fragment : Fragment() {

    private val userAdapter = Adapter()

    override fun onCreateView(
        inflater: LayoutInflater,
        container: ViewGroup?,
        savedInstanceState: Bundle?
    ): View? = inflater.inflate(
        R.layout.fragment,
        container,
        false
    ).apply {
        with(recycler) {
            adapter = userAdapter 
            swipeToDelete { position -> userAdapter.updateItemRemoved(position) }
        }
    }

If I don't call data.removeAt(position), then deletion keeps going for other swipes

from ItemTouchHelper not swiping after item removed from adapter data