Matplotlib: mouse outside of slider axes continues to change plot

I have a simple script that plots data in x- and y-, and have two scrollbars. One scrollbar lets you scroll through the data in x. The second scrollbar lets you change the range of x values you see at a given time (SSCE below). It works almost great: when I slide the position slider the plot slides along as expected. Sliding the 'Width' slider changes the range of data you see.

The one bug is that after I change the width, when I scroll my mouse past the right edge of the position scrollbar, it actually keeps changing the plot. Gif:

I am sort of at a loss I am not sure why it is registering any events once the mouse leaves the scrollbar axes.

Related question
Matplotlib Updating slider widget range

Code to reproduce problem

%matplotlib qt 

import numpy as np
import matplotlib.pyplot as plt
from matplotlib.widgets import Slider

x = np.arange(0, 20, 0.01)
y = np.sin(2*np.pi*x)+np.random.normal(0, 1, x.shape)
width_init = 3

delta = x[1]-x[0]
position = x[0]
final_value = x[-1]-width_init

fig, ax = plt.subplots()
plt.subplots_adjust(bottom=0.25)
plt.plot(x, y, linewidth = 0.5, color = 'grey')
plt.xlim(position, position + width_init)

# position slider
slider_pos = plt.axes([0.2, 0.1, 0.65, 0.03]) 
spos = Slider(slider_pos, 'Pos', valmin = x[0], 
              valmax = final_value, valinit = position)
# width slider
slider_width = plt.axes([0.2, 0.05, 0.5, 0.03])
swidth = Slider(slider_width, 'Width', valmin = delta*4, 
                valmax = x[-1]-delta, valinit = width_init)

def update(val):
    position = spos.val
    width = swidth.val
    slider_pos.valmin = x[0]
    slider_pos.valmax = x[-1]-width-delta
    if width < width_init:
        spos.valmax = x[-1]-width
    if position+width > x[-1]:
        spos.set_val(slider_pos.valmax)
        
    slider_pos.axes.set_xlim(slider_pos.valmin,slider_pos.valmax)   
    ax.set_xlim(position, position+width if position+width <= x[-1] else x[-1])
    fig.canvas.draw_idle()
    
spos.on_changed(update);
swidth.on_changed(update);  

from Matplotlib: mouse outside of slider axes continues to change plot