`matplotlib` clickable plot in Panel row

I have figured out how to make a matplotlib graph clickable so that I can click a point and create a new plot based off of metadata for that point, following the answer here.

Now it comes time to put that clickable plot in a Panel dashboard. The obvious move is to add that figure to a panel.Row. While this displays the figure, I do not get the interactivity.

import matplotlib.pyplot as plt
import panel as pn

class custom_objects_to_plot:
    def __init__(self, x, y, name):
        self.x = x
        self.y = y
        self.name = name

a = custom_objects_to_plot(10, 20, "a")
b = custom_objects_to_plot(30, 5, "b")
c = custom_objects_to_plot(40, 30, "c")
d = custom_objects_to_plot(120, 10, "d")

def on_pick(event):
    my_fig, my_ax = plt.subplots() # New plot with unique name
    my_ax.scatter([1, 2, 3, 4], [5, 6, 7, 8]) # Make the scatterplot
    my_fig.show() # Show the plot

fig, ax = plt.subplots()
for obj in [a, b, c, d]:
    artist = ax.plot(obj.x, obj.y, 'ro', picker=5)[0]
    artist.obj = obj

fig.canvas.callbacks.connect('pick_event', on_pick)

my_row = pn.Row()
my_row.append(fig)
my_row.show()

How would I get the interactive (clickable) plot to appear like how it appears when I run from the command line?

from `matplotlib` clickable plot in Panel row