matplotlib share xaxis with yaxis from another plot

If I want to tie the x and y axis of two separate axes together so that they zoom together I usually do something like this:

fig = plt.figure()
ax1 = fig.add_subplot(121)
ax2 = fig.add_subplot(122,sharex=ax1, sharey=ax1)

But I don't know how to share the xaxis of one plot with the yaxis of another plot. For example the xaxis of one plot is 'time' and I want to share that with the yaxis of another plot which also represents 'time'. Something like this (which doesn't work...):

ax2 = fig.add_subplot(122,sharex=ax1.yaxis, sharey=ax1.xaxis)

Thanks

from matplotlib share xaxis with yaxis from another plot