matplotlib set_major_formatter taking into account range of values

I've written a UI that includes a matplotlib figure. It shows graphs and I'd like to use engineering notation (exponents that are multiples of 3) on the axes. Note that I want the powers to be written out (e.g. "123×10⁶", not "123 M", hence not using EngFormatter).

I can do this with a formatting function (format_eng) I wrote many years ago and the set_major_formatter & fmt_x_data features of matplotlib:

def axes_eng_format(ax, x=True, y=True, **kwargs):
    if x:
        if 'use_si' not in kwargs and 'use_latex' not in kwargs:
            ax.xaxis.set_major_formatter(lambda v, pos: format_eng(v, use_latex=True, **kwargs))
        else:
            ax.xaxis.set_major_formatter(lambda v, pos: format_eng(v, **kwargs))
        ax.fmt_xdata = lambda v: format_eng(v, **kwargs)
    if y:
        if 'use_si' not in kwargs and 'use_latex' not in kwargs:
            ax.yaxis.set_major_formatter(lambda v, pos: format_eng(v, use_latex=True, **kwargs))
        else:
            ax.yaxis.set_major_formatter(lambda v, pos: format_eng(v, **kwargs))
        ax.fmt_ydata = lambda v: format_eng(v, **kwargs)

axes_eng_format(ax)

The formatting function itself shouldn't be relevant to this question (as it is just a dumb formatter that only takes a single parameter and hence cannot consider the range of values), however for reference it is here: https://gist.github.com/abudden/3252252632bd8271c49c886771d8f82d

The formatters only get two parameters: the value (v) and the position on the graph (pos). Using just the value v is okay and works, but the 0.5 is rendered as 500×10⁻³. If the range of values on the major axis is (say) 0 to 700×10⁻³, then that's exactly how I'd like it to be rendered, but if the range of values is (like the second image below) 0 to 3.5, I'd rather it were rendered as 0.5 for consistency with the other values on the axis.

The only way I can think of doing this at the moment is to post-process all the tick values, but that doesn't seem like it would be very robust given the user can zoom in and out etc.

Is there a better way of achieving the effect I'm after? I'm guessing I can do something with pos, but I haven't been able to get my head round what would work. How should I go about using pos to tweak the formatting of a number (or is there a better way)?

Y-axis looks good:

Y axis looks not-so-good:

from matplotlib set_major_formatter taking into account range of values