Disable gestures in parent custom control in iOS

I have a custom control "container" to manage gestures and a Skiasharp drawable control inside. The functionality I'm looking for is, in the first step the user can make zoom/move the image inside the container, then the gestures of the container are disabled and the user can draw in the visible image with the fingers. It used to work months ago, but when moved to xamarin 3.x it started to fail.

This is the code of the container

public class GestureContainer : ContentView
{
    private const double MIN_SCALE = 1;
    private const double MAX_SCALE = 4;
    private double startScale, currentScale;
    private double startX, startY;
    private double xOffset, yOffset;

    private PanGestureRecognizer pan;
    private PinchGestureRecognizer pinchGesture;
    private TapGestureRecognizer tap;

    public static readonly BindableProperty GestureOnProperty =
       BindableProperty.Create("GestureOn", typeof(bool), typeof(GestureContainer), defaultValue: true, defaultBindingMode: BindingMode.TwoWay, propertyChanged: GestureOnChanges);

    public bool GestureOn
    {
        get
        {
            return (bool)GetValue(GestureOnProperty);
        }
        set
        {
            SetValue(GestureOnProperty, value);
        }
    }


    private static void GestureOnChanges(BindableObject bindable, object oldvalue, object newvalue)
    {
        GestureContainer gestureContainer = bindable as GestureContainer;

        if ((bool)newvalue)
            gestureContainer.SetGestures();
        else
            gestureContainer.GestureRecognizers.Clear();

    }


    public GestureContainer()
    {
        pinchGesture = new PinchGestureRecognizer();
        pan = new PanGestureRecognizer();
        tap = new TapGestureRecognizer { NumberOfTapsRequired = 2 };

        SetGestures();

        Scale = MIN_SCALE;
        TranslationX = TranslationY = 0;
    }

    private void SetGestures()
    {
        pinchGesture.PinchUpdated += OnPinchUpdated;
        GestureRecognizers.Add(pinchGesture);

        pan.PanUpdated += OnPanUpdated;
        GestureRecognizers.Add(pan);

        tap.Tapped += OnTapped;
        GestureRecognizers.Add(tap);
    }

    private void OnTapped(object sender, EventArgs e)
    {
       /**/
    }

    void RestoreScaleValues()
    {
        /**/
    }

    void OnPinchUpdated(object sender, PinchGestureUpdatedEventArgs e)
    {
       /**/
    }

    void OnPanUpdated(object sender, PanUpdatedEventArgs e)
    {
       /**/
    }
}

The part with the "weird" function is this:

gestureContainer.GestureRecognizers.Clear();

When I launch the binded property to false, the method is called, the .Clear() is called, but no matter what the pinch/pan/tap are still working, and that makes the drawing touch not to work correctly

from Disable gestures in parent custom control in iOS