Transform transition on button makes child SVG jump position at beginning and end of transition

I am currently creating a Stencil component library and one of the components is a navigation button which has an SVG icon that scales on hover to give a pop effect. To achieve this effect I put a scale transform on the SVG with an ease transition.

The problem is that when I hover over the button, just before the transitions starts, the icon jumps on some of the buttons instances (pretty randomly, depending on where on the page the button is on the page sometimes up, down, left and right) and then when the transient has ended it jumps back (sometimes not even the same amount or direction as the first jump)

<my-element></my-element>
    <my-element></my-element>
    <my-element></my-element>
    <script>
      customElements.define("my-element", class extends HTMLElement {
        constructor() {
          super()
            .attachShadow({mode: "open"})
            .innerHTML = `
                <style>
                  :host {
                    display: inline-block;
                  }

                  button {
                    display: block;
                    border-radius: 8px;
                    margin: 0;
                    padding: 0.6rem;
                    border: none;
                    cursor: pointer;
                  }

                  svg {
                    display: block;
                    width: 4rem;
                    height: 4rem;
                    padding: 0;
                    margin: 0;
                    -webkit-transition: all 1s ease;
                    -moz-transition: all 1s ease;
                    -o-transition: all 1s ease;
                    transition: all 1s ease;
                    transform: scale(1);
                  }

                  :host(:not([active]):not([disabled])) button:hover svg {
                    transform: scale(1.2);
                  }
                </style>
                <button disabled={this.disabled}>
                  <svg>
                    <use href="#circ">
                      <?xml version="1.0" encoding="UTF-8"?>
                      <symbol viewBox="0 0 24 24" id="circ">
                        <svg height="24" width="24">
                        <circle cx="12" cy="12" r="12" stroke="none" stroke-width="3" fill="red" />
                        </symbol>
                      </svg>
                    </use>
                  </svg>
                </button>`;
        }
      });
    </script>

This problem is browser agnostic and I've tried for quite a few hours now to figure out what's causing it. It seems removing all padding and margins resolves the issue but that's not really a solution. What I find very strange is if I put the sag inside of a collared div and add the scale transition to the div instead of the SVG, the div scales smoothly without the jump, but the SVG inside the div does the same weird jumps.

from Transform transition on button makes child SVG jump position at beginning and end of transition