how to apply back and forth scalling to all objects in fabricJS?

I have multiple canvas in one screen using fabricJS.

I have added some ITexts, Text Box, Image, Background in each canvas.

I have some scalling options, like 25%, 50%, 100% to 300% in increasing order of 25%.

If my current default scale is 100% it means scalefactor is 1, now if I apply 125 or 150 or whatever then it is working fine with this code. but once I go down means 25 or 50 then again 150 or 175. then it starts to behave strange. my code is here. I tried to do scaling based on initial scalefactor but its not working as expected.

        const scalefactor = Number(this.selectedScale) / 100;
        this.canvas.forEach((canvas: any, index) => {

            this.canvas[index].setDimensions({
                width: Number(this.canvas[index].originalCanvasWidth * scalefactor),
                height: Number(this.canvas[index].originalCanvasHeight * scalefactor)
            });

            this.canvas[index].setZoom(scalefactor);


            if (this.canvas[index].backgroundImage) {
                // Need to scale background images as well
                let bi = this.canvas[index];
                bi.width = bi.originalBackgroundImageWidth * scalefactor;
                bi.height = bi.originalBackgroundImageHeight * scalefactor;
            }

            let objects = this.canvas[index].getObjects();
            for (let i in objects) {
                const scaleX = objects[i].scaleX;
                const scaleY = objects[i].scaleY;
                const left = objects[i].left;
                const top = objects[i].top;

                const tempScaleX = scaleX * scalefactor;
                const tempScaleY = scaleY * scalefactor;
                const tempLeft = left * scalefactor;
                const tempTop = top * scalefactor;

                objects[i].scaleX = tempScaleX;
                objects[i].scaleY = tempScaleY;
                objects[i].left = tempLeft;
                objects[i].top = tempTop;

                objects[i].setCoords();
            }

            this.canvas[index].renderAll();
            this.canvas[index].calcOffset();
        });

for canvas and background its working fine only for objects its not scalling properly as well as its position is not setting properly.

from how to apply back and forth scalling to all objects in fabricJS?