ViewMatrix calculation messes up models

I have a WebGL application which renders an array of cubes, each with a different height. To size them I use the transformation matrix to scale the cubes. I also have a camera, which can be moved with the keyboard.

To calculate the model matrix I use the following code:

let matrix = matrix4Identity();
matrix = matrix4Translate(matrix, this.position);
matrix = matrix4Rotate(matrix, this.rotation[0], [1, 0, 0]);
matrix = matrix4Rotate(matrix, this.rotation[1], [0, 1, 0]);
matrix = matrix4Rotate(matrix, this.rotation[2], [0, 0, 1]);
matrix = matrix4ScaleWithVector(matrix, this.scale);

and to calculate the view matrix I use this code:

let matrix = matrix4Identity();
matrix = matrix4Rotate(matrix, camera.rotation[0], [1, 0, 0]);
matrix = matrix4Translate(matrix, [camera.position[0], camera.position[1], camera.position[2]]);
return matrix4Inverse(matrix);

and this is the vertex shader code I use:

attribute vec3 vertices;
attribute vec3 normals;

uniform mat4 projectionMatrix;
uniform mat4 viewMatrix;
uniform mat4 modelMatrix;

void main(void) {
    vec4 worldPosition = modelMatrix * vec4(vertices, 1.0);
    vec4 modelPosition = viewMatrix * worldPosition;

    gl_Position = projectionMatrix * modelPosition;
}

However, if you move the camera, the cubes get translated. So when you move the camera upwards, the (visual) Y-position of the cubes change. Furthermore, if you rotate the camera, it actually rotates the cubes around their X-Axis. However, if I remove the scale transformation in the model matrix, the blocks don't change their position anymore on moving the camera.

You can see the project result here. https://renuo.github.io/stayOFline/ Use WASD to move in the X and Z axes, R to move up and F to move down and o+l to rotate the camera.

Update:

When first translating and the rotating the camera, I get the following behaviour:

from ViewMatrix calculation messes up models