Using two tables as input for multi-output regression with keras, the missing matrix product

In the keras MWE below I'm training a NN regression with 1000 samples taking 20 features (X) as input and producing outputs of size 50 (Y). However, I'm missing a step that I fail to wrap my head around and that I miss the word to describe properly. Let me try anyway, and please forgive the mess:

Each one of the 50 outputs is characterized by a set of 10 "feature filters" which are to interact with the 20 features to produce the numeric output. I miss a layer that would train a unique weight matrix of size (20, 10) whose sum subsequently produces the numeric output Y. The idea is that the output reacts to the features in ways that are dictated by those feature filters and that those interactions are consistent across outputs (e.g. high values in one feature filter might lead to a higher reaction to one feature and lower to another one, and those positive/negative relationships are not output-specific but identified for the whole dataset via the common weight matrix of size 10x20).

How could that side matrix (10, 50) of output-specific "feature filters" could enter the network? Convolution, dot/outer product, other? Nothing I've tried survived the first minutes of implementation...

import numpy as np
import tensorflow as tf
from tensorflow import keras

## create dummy input/output matrices
XData = np.ones((1000, 20)) ## 1000 samples, 20 features
SideData = np.ones((10, 50)) ## 10 feature filters, 50 outputs
YData = np.ones((1000, 50)) ## 1000 samples, 50 outputs

## network
model = keras.Sequential(
    [
        # keras.layers.Dense(128, activation="relu"), ## replace this layer by some magic
        keras.layers.Dense(YData.shape[1], activation="linear"),
    ]
)

model.compile(
    optimizer='adam',
    loss='mse')

history = model.fit(
    x = XData,
    y = YData,
    epochs = 10,
    validation_split = 0.3,
    verbose = 2,
    batch_size=10
  )

from Using two tables as input for multi-output regression with keras, the missing matrix product