Limit neural network output to subset of trained classes

Is it possible to pass a vector to a trained neural network so it only chooses from a subset of the classes it was trained to recognize. For example, I have a network trained to recognize numbers and letters, but I know that the images I'm running it on next will not contain lowercase letters (Such as images of serial numbers). Then I pass it a vector telling it not to guess any lowercase letters. Since the classes are exclusive the network ends in a softmax function. Following are just examples of what I'd thought of trying but none really work.

import numpy as np

def softmax(arr):
    return np.exp(arr)/np.exp(arr).sum()

#Stand ins for previous layer/NN output and vector of allowed answers.
output = np.array([ 0.15885351,0.94527385,0.33977026,-0.27237907,0.32012873,
       0.44839673,-0.52375875,-0.99423903,-0.06391236,0.82529586])
restrictions = np.array([1,1,0,0,1,1,1,0,1,1])

#Ideas -----

'''First: Multilpy restricted before sending it through softmax.
I stupidly tried this one.'''
results = softmax(output*restrictions)

'''Second: Multiply the results of the softmax by the restrictions.'''
results = softmax(output)
results = results*restrictions

'''Third: Remove invalid entries before calculating the softmax.'''
result = output*restrictions
result[result != 0] = softmax(result[result != 0])

All of these have issues. The first one causes invalid choices to default to:

1/np.exp(arr).sum()

since inputs to softmax can be negative this can raise the probability given to an invalid choice and make the answer worse. (Should've looked into it before I tried it.)

The second and third both have similar issues in that they wait until right before an answer is given to apply the restriction. For example, if the network is looking at the letter l, but it starts to determine that it's the number 1, this won't be corrected until the very end with these methods. So if it was on it's way to giving the output of 1 with .80 probability but then this option removed it seems the remaining options will redistribute and the highest valid answer won't be as confident as 80%. The remaining options end up a lot more homogeneous. An example of what I'm trying to say:

output
Out[75]: array([ 5.39413513,  3.81445419,  3.75369546,  1.02716988,  0.39189373])

softmax(output)
Out[76]: array([ 0.70454877,  0.14516581,  0.13660832,  0.00894051,  0.00473658])

softmax(output[1:])
Out[77]: array([ 0.49133596,  0.46237183,  0.03026052,  0.01603169])

(Arrays were ordered to make it easier.) In the original output the softmax gives .70 that the answer is [1,0,0,0,0] but if that's an invalid answer and thus removed the redistribution how assigns the 4 remaining options with under 50% probability which could easily be ignored as too low to use.

I've considered passing a vector into the network earlier as another input but I'm not sure how to do this without requiring it to learn what the vector is telling it to do, which I think would increase time required to train.

Thanks for any help, I feel like I'm stuck in a mindset of thinking of the same stupid answers. Sorry for any formatting issues I tied to make it look right.

from Limit neural network output to subset of trained classes