So the main idea is to create a custom scorer, to measure Specificity, because it's not part of the predifined scoring methods of scikit learn.
The method seems to work ok, however, when I try to pass the results from cross_val_score to a dataframe that I later plot using seaborn boxplot, I get the following error.
TypeError: __call__() missing 2 required positional arguments: 'X' and 'y_true'
The process that calls the cross_val_score
mbox = pd.DataFrame()
for metric in metrics:
if metric == 'specificity':
metric = make_scorer(custom_specificity)
results = cross_val_score(model, x_test_m, y_test_m, cv=cv_num, scoring=metric)
print('results', type(results))
if show_raw_data:
raw = np.array2string(results, threshold=np.inf, max_line_width=np.inf, separator=',')
raw.replace('\n', '').replace(' ', '')
print(metric, ": %.3f%% Std.: %.3f%%" % (results.mean() * 100.0, results.std() * 100.0))
print("Metric raw data: ", raw)
mbox[metric] = results
For the rest of the predifined scorers I get the following results.
results <class 'numpy.ndarray'>
accuracy : 87.361% Std.: 4.934%
Metric raw data: [0.8125 ,0.875 ,0.93333333]
For the custom scorer I get the following results.
results <class 'numpy.ndarray'>
make_scorer(custom_specificity) : 83.333% Std.: 11.785%
Metric raw data: [0.75,0.75,1. ]
The custom scorer function
def custom_specificity(y_true, y_pred, **kwargs):
tp, fp, tn, fn = custom_measure(y_true, y_pred)
if tn + fp == 0:
return 1
else:
return tn / (tn + fp)
The custom_measure function
def custom_measure(y_actual, y_hat):
tp, fp, tn, fn = 0, 0, 0, 0
cm = confusion_matrix(y_actual, y_hat)
tn = cm[0][0]
fn = cm[1][0]
tp = cm[1][1]
fp = cm[0][1]
return tp, fp, tn, fn
The stack trace
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-7-0ebc81311532> in <module>
2 trained_models_using_pca_comp = train.train_models(pca_train_x, pca_train_y, pca_test_x, pca_test_y)
3 evaluate.generate_metrics(trained_models_using_pca_comp, pca_test_x, pca_test_y,
----> 4 ['accuracy', 'precision', 'recall', 'f1', 'specificity'], cv_num=cross_val_num, show_raw_data=True)
~\thesisProject\thesis\moduleMetrics.py in generate_metrics(models, x_test, y_test, metrics, cv_num, show_raw_data)
74 print(metric, ": %.3f%% Std.: %.3f%%" % (results.mean() * 100.0, results.std() * 100.0))
75 print("Metric raw data: ", raw)
---> 76 mbox[metric] = results
77
78 # plot confusion matrix
~\anaconda3\lib\site-packages\pandas\core\frame.py in __setitem__(self, key, value)
3023
3024 def __setitem__(self, key, value):
-> 3025 key = com.apply_if_callable(key, self)
3026
3027 # see if we can slice the rows
~\anaconda3\lib\site-packages\pandas\core\common.py in apply_if_callable(maybe_callable, obj, **kwargs)
339 """
340 if callable(maybe_callable):
--> 341 return maybe_callable(obj, **kwargs)
342
343 return maybe_callable
TypeError: __call__() missing 2 required positional arguments: 'X' and 'y_true'
from Getting "missing 2 required positional arguments" due to custom scorer that I pass to cross_val_score
No comments:
Post a Comment