Monday, 21 December 2020

Getting "missing 2 required positional arguments" due to custom scorer that I pass to cross_val_score

So the main idea is to create a custom scorer, to measure Specificity, because it's not part of the predifined scoring methods of scikit learn.

The method seems to work ok, however, when I try to pass the results from cross_val_score to a dataframe that I later plot using seaborn boxplot, I get the following error.

TypeError: __call__() missing 2 required positional arguments: 'X' and 'y_true'

The process that calls the cross_val_score

        mbox = pd.DataFrame()
        for metric in metrics:
            if metric == 'specificity':
                metric = make_scorer(custom_specificity)
            results = cross_val_score(model, x_test_m, y_test_m, cv=cv_num, scoring=metric)
            print('results', type(results))
            if show_raw_data:
                raw = np.array2string(results, threshold=np.inf, max_line_width=np.inf, separator=',')
                raw.replace('\n', '').replace(' ', '')
                print(metric, ": %.3f%% Std.: %.3f%%" % (results.mean() * 100.0, results.std() * 100.0))
                print("Metric raw data: ", raw)
            mbox[metric] = results

For the rest of the predifined scorers I get the following results.

results <class 'numpy.ndarray'>
accuracy : 87.361% Std.: 4.934%
Metric raw data:  [0.8125    ,0.875     ,0.93333333]

For the custom scorer I get the following results.

results <class 'numpy.ndarray'>
make_scorer(custom_specificity) : 83.333% Std.: 11.785%
Metric raw data:  [0.75,0.75,1.  ]

The custom scorer function

def custom_specificity(y_true, y_pred, **kwargs):
    tp, fp, tn, fn = custom_measure(y_true, y_pred)
    if tn + fp == 0:
        return 1
    else:
        return tn / (tn + fp)

The custom_measure function

def custom_measure(y_actual, y_hat):
    tp, fp, tn, fn = 0, 0, 0, 0
    cm = confusion_matrix(y_actual, y_hat)
    tn = cm[0][0]
    fn = cm[1][0]
    tp = cm[1][1]
    fp = cm[0][1]
    return tp, fp, tn, fn

The stack trace

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-7-0ebc81311532> in <module>
      2 trained_models_using_pca_comp = train.train_models(pca_train_x, pca_train_y, pca_test_x, pca_test_y)
      3 evaluate.generate_metrics(trained_models_using_pca_comp, pca_test_x, pca_test_y, 
----> 4                           ['accuracy', 'precision', 'recall', 'f1', 'specificity'], cv_num=cross_val_num, show_raw_data=True)

~\thesisProject\thesis\moduleMetrics.py in generate_metrics(models, x_test, y_test, metrics, cv_num, show_raw_data)
     74                     print(metric, ": %.3f%% Std.: %.3f%%" % (results.mean() * 100.0, results.std() * 100.0))
     75                     print("Metric raw data: ", raw)
---> 76                 mbox[metric] = results
     77 
     78             # plot confusion matrix

~\anaconda3\lib\site-packages\pandas\core\frame.py in __setitem__(self, key, value)
   3023 
   3024     def __setitem__(self, key, value):
-> 3025         key = com.apply_if_callable(key, self)
   3026 
   3027         # see if we can slice the rows

~\anaconda3\lib\site-packages\pandas\core\common.py in apply_if_callable(maybe_callable, obj, **kwargs)
    339     """
    340     if callable(maybe_callable):
--> 341         return maybe_callable(obj, **kwargs)
    342 
    343     return maybe_callable

TypeError: __call__() missing 2 required positional arguments: 'X' and 'y_true'


from Getting "missing 2 required positional arguments" due to custom scorer that I pass to cross_val_score

No comments:

Post a Comment