I have a very large dataset (7 million rows, 54 features) that I would like to fit a regression model to using XGBoost
. To train the best possible model, I want to use BayesSearchCV
from scikit-optimize
to run the fit repeatedly for different hyperparameter combinations until the best performing set is found.
For a given set of hyperparameters, XGBoost
takes a very long time to train a model, so in order to find the best hyperparameters without spending days on every permutation of training folds, hyperparameters, etc., I want to multithread both XGBoost
and BayesSearchCV
. The relevant part of my code looks like this:
xgb_pipe = Pipeline([('clf', XGBRegressor(random_state = 42, objective='reg:squarederror', n_jobs = 1))])
xgb_fit_params = {'clf__early_stopping_rounds': 5, 'clf__eval_metric': 'mae', 'clf__eval_set': [[X_val.values, y_val.values]]}
xgb_kfold = KFold(n_splits = 5, random_state = 42)
xgb_unsm_cv = BayesSearchCV(xgb_pipe, xgb_params, cv = xgb_kfold, n_jobs = 2, n_points = 1, n_iter = 15, random_state = 42, verbose = 4, scoring = 'neg_mean_absolute_error', fit_params = xgb_fit_params)
xgb_unsm_cv.fit(X_train.values, y_train.values)
However, I've found that when n_jobs > 1
in the BayesSearchCV
call, the fit crashes and I get the following error:
TerminatedWorkerError: A worker process managed by the executor was unexpectedly terminated. This could be caused by a segmentation fault while calling the function or by an excessive memory usage causing the Operating System to kill the worker.
The exit codes of the workers are {SIGKILL(-9)}
This error persists whenever I use more than 1 thread in the BayesSearchCV
call, and is independent of the memory I provide.
Is this some fundamental incompatibility between XGBoost
and scikit-optimize
, or can both packages be forced to work together somehow? Without some way of multithreading the optimization, I fear that fitting my model will take weeks to perform. What can I do to fix this?
from XGBoost and scikit-optimize: BayesSearchCV and XGBRegressor are incompatible - why?
No comments:
Post a Comment