Thursday 8 July 2021

Reload Dask worker containers automatically on code change

I have the Dask code below that submits N workers, where each worker is implemented in a Docker container:

default_sums = client.map(process_asset_defaults, build_worker_args(req, numWorkers))
future_total_sum = client.submit(sum, default_sums)
total_defaults_sum = future_total_sum.result()

where process_asset_defaults is a method in a worker.

The problem is that in a development environment when I change the worker's code I need to restart all the containers manually for the change to take effect.

Is there a way to reload the worker with the new code without restarting the workers?

Note: the code resides in a Docker volume, I change it directly in the volume with Visual Studio Code.



from Reload Dask worker containers automatically on code change

No comments:

Post a Comment