Wednesday, 5 May 2021

Celery - Check if worker received SIGTERM

I have a Celery task that is quite long. More than a few minutes.

Sometimes, for various reasons, a worker is marked for termination and another worker starts. This can happen if the machine it runs on needs to be replaced, or a new code version is being deployed. In that case, the worker receives the SIGTERM signal.

I'm wondering if it's possible for the task itself to periodically check whether this worker has received SIGTERM and is pending termination, and in that case just put the task back in the queue and terminate. (The task will then be started on another worker, and will continue doing its work)

EDIT: Clarifying - Is it possible within the task to check whether it is executed on a worker that waits to be terminated. Like this:

# Some long task that can take even a few hours.
def some_task(...):
    for i in range(...):
        do_some_work()
        # That's the missing function:
        if did_this_worker_received_SIGTERM_and_waiting_to_be_terminated():
             # stop the task in the middle, and it will be executed again later


from Celery - Check if worker received SIGTERM

No comments:

Post a Comment