Wednesday, 17 February 2021

Prevent celery task from starting until a different task with similar arguments is done

Let's say I have a celery task which takes two arguments: X(a,b)

I need to implement custom concurrency logic with the following two rules:

  1. Instances of X can run concurrently if they have different values for a. That is, if X(a=1,b=10) is running when X(a=2,b=20) is added to the queue, then the latter is pulled from the queue and executed immediately.

  2. Instances of X can NOT run concurrently if they have the same values for a. That is, if X(a=1,b=10) is running when X(a=1,b=20) is added to the queue, then the latter must wait on the queue until the former is done.

Rule #1 comes out of the box with celery by setting worker_concurrency>1 (docs). Rule #2 is the tricky one.

Distributed task locking, as described in the docs and in this blog, is an approach which gets me close to what I need. There are even libraries out there that implement it for you (celery-singleton). However, looking back at Rule #2, this approach appears to prevent the second task from being queued until the first task completes. I need it to be queued, just not executed on a worker until the first task completes.

Is there anyway to implement this? This SO question asks a similar question but no answer so far.



from Prevent celery task from starting until a different task with similar arguments is done

No comments:

Post a Comment