Let's say I have a celery task which takes two arguments: X(a,b)
I need to implement custom concurrency logic with the following two rules:
-
Instances of
Xcan run concurrently if they have different values fora. That is, ifX(a=1,b=10)is running whenX(a=2,b=20)is added to the queue, then the latter is pulled from the queue and executed immediately. -
Instances of
Xcan NOT run concurrently if they have the same values fora. That is, ifX(a=1,b=10)is running whenX(a=1,b=20)is added to the queue, then the latter must wait on the queue until the former is done.
Rule #1 comes out of the box with celery by setting worker_concurrency>1 (docs). Rule #2 is the tricky one.
Distributed task locking, as described in the docs and in this blog, is an approach which gets me close to what I need. There are even libraries out there that implement it for you (celery-singleton). However, looking back at Rule #2, this approach appears to prevent the second task from being queued until the first task completes. I need it to be queued, just not executed on a worker until the first task completes.
Is there anyway to implement this? This SO question asks a similar question but no answer so far.
from Prevent celery task from starting until a different task with similar arguments is done
No comments:
Post a Comment