Saturday 13 March 2021

Async celery tasks overhead

I am using this code to send notifications for some thousands of emails. It seems like it create some overhead in the web server (gunicorn) just after running the for loop. When sending the emails everything is fine.

users = User.query.all()

for user in users:
    send_email_celery(user.email, gettext(u'New email'), 'users/email/new_data', #plus some parameters)

So I am thinking about use BCC but then I realize I can't since each user has a unique uuid unsubscribe and the BCC has some limits in the mail clients.

So, what is the correct way to handle this type of action with bulk mails ?

@celery.task
def send_async_email_celery(msg):
    mail.send(msg)
 
def send_email_celery(to, subject, template, **kwargs):
    countdown = kwargs.get('countdown', 600)
    app = current_app._get_current_object()
    msg = Message(subject, sender=app.config['MAIL_SENDER'], recipients=[to])
    msg.html = render_template(template + '.html', **kwargs)
    send_async_email_celery.apply_async(args=[msg], countdown=countdown)

gunicorn --error-logfile gunicorn-error.log --timeout 600 --max-requests 500 --max-requests-jitter 50 --workers 5 app:app -b localhost:8080

EDIT 1: after some debugging using top, cpu is not full load. Is something else that halt temporarily gunicorn during some seconds.

EDIT 2: after the changes based on the answer of 2ps now the loop is fast, but the web server is still locked some seconds after the loop execution.

EDIT 3: Try to change rabbitmq to reddis, same problem. Ignoring results, same problem.

EDIT 4: After moving the loop of users to the @celery route, the problem still persists

EDIT 5: The issue has been solved by using gevent worker.



from Async celery tasks overhead

No comments:

Post a Comment