django-celery for non-blocking Django signals.
by sethtrain
So I am working on a project now that requires sending a SMS message when a certain task is performed on a model. So my first thought was, “Okay, easy. I can just perform those tasks with signals.” Well, kinda. Signals in Django are blocking and that just wasn’t going to work. I didn’t want my users to have to wait for HTTP responses to return from sending the SMS messages. That is where Celery and django-celery came into the picture.
What is Celery?
“Celery is an asynchronous task queue/job queue based on distributed message passing. It is focused on real-time operation, but supports scheduling as well.”
Celery’s default message broker is RabbitMQ. Since I have had a little bit of experience with RabbitMQ from a Clojure project I worked on over a year ago that is what I chose. To install RabbitMQ on Ubuntu:
sudo apt-get install rabbitmq-server
You can read all about how to setup django-celery in the documentation. It is somewhat brief but along with celery’s documentation it will get you all the way there.
So what did I do to get non-blocking signals? Well I used Django’s signals to call my celery tasks defined in tasks.py in my application.
@receiver(post_save, sender=Reading)
def alert_delay(sender, **kwargs):
if kwargs["created"]:
send_alerts.delay(kwargs["instance"])
This delegates send_alerts to be called “at a later time”. Celery handles this by pickling the arguments, queuing the task in RabbitMQ and handling the execution of the tasks “later”.
As I said, I did a little bit of work with Clojure and RabbitMQ. It sure wasn’t as easy as this. I am quite impressed, impressed enough to blog about it.
Comments
Thanks for the post.
I think you can also run signals asynchronously using threading. Though Celery is more suitable for tasks outside Python (like executing system scripts).
Yeah, you are probably right. I chose this solution because I know that a couple of other features are going to need this functionality so I went ahead and did it this way. I also like the integration into Django so that I can view the status of tasks and have somewhat decent logging of activity when I need it.