Introduction to Celery
Celery is a powerful, open-source distributed task queue that allows you to execute tasks asynchronously, outside of the regular request/response cycle in web applications. This is particularly useful for handling background jobs that can be time-consuming or require additional resources, such as sending emails, processing large datasets, or scheduling periodic tasks. By delegating such tasks to Celery, web applications can remain responsive to user actions while offloading intensive operations to a separate worker process.
In the context of Django, Celery integrates seamlessly to manage these background tasks. It allows you to perform asynchronous operations without blocking the user interface or slowing down server response times. Celery can scale efficiently by adding more workers, which means that as your application grows, it can handle a higher load without impacting performance. Additionally, Celery supports task scheduling, making it an ideal tool for periodic operations like scheduled notifications or database maintenance tasks.
To function properly, Celery requires a message broker (such as RabbitMQ or Redis) to handle the communication between the Django application and Celery workers. Once configured, Celery ensures that tasks are reliably queued, executed, and monitored, making it an essential component for any Django project that needs efficient background processing.
Setting Up Django with Celery
Setting up Celery with Django involves several steps to ensure that the two work seamlessly together for handling asynchronous tasks. This section will guide you through the entire process, from installing Celery to configuring it to work with Django.
Step 1: Installing Celery and Message Broker
pip install celery
This command installs Celery and its dependencies. Celery needs a message broker (e.g., Redis or RabbitMQ) to function, but we won’t focus on the installation of these brokers here.
Step 2: Configure Celery in Django
To integrate Celery with Django, you’ll need to define a Celery instance in your Django project. This configuration is typically placed in a new Python module (e.g., celery.py) within your project directory, at the same level as settings.py.
Create the celery.py file:
Inside your Django project directory (next to settings.py), create a file named celery.py and define the Celery instance:
# myproject/celery.py
from __future__ import absolute_import, unicode_literals
import os
from celery import Celery
# Set default Django settings module for 'celery'
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myproject.settings')
# Instantiate Celery object
app = Celery('myproject')
# Configure Celery using settings defined in Django settings file
app.config_from_object('django.conf:settings', namespace='CELERY')
# Automatically discover tasks.py files in installed apps
app.autodiscover_tasks()
@app.task(bind=True)
def debug_task(self):
print(f'Request: {self.request!r}')
Configure Celery in Django settings:
You’ll need to tell Django about Celery and define settings related to task execution and scheduling. Add the following to your settings.py:
# settings.py CELERY_BROKER_URL = 'redis://localhost:6379/0' # Example for Redis CELERY_ACCEPT_CONTENT = ['json'] CELERY_TASK_SERIALIZER = 'json' CELERY_RESULT_BACKEND = 'redis://localhost:6379/0' # Same as broker for simplicity CELERY_TIMEZONE = 'UTC'
This configuration sets up the broker URL, specifies JSON as the serialization format for tasks, and ensures that Celery uses Redis as both the broker and result backend. These settings can be customized depending on your message broker and needs.
Step 3: Modify __init__.py to Load Celery App
You also need to modify your project’s __init__.py file so that Celery gets loaded when Django starts. Add the following code to myproject/__init__.py:
# myproject/__init__.py from __future__ import absolute_import, unicode_literals # This will make sure the app is always imported when Django starts from .celery import app as celery_app __all__ = ('celery_app',)
This ensures that your Celery app is ready for use every time your Django project starts. Now, Django and Celery are fully integrated.
Step 4: Create Tasks
Celery tasks are Python functions that are executed in the background. In each Django app where you want to define tasks, create a file named tasks.py. Here’s a simple example:
# myapp/tasks.py
from celery import shared_task
@shared_task
def add(x, y):
return x + y
By using the @shared_task decorator, Celery ensures that the task is registered and available for asynchronous execution.
Step 5: Running Celery
Now that everything is set up, you can start Celery by running the following command in the terminal:
celery -A myproject worker --loglevel=info
This command starts the Celery worker process, which listens for tasks and executes them in the background. The -A myproject flag specifies the Django project to associate with Celery, and --loglevel=info ensures that you get detailed logs for monitoring purposes.
Step 6: Testing Your First Celery Task
To test whether everything is set up correctly, you can trigger the add task you defined earlier. Open up the Django shell by running:
python manage.py shell
Then, trigger the task:
from myapp.tasks import add add.delay(4, 6)
The delay() method schedules the task to run asynchronously. The result of the task will not be immediately available in the shell, as it will be executed by the Celery worker. You can check the worker logs to see the task being processed.
Conclusion
Once Celery is configured, you can begin using it to handle background tasks in your Django applications, improving performance and scalability. This setup allows you to offload resource-heavy tasks to background workers, making your Django application more efficient for end users. To deep dive more continue reading.
To learn more about celery you can visit the official documentation page
To explore more related to Django, visit our articles