Part 5 - Using Celery to Automate Maintenance Chores

Scenario

In Part 1 you created your own Authorization Server and it’s running along just fine. However, the database is getting cluttered with expired tokens. You can periodically run the cleartokens management command, but why not automate this with Celery?

Set up RabbitMQ

Celery components communicate via a message queue. We’ll use RabbitMQ.

Install RabbitMQ on MacOS

If you are using MacOS it’s likely you are already using Homebrew. If not, now’s the time to install this fantastic package manager.

brew install rabbitmq
brew service start rabbitmq

Install RabbitMQ with Docker

This will start up a docker image that just works:

docker run -it --rm --name rabbitmq -p 5672:5672 -p 15672:15672 rabbitmq:3.9-management

Install RabbitMQ on Windows

See the RabbitMQ Installing on Windows instructions.

Add Celery

Make sure you virtualenv is active and install celery and django-celery-beat.

pip install celery django-celery-beat

Update your list of installed apps to include both your Authorization Server app – we’ll call it tutorial, and django_celery_beat which extends your Django project to store your periodic task schedule in the database and adds a Django Admin interface for configuring them.

INSTALLED_APPS = {
    # ...
    "tutorial",
    "django_celery_beat",
}

Now add a new file to your app to add Celery: tutorial/celery.py:

import os

from celery import Celery

# Set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'tutorial.settings')
app = Celery('tutorial', broker="pyamqp://guest@localhost//")
app.config_from_object('django.conf:settings', namespace='CELERY')

# Load task modules from all registered Django apps.
app.autodiscover_tasks()

This will autodiscover any tasks.py files in the list of installed apps. We’ll add ours now in tutorial/tasks.py:

from celery import shared_task

@shared_task
def clear_tokens():
    from oauth2_provider.models import clear_expired

    clear_expired()

Finally, update tutorial/__init__.py to make sure Celery gets loaded when the app starts up:

from .celery import app as celery_app

__all__ = ('celery_app',)

Run Celery Beat and the Worker

RabbitMQ should already be running; it’s the “glue” between Beat and the Worker.

It’s best to run each of these in its own terminal window so you can see the log messages.

Start Celery Beat

celery -A tutorial beat -l INFO  --scheduler django_celery_beat.schedulers:DatabaseScheduler

Start Celery Worker

celery -A tutorial worker -l INFO

Configure the clear_tokens task

Go into Django Admin and you’ll see a new section for periodic tasks:

Django Admin interface screenshot

Now let’s define a fairly short (10 second) interval. Go to: http://127.0.0.1:8000/admin/django_celery_beat/intervalschedule/ and select Add Interval, set number of intervals to 10 and interval period to seconds and Save.

Then go to http://127.0.0.1:8000/admin/django_celery_beat/periodictask/ to add a new periodic task by selecting Add Periodic Task and select tutorial.tasks.clear_tokens, choose the every 10 seconds interval schedule, and “Save.”

Django Admin interface screenshot

Now your Celery Beat and Celery Workers should start running the task every 10 seconds.

The Beat console will look like this:

[2022-03-19 22:06:35,605: INFO/MainProcess] Scheduler: Sending due task clear stale tokens (tutorial.tasks.clear_tokens)

And the Workers console like this:

[2022-03-19 22:06:35,614: INFO/MainProcess] Task tutorial.tasks.clear_tokens[5ec25fb8-5ce3-4d15-b9ad-750b80fc07e0] received
[2022-03-19 22:06:35,616: INFO/ForkPoolWorker-8] refresh_expire_at is None. No refresh tokens deleted.
[2022-03-19 22:06:35,629: INFO/ForkPoolWorker-8] 0 Expired access tokens deleted
[2022-03-19 22:06:35,631: INFO/ForkPoolWorker-8] 0 Expired grant tokens deleted
[2022-03-19 22:06:35,632: INFO/ForkPoolWorker-8] Task tutorial.tasks.clear_tokens[5ec25fb8-5ce3-4d15-b9ad-750b80fc07e0] succeeded in 0.016124433999999965s: None