Setup Delayed Background Task Using Google App Engine Deferred

January 16, 2018

Task Queue is usually used to execute background task, where you can even setup Cron Jobs.

But if you have a small task which you intent to run in background after each request, do consider using deferred. Sample use cases are:

  • Send email or sms after each request (avoid blocking the request)
  • Post for FB/Twitter after 5 minutes delay (allow user window to make edits)

The following code will execute defer_post in background after the request complete.

import logging
from google.appengine.ext import ndb, deferred

def defer_post(item_key=None):
    try:
        item = item_key.get()
        # do something
    except Exception as e:
        # this to avoid 
        logging.info('item_key: %s', item_key.urlsafe())
        logging.exception(e)
        raise deferred.PermanentTaskFailure()

@app.route('/test_refer/<item_key>')
def test_defer(item_key):
    item_key = ndb.Key(urlsafe=item_key)
    deferred.defer(defer_post, item_key=item_key)
    return "defer_post queued"

NOTE: implementing deferred.PermanentTaskFailure is optional, where it will keep on retrying after specific intervals.

NOTE: deferred function must be a module function, not inner function or anonymous function.

You can delay the execution using _countdown parameter.

# wait 60s then run in background
deferred.defer(defer_post, item_key=item_key, _countdown=60)

If you setup a Push Queue to specify the processing rate, you can specify the queue name.

NOTE: Queue name must match pattern ^[a-zA-Z0-9-]{1,100}$

# wait 60s then run in background
deferred.defer(defer_post, item_key=item_key, _countdown=60, _queue="my-defer-queue")

There are a few gotchas using deferred, mainly:

  • Don’t pass entities as parameter (the entity might be stale by the time background task execute), use entity key.
  • Avoid permanent repeated failure by raising deferred.PermanentTaskFailure.
  • If you are using modules imported using sys.path.append, make sure you include them in the current module.
  • All argument must be pickable.
  • Don’t pass in nested function, methods of nested classes, lambda function, static method. Deferred function must be a module function, not inner function or anonymous function.
  • Use this for small task only.

References:

This work is licensed under a
Creative Commons Attribution-NonCommercial 4.0 International License.