Queue (Push Task) Cloud Functions With Google Cloud Tasks

Overview

Cloud Tasks currently have 2 main components

Cloud Tasks queues with App Engine targets: I believe this is the replacement for App Engine Python 2.7 Task Queue push queues, and maybe Google Cloud Pub/Sub is for pull queues. Refer to Migrating from Task Queues to Cloud Tasks.
Cloud Tasks queues with HTTP targets (Beta): forwards the task request to the worker, located at any generic HTTP endpoint, such as Cloud Functions, Cloud Run, GKE, Compute Engine, or even an on-prem web server, and based on how the task is configured.

This article shall focus on HTTP Target Tasks for cloud functions.

Setup

To access the Cloud Tasks service using Cloud Tasks API , you must have a project that contains an App Engine app located in a specific region. This location will serve as the LOCATION_ID parameter for your Cloud Tasks requests, so make a note of it. Note that two locations, called europe-west and us-central in App Engine commands, are called, respectively, europe-west1 and us-central1 in Cloud Tasks commands. The app serves as the location for whatever queues the developer creates. The underlying Cloud Tasks service itself runs in that same location. - Source.

Create Task Queue

Refer Creating Cloud Tasks Queues.

gcloud tasks queues [QUEUE_ID] --project travelopyz

NOTE: Queue ID can contain only letters (A-Za-z), numbers (0-9), or hyphens (-). Queue ID must be between 1 and 100 characters.

NOTE: Sadly there is no Web UI for task queue creation at the moment

You can check Google Cloud Console -> Cloud Tasks for a list of task queue. NOTE: Previously task queue created via App Engine Python 2.7 queue.yaml is listed here as well.

or check via the following command

gcloud tasks queues describe [QUEUE_ID]

name: projects/PROJECT_ID/locations/us-central1/queues/QUEUE_ID
rateLimits:
  maxBurstSize: 100
  maxConcurrentDispatches: 1000
  maxDispatchesPerSecond: 500.0
retryConfig:
  maxAttempts: 100
  maxBackoff: 3600s
  maxDoublings: 16
  minBackoff: 0.100s
state: RUNNING

I wanted a very slow queue where it only process 1 request per seconds, and only 1 concurrent task

gcloud tasks queues update [QUEUE_ID] --max-dispatches-per-second=1 --max-concurrent-dispatches=1 --project travelopyz

I set it to only retry 3 times upon error, and to wait at least 5s between retry.

gcloud tasks queues update [QUEUE_ID] --max-attempts=3 --min-backoff=5s --project travelopyz

Check queue again to make sure all configuartion is correct.

gcloud tasks queues describe [QUEUE_ID]

name: projects/PROJECT_ID/locations/us-central1/queues/QUEUE_ID
rateLimits:
  maxBurstSize: 10
  maxConcurrentDispatches: 1
  maxDispatchesPerSecond: 1.0
retryConfig:
  maxAttempts: 3
  maxBackoff: 3600s
  maxDoublings: 16
  minBackoff: 5s
state: RUNNING

Code

pip install google-cloud-tasks

Edit requirements.txt.

# https://pypi.org/project/google-cloud-tasks/
google-cloud-tasks==1.1.0

from google.cloud import tasks_v2beta3LOCATION = 'us-central1'PROJECT_ID = '...'QUEUE_ID = '...'client = tasks_v2beta3.CloudTasksClient()parent = client.queue_path(PROJECT_ID, location=LOCATION, queue=QUEUE_ID)task = {    'http_request': {  # Specify the type of request.        'http_method': 'POST',        'url': f"https://{LOCATION}-{PROJECT_ID}.cloudfunctions.net/test_payload"    }}response = client.create_task(parent, task)

NOTE: Refer to Google Tasks With Payload Parameters (Query String, POST and JSON).

Secure with Authentication

We want to secure the cloud functions to make sure it could be triggered by authorized party only.

SERVICE_ACCOUNT_EMAIL = 'NAME@PROJECT_ID.iam.gserviceaccount.com'task = {    'http_request': {  # Specify the type of request.        'http_method': 'POST',        'url': f"https://{LOCATION}-{PROJECT_ID}.cloudfunctions.net/test_payload",        'oidc_token': {            'service_account_email': SERVICE_ACCOUNT_EMAIL        },    }}

NOTE: Service account: e.g. [email protected] or NAME@PROJECT_ID.iam.gserviceaccount.com. Pick an existing email from Google Cloud Console -> Service account or create a new service account for authentication purpose (Roles, Grant users access and Create key is not required).

NOTE: Refer to Secure Cloud Functions With OIDC Account (from Cloud Scheduler).

References: