App Engine Standard Python3.7 Correlate Application Log With Request Log

NOTE: Too bad I have yet to found a solution, but this is my research so far.

When you use Google App Engine Standard Python 3.7, one of the many differences from Python 2.7 is logging.

Request logs are no longer automatically correlated but will still appear in Stackdriver Logging. Use the Stackdriver Logging client libraries to implement your desired logging behavior.

As per Writing application logs.

When your application handles a request, it can write its own logging messages to stdout and stderr. Write your application logs using stdout for output and stderr for errors. These files are automatically collected and can be viewed in the Logs Viewer. Note that this does not provide log levels that you can use for filtering in the Logs Viewer; however, the Logs Viewer does provide other filtering, such as text, timestamp, etc. Only the most recent entries in the Logs Viewer are retained in order to limit their size.

NOTE: seems like log level doesn't work by default as well.

When you are using the old Python 2.7 app engine standard, all logs are grouped by request. You can expand on a request log (in Log Viewer) and view all the corresponding request log. In Python 3.7 app engine standard, application logs still show up in Log Viewer but they are not group together/correlate with request logs (so it is very hard to tell the application logs belong to which HTTP request).

According to Writing Application Logs for Python 3.7 and Linking app logs and requests mentioned a few things needed to be done to produce correlated logs.

In the Logs Viewer, log entries correlated by the same trace can be viewed in a "parent-child" format.

In your application code, look for the X-Cloud-Trace-Context HTTP header of incoming requests. Extract the trace identifier from the header.

Set the trace identifier in the LogEntry trace field of your app log entries. The expected format is projects/PROJECT_ID/traces/TRACE_ID

Set the httpRequest.requestUrl field for the top-level log.

I found an article which some sample code, but sadly google-cloud-logging / Stackdriver logging v1.6 have an issue to log trace.

I have experimented with the following code but fail to achive correlated application and request logs.

import loggingfrom google.cloud import logging as gcloud_loggingfrom flask import Flask, requestapp = Flask(__name__)app.logger.setLevel(logging.INFO)logging.basicConfig(level=logging.INFO)# https://cloud.google.com/logging/docs/setup/pythonlogging_client = gcloud_logging.Client()# logging_client.setup_logging()@app.route('/test_logging')def test_logging():    app.logger.info('app.logger')    logging.info('logging')    print('print')    # https://cloud.google.com/logging/docs/setup/python    # https://googlecloudplatform.github.io/google-cloud-python/latest/logging/usage.html    logger = logging_client.logger('parent')    # this not showing    logger.log_text('parent logger')    # trace is invalid paremeter    TRACE = f"projects/{logging_client.project}/traces/{request.headers.get('X-Cloud-Trace-Context')}"    # logger.log_text(f'trace: {TRACE}', trace=TRACE)    from google.cloud.logging.handlers import AppEngineHandler, setup_logging    # handler = logging_client.get_default_handler()    handler = AppEngineHandler(logging_client)    cloud_logger = logging.getLogger('cloudLogger')    cloud_logger.setLevel(logging.INFO)    # setup_logging(handler)    cloud_logger.addHandler(handler)    cloud_logger.info('cloud logger')    # ERROR:google.cloud.logging.handlers.transports.background_thread:Failed to submit 1 logs.    return 'Test Logging'

PS: I Send Feedback on Writing Application Logs documentation page requesting the documentation team to include a sample code. I you are facing the same problem, do send them a feedback.

References: