Cloud Storage Write Streaming Data (Upload Data by Chunks)

June 7, 2019

Solution 1: gcsfs

gcsfs is a 3rd party library (not by Google)

import gcsfs
fs = gcsfs.GCSFileSystem(token='keys/travelopyz-appengine.json')

BUCKET_NAME = '...'
filename = f"{BUCKET_NAME}/DIRECTORY_NAME/test_file.txt"

# with gcsfs.core.GCSFile(fs, f'travelopyz.appspot.com/generate/get_items_test/test_file4', 'wb', acl='publicread', consistency='none', block_size=5242880) as f:
with fs.open(filename, 'wb', acl='publicread') as f:
    f.write('Item 1'.encode())
    f.write('Item 2'.encode())

NOTE: acl - publicread doesn’t seems to work, checking the source code would note that the acl args is not used. I tried using GCSFile and acl doesn’t seems to work as well.

Solution 2: google-resumable-media

I have yet to try this, but following are some relevant resources.

Solution 3: google-cloud-storage

There is a request issue raised.

The source code has some hint/potential for resumable multipart upload.

This work is licensed under a
Creative Commons Attribution-NonCommercial 4.0 International License.