Python Firestore Multiprocessing Core Dump

NOTE: I thought that I solved firestore using multiprocessing, but more testing reveal a core dump issue.

Environment

Ubuntu 20.04.1 LTS

Python 3.8.6

firebase-admin==4.4.0
google-cloud-core==1.4.3
google-cloud-firestore==2.0.1
grpcio==1.33.2

Replicate Core Dump

main will listen to snapshot.
main will launch a new process: run_process.
run_process: will listen to snapshot, perform get and update in a loop.

import multiprocessing as mp# import concurrent.futures as cfimport firebase_adminfrom firebase_admin import credentials, firestore, authdef run_process(cret_file):    cred = credentials.Certificate(cret_file)    app = firebase_admin.initialize_app(cred, name='new-process')    db = firestore.client(app=app)    def on_snapshot(doc_snapshot, changes, read_time):        print('user changes in process')    watch = db.collection('user').on_snapshot(on_snapshot)    # asyncio.run(run_async(app))    test_ref = db.collection('test').document('test01')    while True:        try:            doc = test_ref.get()            print('count', doc.get('count'))            test_ref.update({                'count': doc.get('count') + 1            })        except Exception as e:            print(e)        # raise Exception('TEST')        print('p', end='', flush=True)        time.sleep(2)    watch.unsubscribe()    print('process END')    return 'OK'def main():    cert_file = 'firebase-adminsdk.json'    cred = credentials.Certificate(cert_file)    app = firebase_admin.initialize_app(cred)    db = firestore.client(app=app)    def on_snapshot(doc_snapshot, changes, read_time):        print('user changes in main')    watch = db.collection('user').on_snapshot(on_snapshot)    p = mp.Process(target=run_process, args=(cert_file,))    p.start()    p.join()    print('process exit code', p.exitcode)    '''    with cf.ProcessPoolExecutor() as executor:        f = executor.submit(run_process, cret_file)        try:            print(f.result())        except Exception as e:            print(e)            print(f.exception())    '''        watch.unsubscribe()    print('main END')if __name__ == '__main__':    main()

If you run the code long enough (between 2 - 20 loops), you will either bump into

Core dump in the main process
or Process exit with exitcode(-11): which is SIGSEGV 11 Terminate (core dump) Invalid memory reference

Solution

Sadly, I don't have a real solution.

Alternative workaround to wrap the new process code in a script, and use subprocess to launch the script.

Another solution is to use Firestore in a single process only, where other processes send signal to firestore process to read or write data.