Simple Guide to Python Multiprocessing

Nov 18, 2020

Note

Refer Threading vs Multiprocessing vs Asyncio

Multiprocessing can utilize multiple CPUs (bypassing GIL), while Threading could only utilize 1 CPU.

For IO bound task, you might want to consider Asyncio

Multiple Process

import multiprocessing as mpfrom time import sleepdef run(name):    for i in range(10):        sleep(0.5)        print(name, end='', flush=True)    print(f"{name}[END]", end='', flush=True)for name in ['a', 'b', 'c']:    p = mp.Process(target=run, args=(name,))    p.start()  print('STOP')
STOP
acbcbacabbaccabcabbacbacbcabcc[END]b[END]aa[END]

You realize the main process stop before all the child process end.

To wait for all the child process, do the following.

jobs = []for name in ['a', 'b', 'c']:    p = mp.Process(target=run, args=(name,))    p.start()    jobs.append(p)# wait for all child process to endfor p in jobs:    p.join()print('STOP')

Shared Value

Increment a counter value

import multiprocessing as mpfrom time import sleepdef run(name, count):    for i in range(10):        sleep(0.5)        with count.get_lock():            count.value += 1            print(f"{name}[{count.value}]", end='', flush=True)    print(f"{name}[END]", end='', flush=True)jobs = []count = mp.Value('l', 0) # https://docs.python.org/3/library/array.htmlfor name in ['a', 'b', 'c']:    p = mp.Process(target=run, args=(name,count))    p.start()    jobs.append(p)for p in jobs:    p.join()print(f"STOP[{count.value}]")
a[1]c[2]b[3]c[4]a[5]b[6]c[7]a[8]b[9]c[10]a[11]b[12]c[13]a[14]b[15]c[16]a[17]b[18]c[19]a[20]b[21]c[22]a[23]b[24]c[25]b[26]a[27]c[28]c[END]b[29]b[END]a[30]a[END]STOP[30]

❤️ Is this article helpful?

Buy me a coffee ☕ or support my work via PayPal to keep this space 🖖 and ad-free.

Do send some 💖 to @d_luaz or share this article.

✨ By Desmond Lua

A dream boy who enjoys making apps, travelling and making youtube videos. Follow me on @d_luaz

👶 Apps I built

Travelopy - discover travel places in Malaysia, Singapore, Taiwan, Japan.