Simple Guide to Python Multiprocessing

November 18, 2020

python

Note

Refer Threading vs Multiprocessing vs Asyncio

Multiprocessing can utilize multiple CPUs (bypassing GIL), while Threading could only utilize 1 CPU.

For IO bound task, you might want to consider Asyncio

Multiple Process

import multiprocessing as mp
from time import sleep

def run(name):
    for i in range(10):
        sleep(0.5)
        print(name, end='', flush=True)
    print(f"{name}[END]", end='', flush=True)

for name in ['a', 'b', 'c']:
    p = mp.Process(target=run, args=(name,))
    p.start()
  
print('STOP')

STOP
acbcbacabbaccabcabbacbacbcabcc[END]b[END]aa[END]

You realize the main process stop before all the child process end.

To wait for all the child process, do the following.

jobs = []
for name in ['a', 'b', 'c']:
    p = mp.Process(target=run, args=(name,))
    p.start()
    jobs.append(p)

# wait for all child process to end
for p in jobs:
    p.join()

print('STOP')

Shared Value

Increment a counter value

import multiprocessing as mp
from time import sleep

def run(name, count):
    for i in range(10):
        sleep(0.5)
        with count.get_lock():
            count.value += 1
            print(f"{name}[{count.value}]", end='', flush=True)
    print(f"{name}[END]", end='', flush=True)

jobs = []
count = mp.Value('l', 0) # https://docs.python.org/3/library/array.html
for name in ['a', 'b', 'c']:
    p = mp.Process(target=run, args=(name,count))
    p.start()
    jobs.append(p)

for p in jobs:
    p.join()

print(f"STOP[{count.value}]")

a[1]c[2]b[3]c[4]a[5]b[6]c[7]a[8]b[9]c[10]a[11]b[12]c[13]a[14]b[15]c[16]a[17]b[18]c[19]a[20]b[21]c[22]a[23]b[24]c[25]b[26]a[27]c[28]c[END]b[29]b[END]a[30]a[END]STOP[30]

❤️ Is this article helpful?

Buy me a coffee☕ or support my work to keep this space 🖖 and ad-free.
If you can't, do send some 💖 to @d_luaz or help to share this article.

👶 Apps I built

Travelopy - discover travel places in Malaysia, Singapore, Taiwan, Japan.
Pixtory App (Alpha) - easily organize photos on your phone into a blog.
暖心芽 (WIP) 🌞❤️🌱 - reminder of hope, warmth, thoughts and feelings (or just quotes).
LuaPass - offline password manager

By Desmond Lua

A dream boy who enjoys making apps, travelling and gardening. Follow me on @d_luaz.

Tags:

python

Related entries:

Python Encryption Using Tink

Python Only Allow Single Instance to Run (or Kill Previous Instance)

Python Asyncio Graceful Shutdown (Interrupt Sleep)

Simple Guide to Subprocess (launch another python script)

Simple Guide to Python Asyncio