1. Tutorials
  2. Python

Simple Guide to Python Multiprocessing

November 18, 2020

Note

Refer Threading vs Multiprocessing vs Asyncio

Multiprocessing can utilize multiple CPUs (bypassing GIL), while Threading could only utilize 1 CPU.

For IO bound task, you might want to consider Asyncio

Multiple Process

import multiprocessing as mp
from time import sleep

def run(name):
    for i in range(10):
        sleep(0.5)
        print(name, end='', flush=True)
    print(f"{name}[END]", end='', flush=True)

for name in ['a', 'b', 'c']:
    p = mp.Process(target=run, args=(name,))
    p.start()
  
print('STOP')

STOP
acbcbacabbaccabcabbacbacbcabcc[END]b[END]aa[END]

You realize the main process stop before all the child process end.

To wait for all the child process, do the following.

jobs = []
for name in ['a', 'b', 'c']:
    p = mp.Process(target=run, args=(name,))
    p.start()
    jobs.append(p)

# wait for all child process to end
for p in jobs:
    p.join()

print('STOP')

Shared Value

Increment a counter value

import multiprocessing as mp
from time import sleep

def run(name, count):
    for i in range(10):
        sleep(0.5)
        with count.get_lock():
            count.value += 1
            print(f"{name}[{count.value}]", end='', flush=True)
    print(f"{name}[END]", end='', flush=True)

jobs = []
count = mp.Value('l', 0) # https://docs.python.org/3/library/array.html
for name in ['a', 'b', 'c']:
    p = mp.Process(target=run, args=(name,count))
    p.start()
    jobs.append(p)

for p in jobs:
    p.join()

print(f"STOP[{count.value}]")

a[1]c[2]b[3]c[4]a[5]b[6]c[7]a[8]b[9]c[10]a[11]b[12]c[13]a[14]b[15]c[16]a[17]b[18]c[19]a[20]b[21]c[22]a[23]b[24]c[25]b[26]a[27]c[28]c[END]b[29]b[END]a[30]a[END]STOP[30]
❤️ Is this article helpful?
Buy me a coffee☕ or support my work to keep this space 🖖 and ad-free.
If you can't, do send some 💖 to @d_luaz or help to share this article.
👶 Apps I built
Pixtory App (Alpha) - easily organize photos on your phone into a blog.
COVID-19 - data, chart, information & news.
暖心芽 (WIP) 🌞❤️🌱 - reminder of hope, warmth, thoughts and feelings.
Travelopy - travel discovery and journal
LuaPass - offline password manager
WhatIDoNow - a public log of things I am working on now
By Desmond Lua
A dream boy who enjoys making apps, travelling and gardening. Follow me on @d_luaz.
This work is licensed under a
Creative Commons Attribution-NonCommercial 4.0 International License.