Simple Guide to Python Multiprocessing
November 18, 2020Note
Refer Threading vs Multiprocessing vs Asyncio
Multiprocessing can utilize multiple CPUs (bypassing GIL), while Threading could only utilize 1 CPU.
For IO bound task, you might want to consider Asyncio
Multiple Process
import multiprocessing as mp
from time import sleep
def run(name):
for i in range(10):
sleep(0.5)
print(name, end='', flush=True)
print(f"{name}[END]", end='', flush=True)
for name in ['a', 'b', 'c']:
p = mp.Process(target=run, args=(name,))
p.start()
print('STOP')
STOP
acbcbacabbaccabcabbacbacbcabcc[END]b[END]aa[END]
You realize the main process stop before all the child process end.
To wait for all the child process, do the following.
jobs = []
for name in ['a', 'b', 'c']:
p = mp.Process(target=run, args=(name,))
p.start()
jobs.append(p)
# wait for all child process to end
for p in jobs:
p.join()
print('STOP')
Shared Value
Increment a counter value
import multiprocessing as mp
from time import sleep
def run(name, count):
for i in range(10):
sleep(0.5)
with count.get_lock():
count.value += 1
print(f"{name}[{count.value}]", end='', flush=True)
print(f"{name}[END]", end='', flush=True)
jobs = []
count = mp.Value('l', 0) # https://docs.python.org/3/library/array.html
for name in ['a', 'b', 'c']:
p = mp.Process(target=run, args=(name,count))
p.start()
jobs.append(p)
for p in jobs:
p.join()
print(f"STOP[{count.value}]")
a[1]c[2]b[3]c[4]a[5]b[6]c[7]a[8]b[9]c[10]a[11]b[12]c[13]a[14]b[15]c[16]a[17]b[18]c[19]a[20]b[21]c[22]a[23]b[24]c[25]b[26]a[27]c[28]c[END]b[29]b[END]a[30]a[END]STOP[30]
❤️ Is this article helpful?
Buy me a coffee☕ or support my work to keep this space 🖖 and ad-free.
If you can't, do send some 💖 to @d_luaz or help to share this article.
If you can't, do send some 💖 to @d_luaz or help to share this article.
👶 Apps I built
Pixtory App (Alpha) - easily organize photos on your phone into a blog.COVID-19 - data, chart, information & news.
暖心芽 (WIP) 🌞❤️🌱 - reminder of hope, warmth, thoughts and feelings.
Travelopy - travel discovery and journal
LuaPass - offline password manager
WhatIDoNow - a public log of things I am working on now
By Desmond Lua
- algolia
- analytics
- android
- android-ktx
- android-permission
- android-studio
- apps-script
- bash
- bootstrap
- bootstrapvue
- chartjs
- chrome
- cloud-functions
- coding-interview
- coroutines
- crashlytics
- css
- dagger2
- datastore
- datetime
- docker
- eslint
- firebase
- firebase-auth
- firebase-hosting
- firestore
- firestore-security-rules
- flask
- fontawesome
- fresco
- git
- github
- glide
- google-app-engine
- google-cloud-storage
- google-colab
- google-drive
- google-maps
- google-places
- google-play
- google-sheets
- gradle
- html
- hugo
- inkscape
- java
- java-time
- javascript
- jetson-nano
- kotlin
- layout
- lets-encrypt
- lifecycle
- linux
- logging
- lubuntu
- markdown
- mate
- material-design
- matplotlib
- md5
- mongodb
- moshi
- mplfinance
- mysql
- navigation
- nginx
- nodejs
- npm
- nuxtjs
- nvm
- pandas
- payment
- pip
- pwa
- pyenv
- python
- recylerview
- regex
- room
- rxjava
- scoped-storage
- selenium
- social-media
- ssh
- ssl
- static-site-generator
- static-website-hosting
- sublime-text
- ubuntu
- unit-test
- uwsgi
- viewmodel
- viewpager2
- virtualbox
- vue-chartjs
- vue-cli
- vue-router
- vuejs
- vuelidate
- vuepress
- web-development
- web-hosting
- webpack
- windows
- workmanager
- wsl
- yarn