Contents

Utilizing multiple CPUs in Julia

When you don’t want other core sitting idle.

Source

A Youtube video about parallelism

Before going multi-core, improve the serial performance first!

High level packages

Recommended for regular users.

  • Folds.jl with a unified interfae for a variety of executers (multithreading, multiprocessing, CUDA)
  • FLoops.jl with @floop and @reduce macros.
  • ThreadsX.jl for parallelized Base functions.

Low level constructs

Task-based asynchronous programming

Multithreading within in a CPU core, a.k.a “Green threading”.

Multi-threading

  • Caveats in multithreading
  • Threads share memory causing data racing if data access is not properly handled.
  • Suitable for SMP (Symmetric multiprocessing)

To increase the threads avaiable to Julia, start Julia with -t / --threads argument. e.g.

1
julia -t auto

Or through an environment variable, JULIA_NUM_THREADS=n. For example in ~/.profile:

1
export JULIA_NUM_THREADS=4  # or $(nproc), the # of cpu threads available

You could check how many thread are available in the Julia session

1
Threads.nthreads()

Enable multithreading by

  • Threads.@threads for loop
  • Threads.@spawn expr

Multiprocessing

Multiprocessing @ Julia manual

  • Suited for clusters
  • Independent memory pool by default, except for shared memories.
    • Unlike Python, where multiprocessing is done by forking and worker processes inherit data from the main process.
    • As a result, @everywhere is needed for code running in parallel to copy the data to the worker processes.

A simple example to start julia with multiple processes:

1
2
# start julia with 1 main process + 5 worker processes with -p
julia -p 5

To load files for all processes without @everywhere, use -L file

1
2
3
4
# Julia with 1 main process + 7 worker processes with -p
# Load mod1.jl and mod2.jl for all processes with -L
# And execute run.jl
julia -p 7 -L mod1.jl -L mod2.jl run.jl