GNU parallel

GNU parallel runs commands in parallel.

For instance, to find all jupyter notebook files (docs/*.ipynb) and execute them in parallel, with 2 processes (-j2):

find docs -type f -name '*.ipynb' | parallel -j2 jupyter nbconvert --to notebook --execute --inplace {}

You can set the number of parallel jobs with the -j flag. e.g. -j8 runs 8 jobs in parallel.

Display execution log

Use --joblog <logfile>. For example,

find docs -type f -name '*.ipynb' | parallel --joblog /tmp/log -j2 jupyter nbconvert --to notebook --execute --inplace {}

cat /tmp/log