Multiprocessing Niceness in Python

Andrew Bolster

Senior R&D Manager (Data Science) at Synopsys Software Integrity Group and Treasurer @ Bsides Belfast and NI OpenGovernment Network

Quick and dirty one that tripped me up.

Recently I’ve been doing lots of multiprocessing and joblib-based parallel processing, with loooong simulation times.

In an effort to make sure that my machine was still useable during these runs, I changed the ‘niceness’ value of the spawned processes… or so I thought.

import os
def thread_mask(args):
    # Properly Parallel RNG


    return long_simulation(args)

First part is a handy way to make sure that your subprocess simulations actually use different random numbers…. which for Monte Carlo style simulation is pretty damned important…

But, simple enough os.nice call. I reads like it says “Set the niceness value of this process to 5”.

It does not, it Increments the niceness value. (And returns the new value)

This means that after a few repeated iterations of this simulation, my processes end up with the maximum niceness (i.e. least priority) of 19. Which is not ideal.

Simple enough fix however; swap the os.nice(5) call with:

    # Be Nice
blog comments powered by Disqus