multiSMC seeding does not work with scipy frozen distribution about particles HOT 4 CLOSED

Just to comment on what Nicolas said, when freezing a scipy distribution, you not only freeze the parameters, you also freeze the numpy random state that came with it (see [this line]( https://github.com/scipy/scipy/blob/adc4f4f7bab120ccfab9383aba272954a0a12fb0/scipy/stats/_distn_infrastructure.py#L594) in scipy source code). So now how does that affect the multiprocessing? Fundamentally there are two conflicting phenomena at stake here: global state and pickling. The way numpy seeding works is by modifying a global object generator object (you can find it under `np.random.mtrand._rand`). Now before forking new processes and running the function you want to parallelize, multiprocessing pickles the arguments, and at some point ends up pickling the global random state registered within the distribution. When you now unpickle the arguments in their respective processes, the reference to the global state is lost and you end up having two states, one local to your distribution, and the "new" global state that is specific to the process. Calling np.random.seed changes the latter, and not the former, hence the observed behaviour. This is probably more easily seen on the minimal example below: ```python import multiprocessing import numpy as np import scipy.stats as st loc = 0. scale = 1. my_dist = st.norm(loc, scale) class seeder(object): def __init__(self, dist): self.dist = dist def __call__(self, seed): np.random.seed(seed) print("global seed:", np.random.get_state()[1][0]) print("local seed:", self.dist.random_state.get_state()[1][0]) return self.dist.rvs() seeded_dist = seeder(my_dist) np.random.seed(0) seeds = [1, 2, 3, 4] with multiprocessing.Pool(len(seeds)) as p: res = p.map(seeded_dist, seeds) print(res) ``` Le sam. 6 juin 2020 à 15:12, Nicolas Chopin <[email protected]> a écrit :