Possible bad parallelism in adaptiveSampling about bet HOT 3 CLOSED

I accidentally pushed the resolution of this directly to the tests_and_parallel branch (see commit e5291d3). I approve your fix for the case of no mpi4py. I noticed troubling parallelism in the collection of the samples and data arrays.

comm.Allgather was not correctly ordering arrays to match the ordering we prefer in bet.util.get_global_values. I added the use of np.ravel() to fix this issue in bet.util.get_global_values and now use bet.util.get_global_values to gather the arrays in adaptiveSampling.sampler.generalized_chains. I will make the appropriate changes in the dev branch. Furthermore, we should use bet.util.get_global_values instead of comm.Allgather to prevent similar errors.

from bet.

This is better for samples and data for sure, but it doesn't address the initial issue.

There is still an issue with all_step_ratios. Doing:
MYall_step_ratios = np.copy(all_step_ratios)
all_step_ratios = np.empty((self.num_chains, self.chain_length), dtype=np.float64)
comm.Allgather([MYall_step_ratios, MPI.DOUBLE], [all_step_ratios, MPI.DOUBLE])

with mpi4py on 1 processor gives a different result than:
all_step_ratios = np.reshape(all_step_ratios, (self.num_chains, self.chain_length))

I think maybe you should to a reshape and then do an Allgather or something like that.

from bet.

Maybe I am misunderstanding something. When I run the following code with mpi4py on 1 processor I get the same result for all the variations of creating all_step_ratios. When I run on more than 1 processor then single_all_step_ratios does NOT have the same order as the other versions of all_step_ratios, but the other versions of all_step_ratios have the same order which matches the ordering of samples and data. Currently the ordering is [a11, a12, a21, a22, b11, b12, b21, b22, c11, c12, c21, c22] where a,b,c are the different processors and there are two chains of length 2 per processor. Now I could reorder so that the ordering is [a11, a12, b11, b12, c11, c12, a21, a22, b21, b22, c21, c22], but I don't think that's necessary as long as the ordering matches between samples, data, and all_step_ratios and we know and state what that ordering is.

# -*- coding: utf-8 -*-
from bet.Comm import *
import numpy as np
import bet.util as util

num_samples = 2*3*5*3
chain_length = 3
num_chains = num_samples/chain_length
samplespproc = num_samples/size

MYsamples = (rank+1)*np.array([np.arange(samplespproc), np.arange(samplespproc), np.arange(samplespproc)]).transpose()
MYdata = (rank+1)*np.array([np.arange(samplespproc), np.arange(samplespproc)]).transpose()
MYall_step_ratios = (rank+1)*np.arange(float(samplespproc))

print 'MYsamples', MYsamples
print 'MYdata', MYdata
print 'MYall_step_ratios', MYall_step_ratios

samples = util.get_global_values(MYsamples, (num_samples, 3))
data = util.get_global_values(MYdata, (num_samples, 2))

all_step_ratios = np.empty((num_chains, chain_length), dtype=np.float64)
reshape_all_step_ratios = np.empty((num_chains, chain_length), dtype=np.float64)
UNall_step_ratios = np.empty((num_samples,), dtype=np.float64)
MYreshape = np.reshape(MYall_step_ratios, (num_chains/size, chain_length))

comm.Allgather([MYall_step_ratios, MPI.DOUBLE], [all_step_ratios, MPI.DOUBLE])
comm.Allgather([MYall_step_ratios, MPI.DOUBLE], [UNall_step_ratios, MPI.DOUBLE])
comm.Allgather([MYreshape, MPI.DOUBLE], [reshape_all_step_ratios, MPI.DOUBLE])
UNall_step_ratios = np.reshape(UNall_step_ratios, (num_chains, chain_length))

single = (rank+1)*np.arange(float(num_samples))
single_all_step_ratios = np.reshape(single, (num_chains, chain_length))

if rank == 0:
    print 'samples', samples, samples.shape

    print 'data', data, data.shape
    print 'all_step_ratios', all_step_ratios, all_step_ratios.shape
    print 'UNall_step_ratios', UNall_step_ratios, all_step_ratios.shape
    print 'single_all_step_ratios', single_all_step_ratios,single_all_step_ratios.shape
    print 'reshape_all_step_ratios', reshape_all_step_ratios,reshape_all_step_ratios.shape

from bet.