Comments (3)
I accidentally pushed the resolution of this directly to the tests_and_parallel branch (see commit e5291d3). I approve your fix for the case of no mpi4py. I noticed troubling parallelism in the collection of the samples and data arrays.
comm.Allgather was not correctly ordering arrays to match the ordering we prefer in bet.util.get_global_values. I added the use of np.ravel() to fix this issue in bet.util.get_global_values and now use bet.util.get_global_values to gather the arrays in adaptiveSampling.sampler.generalized_chains. I will make the appropriate changes in the dev branch. Furthermore, we should use bet.util.get_global_values instead of comm.Allgather to prevent similar errors.
from bet.
This is better for samples and data for sure, but it doesn't address the initial issue.
There is still an issue with all_step_ratios. Doing:
MYall_step_ratios = np.copy(all_step_ratios)
all_step_ratios = np.empty((self.num_chains, self.chain_length), dtype=np.float64)
comm.Allgather([MYall_step_ratios, MPI.DOUBLE], [all_step_ratios, MPI.DOUBLE])
with mpi4py on 1 processor gives a different result than:
all_step_ratios = np.reshape(all_step_ratios, (self.num_chains, self.chain_length))
I think maybe you should to a reshape and then do an Allgather or something like that.
from bet.
Maybe I am misunderstanding something. When I run the following code with mpi4py on 1 processor I get the same result for all the variations of creating all_step_ratios. When I run on more than 1 processor then single_all_step_ratios does NOT have the same order as the other versions of all_step_ratios, but the other versions of all_step_ratios have the same order which matches the ordering of samples and data. Currently the ordering is [a11, a12, a21, a22, b11, b12, b21, b22, c11, c12, c21, c22] where a,b,c are the different processors and there are two chains of length 2 per processor. Now I could reorder so that the ordering is [a11, a12, b11, b12, c11, c12, a21, a22, b21, b22, c21, c22], but I don't think that's necessary as long as the ordering matches between samples, data, and all_step_ratios and we know and state what that ordering is.
# -*- coding: utf-8 -*-
from bet.Comm import *
import numpy as np
import bet.util as util
num_samples = 2*3*5*3
chain_length = 3
num_chains = num_samples/chain_length
samplespproc = num_samples/size
MYsamples = (rank+1)*np.array([np.arange(samplespproc), np.arange(samplespproc), np.arange(samplespproc)]).transpose()
MYdata = (rank+1)*np.array([np.arange(samplespproc), np.arange(samplespproc)]).transpose()
MYall_step_ratios = (rank+1)*np.arange(float(samplespproc))
print 'MYsamples', MYsamples
print 'MYdata', MYdata
print 'MYall_step_ratios', MYall_step_ratios
samples = util.get_global_values(MYsamples, (num_samples, 3))
data = util.get_global_values(MYdata, (num_samples, 2))
all_step_ratios = np.empty((num_chains, chain_length), dtype=np.float64)
reshape_all_step_ratios = np.empty((num_chains, chain_length), dtype=np.float64)
UNall_step_ratios = np.empty((num_samples,), dtype=np.float64)
MYreshape = np.reshape(MYall_step_ratios, (num_chains/size, chain_length))
comm.Allgather([MYall_step_ratios, MPI.DOUBLE], [all_step_ratios, MPI.DOUBLE])
comm.Allgather([MYall_step_ratios, MPI.DOUBLE], [UNall_step_ratios, MPI.DOUBLE])
comm.Allgather([MYreshape, MPI.DOUBLE], [reshape_all_step_ratios, MPI.DOUBLE])
UNall_step_ratios = np.reshape(UNall_step_ratios, (num_chains, chain_length))
single = (rank+1)*np.arange(float(num_samples))
single_all_step_ratios = np.reshape(single, (num_chains, chain_length))
if rank == 0:
print 'samples', samples, samples.shape
print 'data', data, data.shape
print 'all_step_ratios', all_step_ratios, all_step_ratios.shape
print 'UNall_step_ratios', UNall_step_ratios, all_step_ratios.shape
print 'single_all_step_ratios', single_all_step_ratios,single_all_step_ratios.shape
print 'reshape_all_step_ratios', reshape_all_step_ratios,reshape_all_step_ratios.shape
from bet.
Related Issues (20)
- TODO: fix parallel loading of sample sets HOT 10
- revisit PendingDeprecationWarning HOT 1
- Mismatch of docs and definition of postTools.sample_prob HOT 4
- TODO: Feature Scaling HOT 1
- How we do installs HOT 4
- TODO: un-lock scipy HOT 6
- TODO: clean up some tests
- why no pickle? HOT 10
- discussion of decorators
- sampler class options HOT 2
- document when predicted distribution is erased
- nose deprecation HOT 9
- data-consistent methods HOT 5
- Deprecate adaptive sampling? HOT 3
- no mpi4py + mpirun for plotting HOT 2
- May 19 Meeting HOT 9
- Improve Fitting Random Variables
- Parallel Tests HOT 19
- New features for estimating observed and predicted distributions (other than KDE) HOT 4
- Create better (and simpler) plotting routines for density based approach
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bet.