ut-chg / bet Goto Github PK
View Code? Open in Web Editor NEWPython package for data-consistent stochastic inverse and forward problems.
Home Page: http://ut-chg.github.io/BET
License: Other
Python package for data-consistent stochastic inverse and forward problems.
Home Page: http://ut-chg.github.io/BET
License: Other
The docstrings are not very descriptive in plotDomains, and there are almost no comments. Variable names are not descriptive enough for not having commments. Also, default input arguments should be added to many of the functions.
Change the links in the example .rst files to point to the github repo for the example rather than download the file.
Properly reclass the heuristics in adaptiveSampling so that they all inherit from the heuristic class.
Plotting 2d and then 1d (or reversing the order) marginals will combine the plots in the last saved files.
Add a test to verify that prob_emulated(samples, data, rho_D_M, d_distr_samples) = prob(samples, data, rho_D_M, d_distr_samples)
Allreduce calculations for integers are all wrong. They are begin treated as numpy arrays of integers which is not giving an error but is doing something completely wrong. Should be switched to allreduce. Many parts of simpleFunP and calculateP are affected.
The solutions, P, from prob,prob_samples_mc, prob_samples_qhull need to be compared to the solutions from the corresponding MATLAB codes Q_singleV, Q_singleVmc, and Q_singleVex (make sure the bin_sizes are equivalent). The bin_size in the MATLAB code is 1/bin_size in the Python code.
Write it.
Update plotDomains so that the dimensions of Q_ref (and similar variables) are automatically fixed if necessary. Automatically create a figs folder if it doesn't already exist and if a custom file name is not specified.
Make sure examples work with changes to input and output arguments.
Currently the dimensions for samples in the parameter space (\Lambda) and samples in the data space (D) are:
param, lambda -- (ndim, num_samples)
data -- (num_samples, mdim)
We desire the dimensions of samples from both spaces to match (num_samples, dim). Possible modules this needs to be fixed in are:
adaptiveSampling
basicSampling
simpleFunP
calculateP
plot_D_2D
In line 35 of calculateP in the emulate_iid_lebesgue method:
num_l_emulate = int(num_l_emulate/size)+1
Apparently size comes from bet.Comm and is the number of processors used in the computation, so when this operation is called with only a single processor it creates 1 more emulated sample than called for by num_l_emulate parameter.
I came across this while creating the verification example to re-create a uniform distribution on \Lambda and worked around it by passing num_l_emulate-1 into this method. I think the fix is to compute
remainder = num_l_emulate % size
and then for the first "remainder" number of processors add 1 to num_l_emulate. I am not quite sure how to code this though.
Write model running rig that takes care of load balancing within a single TACC(or other HPC) job allocation. We should use mpi4py and maybe look into Pylauncher by TACC. This should take care of running both serial and parallel jobs in parallel. It might make sense to use GNU parallel (but it probably doesn't make sense). Pushkar is interested in this so we should include him.
The documentation needs to be remade.
Merge master into dev without clobbering developmental portions of dev that differ from master. This is to keep dev in line with the current state of master since there were bugs/issues that were resolved with the addition of the sensitivity package to master.
Change copyright at the tops of files to "The BET Development Team" instead of "Lindley Graham and Steven Mattis".
@smattis is the tests_and_parallel branch ready for merging? If so want me to go ahead and merge it to master?
Add nose as a dependency. Make sure the install documentation via Sphinx is up to date. Add note on how to run tests after installation.
Finish creating sensitivity package. Create master-sensitivity branch from dev-sensitivity. Remove developmental code from master-sensitivity. Merge master-sensitivity into master. Update documentation. Release as version 0.3 or version 1.0 if all examples are finished.
these templates should be for parallel and serial versions. make sure PolyADCIRC examples and other examples conform to these templates
In line 348 the data arrays are concatenated along the 0-axis, but it should be done along the 1-axis. This may mess up the fromMap example, so I didn't want to change it yet.
Check methods that use center_pts_per_edge. Update these methods so that the argument can be either an int or a list(). If the method receives a list check it to make sure the length matches the dimension of data. If the method receives a int, expand it into an appropriately sized array.
Add way to create uniform hyperrectangle by just specifying the dimensions of that rectangle.
eliminate unnecessary returns and collecting of lambda_emulate for prob at least.
make sure to not return inputs as outputs.
In postTools.sample_highest_prob, if you set top_percentile=1.0, it returns the correct volume of the region in Lambda, but it return num_samples= all samples. It looks like this is because in line 63 postTools
num_samples = np.sum(P_sum <= top_percentile)
So Psum is always <= 1.0, and it returns all samples. Maybe <= top_percentile - epsilon, or an
if top_percentile==1.0
num_samples = numbers of nonzeros in P_samples
Be careful with old code using postTools. Recent additions by @lcgraham break backward compatibility. I didn't realize this when I merged it in because the tests were rewritten for the new syntax. I am going to make this more user friendly, but I guess I'll follow the new form.
We will "wailk" through the following example. This
http://ut-chg.github.io/BET/examples/example_rst_files/validation.html#validation
Change all references to "true" or "truth" to "reference" i.e. "q_true" changes to "q_ref" and "heuristic" to "kernel."
When checking to see if an object is None object == None
use type(object) == NoneType
instead.
Add indicatorFunctions.py to master from dev branch. Create a class/set of methods that returns the indicator functions and their simple function approximations together since most of these methods take the same arguments. We can create a similar set up in sampling on the dev branch for the gradient based adaptive samplers.
Add forward problem capabilities and remove the references to a specific \D or \Lambda space and just say metric space.
Update parallel, overview, and examples to include information about sensitivity package.
There is some troubling parallelism in adaptiveSampling.sampler.generalized_chains regarding all_step_ratios. Especially the lines :
MYall_step_ratios = np.copy(all_step_ratios)
all_step_ratios = np.empty((self.num_chains, self.chain_length), dtype=np.float64)
comm.Allgather([MYall_step_ratios, MPI.DOUBLE], [all_step_ratios, MPI.DOUBLE])
In this process, all_step_ratios is being broadcast into a different shape. We need to be careful about the ordering here, I think. The order of entries in all_step_ratios is different after this than doing numpy.reshape. Tests were failing without mpi4py because of this.
I put a temporary fix in the tests_and_parallel branch. I would like @lcgraham to look over this and make sure that there are no errors, and that my fix for the case of no mpi4py is valid. It does make all of the test pass. If you do make changes, please do it in the tests_and_parallel branch, to avoid conflicts when that is merged with master, which should be done very soon.
See todos in *.rst files.
There are many dimension-dependent calls of numpy.meshgrid in voronoiHistogram. We should use util.meshgrid_ndim to make things more robust.
BET/bet/calculateP/calculateP.py:262: RuntimeWarning: invalid value encountered in divide
P[Itemp] = rho_D_M[i]*lam_vol[Itemp]/np.sum(lam_vol[Itemp])
loadmat tests are failing in parallel with bet.sampling.adaptiveSampling and bet.sampling.basicSampling
Need to make sure *.mat files from parallel and serial runs are getting properly removed in the tearDown method
Adjust the plotDomains tests so that they run in seconds instead of minutes.
Port stable features from polysim.run_framework.adaptive_sampling into bet.sampling.adaptiveSampling. Eventually, the adaptive sampling module in PolySim should be completely removed and examples/data in hershel_bin/nonsubdomain/adaptive and the corresponding scripts in lonestar_landuse_bin should reflect this change.
make sure names for plotP and plotDomains match up with respect to function calls, same syntax
Tests for bet.postProcess.plotP fail when run in parallel.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.