Add ability to load experiment in same way as Bishop

Experiment.Replicate() should ensure it can actually run the simulation. Otherwise, the error gets repeated N times.

Add ability to search for minimum bias in a participant model when user specifies power

Add sample size searchers through binary search that return a single sample size with the power the user sets

Have Tests only work on one or two conditions and let DataTest run it in parallel and create an aggregate choice (Unless the test already handles multiple conditions).

Transfer meta-analysis hierarchical stuff to Bunny. Preferably as a new class. This will be the core part of Bunny.chomp()

Add support for saving experiment objects.

Needs work!

Test = Bunny.DataTest(Bunny.TestLibrary.TTest())

Experiment = Bunny.Experiment([Effort,NoEffort],Test,"Baby effort experiment")

Bunny.Explore(Experiment)
/Users/julianjara-ettinger/anaconda/lib/python2.7/site-packages/numpy/core/_methods.py:83: RuntimeWarning: Degrees of freedom <= 0 for slice
warnings.warn("Degrees of freedom <= 0 for slice", RuntimeWarning)
/Users/julianjara-ettinger/anaconda/lib/python2.7/site-packages/numpy/core/_methods.py:117: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
/Users/julianjara-ettinger/anaconda/lib/python2.7/site-packages/scipy/stats/stats.py:3107: RuntimeWarning: invalid value encountered in absolute
prob = distributions.t.sf(np.abs(t), df) * 2 # use np.abs to get upper tail
Exploring sample sizes ...
No limit specified. Testing samples between 15 and 35 ...
Estimating time ....
Traceback (most recent call last):
File "", line 1, in
File "Bunny/BunnyFunctions.py", line 14, in Explore
res = ExploreSampleSize(Exp)
File "Bunny/BunnyFunctions.py", line 148, in ExploreSampleSize
Power.append(Exp.GetPower(samples))
File "Bunny/Experiment.py", line 55, in GetPower
return sum(results)*1.0/len(results)
TypeError: unsupported operand type(s) for +: 'int' and 'list'

Have validation test run simulating the data and testing it and catch any errors it gets

Bunny.Hop uses binary search, which assumes that the power function p:SampleSize->[0,1] is increasing.

If you use a null model, the goal is to find a sample size where your test will only detect something less than (1-power)% of the time. So Hop won't work:

Bunny.Hop(Experiment,limit=100,power=0.05)
Power: 0.05
Limit: 100
Replications per proposal: 10000
Searching for your sample size...
Simulating with 50 participants per condition... Power=0.0577
Simulating with 25 participants per condition... Power=0.0631
Simulating with 13 participants per condition... Power=0.0812
Simulating with 7 participants per condition... Power=0.1178
Simulating with 4 participants per condition... Power=0.1809
Simulating with 2 participants per condition... Power=0.4001
Simulating with 1 participants per condition... Power=0.9058
[1, 0.9]

Example docstrings show different parameters than the defaults ones, which makes it really confusing!

Add support for pre-registering a design

You'll need the test functions to also return the key statistics (e.g., in binomial, the test should also return the percentage succeeding).

Do simulation analysis for pilot data to test if result is reliable after correcting for piloting

If a datatest only works on one condition, it should run in parallel across all conditions of a behavior model.

Openssl has problems finding the certificate in some anaconda installations:

WARNING: can't open config file: /opt/anaconda1anaconda2anaconda3/ssl/openssl.cnf

This is a known issue (ContinuumIO/anaconda-issues#137) and not a Bunny problem, but I need to find some other way of pre-registering.

Add function to load most common types of experiment designs

BehaviorTestModel = Participant(ParticipantLibrary.BinomialAgent(0.75))
BehaviorControlModel = Participant(ParticipantLibrary.BinomialAgent(0.5))

The lest library comes with a test that does this:

Test = DataTest(TestLibrary.BinomialWithControl("OT"))

You will see a message letting you know that Bunny is assuming that the first condition is test and the second one is control.

MyExperiment = Experiment([BehaviorTestModel, BehaviorControlModel], Test, "2-AFC task")

Add time estimate to hop function

Auto-binning is kind of funky. Need to adjust bin size better

ParticipantLibrary should have a function that takes an observed vector and returns random samples from it.

Bunny.Save(Experiment,"Test")
Traceback (most recent call last):
File "", line 1, in
File "Bunny/BunnyFunctions.py", line 93, in Save
pickle.dump(Exp,open(Filename, "wb"))
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 1370, in dump
Pickler(file, protocol).dump(obj)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 600, in save_list
self._batch_appends(iter(obj))
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 615, in _batch_appends
save(x)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 419, in save_reduce
save(state)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 649, in save_dict
self._batch_setitems(obj.iteritems())
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 663, in _batch_setitems
save(v)
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
File "/Users/julianjara-ettinger/anaconda/lib/python2.7/pickle.py", line 748, in save_global
(obj, module, name))
pickle.PicklingError: Can't pickle <function F at 0x1056c9050>: it's not found as Bunny.ParticipantLibrary.F

Add capability to read csv files for chomp to process.

AttributeError Traceback (most recent call last)
in ()
----> 1 Hop(MyExperiment)

/Users/julianjara-ettinger/anaconda/lib/python2.7/site-packages/Bunny-0.1.3-py2.7.egg/Bunny/BunnyFunctions.pyc in Hop(Exp, limit, power, samples, Verbose)
92 sys.stdout.flush()
93 Exp.SetSampleSize(current)
---> 94 p = Exp.GetPower(samples)
95 if Verbose:
96 sys.stdout.write("Power=" + str(p) + "\n")

/Users/julianjara-ettinger/anaconda/lib/python2.7/site-packages/Bunny-0.1.3-py2.7.egg/Bunny/Experiment.pyc in GetPower(self, samples)
119 >> MyExperiment.UpdatePower()
120 """
--> 121 results = self.ExtractDecision(self.Replicate(samples))
122 return sum(results) * 1.0 / len(results)
123

/Users/julianjara-ettinger/anaconda/lib/python2.7/site-packages/Bunny-0.1.3-py2.7.egg/Bunny/Experiment.pyc in ExtractDecision(self, Data)
136 power (bool): Indicator about whether simulation fulfilled DataTest's final criteria.
137 """
--> 138 return [Data[i].aggregatedecision for i in range(len(Data))]
139
140 def Replicate(self, N=1000):

AttributeError: 'NoneType' object has no attribute 'aggregatedecision'

to test library

Add Bunny.Inpsect(Experiment) as a supporting function to print an experiment's summary and power (to be used for preregistration)

If there's more than one test it should just run everything and output if they all succeeded. I might need to add some support for selecting which conditions matter.

What if bunny stat and participant libraries included a potential intuitive description of the test. The preregistration method could then also describe the model with words, also useful for writing methods sections explaining the power analysis.

I'm starting to add way too much staff to stay tests. They should really be create an object of simulationsummary or something! Then they could also have a list of critical statistics of something.