While we have been using the ditheredRA and ditheredDec for creating HealpixelizedOpSim, we have seen that this leads to lowering of number of visits in the DDF areas. It is understood that finally, DDF visits will have a smaller dither, and we will approximate this by setting the dither to 0 for DDF fields and rebuilding HealpixelizedOpSim.

• Recreate OpSimOutput so that any record with ddf propid is preferentially chosen as DDF.
• Change ditheredRA, ditheredDecto fieldRA,fieldDec
• Change code in HealpixelizedOpSim to use the new ditheredRA , ditheredDec

The values of the fact keyword and inclusive keywords are used in the query_disc function are hardcoded in, with no way of changing later. We need to allow this to be a parameter.

Currently, the class is initialized with a pandas DataFrame object, and reading the opsim output into it is the user's responsibility. This is useful, in that the user can select different parts of the opsim through sql queries before writing to dataFrame. On the other hand, the user has to do these things.

The goal would be to allow the same flexibility but offer an easier one line method to read in standard
opsim outputs.

• test reading DB,
• test reading HDF

Currently, the subset attribute of OpSimOut acts more like a registry to a sequence of propIDs. However, even when we know the propID values we want to select, we cannot set subset to None in the argument. (We need to set it to the default combined) for it get the correct tables.

• We need to fix this problem.

A way to do this is by actually creating a registry and populating it from the proposal table. But it is not clear if that is what we want to do.

• Would like to add it for backward compatibility, but will not happen immediately.

Breaks examples.

Since these are only used for writing out SNANA simlibs, it is appropriate to move them to the writesimlib function.

• Use filtSkyBrightness and OpSim Seeing to estimate the sky noise putting in the correct values of quantities in

Add example showing the way the healpixels and boundaries work

#43 was fixed by a quick and dirty solution of adding a new script to add the proposal table to the test OpSim database. What we want to do is to

• keep the methods in make_smallOpSim available as part of a utils library in opsimsummary or elsewhere
• make methods of writing such scripts easier by adding query and write methods to OpSimOut or appending different records, and use those to write a single script.

• add a script to create an opsim HDF file
• create example hdf file in example_data directory for testing

Given that the class has an attribute simlibDict with keys of fieldID, this is superfluous. It may be convenient to redo this function to give the data attribute of FieldSimlib.

Add mechanism for pruning pointings for a set of points.

Pruning is solved for HealpixTiles, but should be kept in the general case. Perhaps this is more an issue for varsim, rather than OpSimSummary

Currently, opsimsummary.HealpixelizedOpSim can only be instantiated from a dataFrame, which the user must provide. Create class methods

• fromOpSimDB to create these from opsim sqlite databases,
• fromOpSimHDF and convenient storage of the Summary table on hdf file with argument 'table'. This will allow MapTile2Pointing (#41) work more easily.

htree = HealpixTree(nside=1, nest=True)
htree.pixelsAtResolutionLevel(0, 8, nside=None)

throws the errors:

global name 'hpt' is not defined

Currently, OpSimSummary imports objects from the visualization.py module from the __init__, implying that not having a basemap will cause the install of opsimsummary to fail. The problem is that basemap seems somewhat hard to install (though it is usually easy to install via conda install -c anaconda basemap). For example, this seems to be the case on Cori, after installing the python modules and the stack for reasons I have not figured out. However, basemap is not required for everything, so it is better to separate this out, so that opsimsummary install can proceed without basemap with an appropriate warning, rather than an exception.

There should be an option to allow SimulationTile to work with a fixed set of pointings. The best way to ensure this is by allowing a numpy format associated with it.

Currently, a Healpixelized OpSim database writtent to disc does not contain any provenance information. Add this information in the form of a table Called metadata:

This table should contain the following information

version : string
OpSim DataBase : string
NSIDE : int
factor : int
inclusive : True/False
date : time at which this is written
machine: host information
indexed : True/False

• Check if the script is running in travis CI or not.
  This stack overflow answer explains how this can be done
• If it is in travis, set location of the MINICONDA script to $HOME, else read a file off a hidden file to set the tarball location.
• Create a conda environment rather than install directly.

Currently, the pointings observing each point source is calculated either by query disc to provide a maximal set of pointings overlapping the tile, and then pruned according to the position of the point. The pointings overlapping the tile are calculated through functions like Healpy.query_disc and the process is slow if the million observations of the entire OpSim is considered along with millions of Healpixels. The gain in using HealpixelizedOpSim is that there are numerous sources in each Healpixel, and don't require reading through the entire OpSim. This is often done with a preComputed HealpixelizedOpSim, but these are large files which need to be stored.

The proposal is to use BallTree methods to speed up the calculation of pointings for each SN.

The precomputed HealpixelizedOpSim stored for NSIDE=128 (NSIDE=256 is too large ~ 1GB) corresponding to the enigma_1189_micro.db database used for continuous integration is ~200MB. This can be brought down to ~50 MB using gzip compression as shown in scripts/make_healpixeldb.py. While ~200MB will probably work, it is still a lot of space, and it would be good to replace this with the ~50 MB file. This also means that tests will have to start from gzipped opsim databases.

• include gzipped version and add script in .travis installs to unzip the database in situ
• Improve support by using sqlalchemy + gzip libraries to make opsim_output.py be able to do this.

Currently, OpSimOutput and its class method fromOpSimDB is designed to use information from the ProposalTable. In order to use new opsim like simulations, where the structure is slightly different, we should allow it to work even if the proposal table does not exist, as long as a propDict is provided.

Work out three different bands of equal area covered by the WFD field, and the approximate time spent on these areas.

Bad tiles are:

1. indexed too large for the NSIDE
2. No pointings for those tiles even though they are valid tiles for healpix with given NSIDE.

When I wrote this, I set the MWEBV values to 0.0 to get started. This is clearly wrong, and needs to be corrected by using a proper set of values.

This test has been set to not.test in
https://github.com/rbiswas4/OpSimSummary/blob/Issue%2352/HealpixDB_test_script/tests/test_healpix.py#L93

This is because the standard db was rewritten. It is not clear what is the best way to keep this. Perhaps using an older standard database or writing that in a better way helps.

Cadence Plot shows nights as the time axis, and should also (in fact by default) show MJD.
There is also no way to specify an MJD and an MJD range to make the plot. These are both desirable features.

While this is allowed (raCol and decCol are parameters to the constructor), we are missing the use of raCol and decCol in the call to addVector call if the vectors need to be made.

• We have a class opsimsummary.HealPixelizedOpSim which now provides an API for many of the tasks in scripts to write out healpixels to databases. We should now rewrite the scripts to uses this API
• There should be a test of the writing to database API tested against the sparse matrix algebra.

We need a script to produce healpixelized versions of enigma_1189_micro.db for tests. This blocks
rbiswas4/SNsims#19

Currently, we access the example_data directory from the package path by looking at the package file as in the following way:

pkgDir = os.path.split(oss.__file__)[0]
dbname = os.path.join(pkgDir, 'example_data', 'enigma_1189_micro.db')

By defining the pkgDir variable in the __init__.py of the package, we should be able to find the pkgdir as

pkgDir = opsimsummary.pkgDir
data_dir = oss.example_data

Making it more convenient to access in examples and tests.

We need to represent NSIDE or npix on the sqlite database. This is essential for rbiswas4/SNsims#23

A scipy dependency was introduced due to the usage of scipy.sparse and the install of scipy through pip fails on travis ci. There are a large number of articles on the ne, suggesting this is a common problem and the best cure is to move to conda install.

It seems that it will be easier to have the tessellation codes in this package

This is a problem which has been on and has been found elsewhere too. This problem happens if a username is not specified in SummaryOpSim. This stack exchange post explains what needs to be done:

http://stackoverflow.com/questions/842059/is-there-a-portable-way-to-get-the-current-username-in-python

SNANA expects the coordinates to be in degrees, but the writesimlib function is writing them out in radians.

Currently there are two ways envisaged for obtaining the ObsHistID values for a tile ID.

1. using a computation from a dataframe representing the opsim output which is implemented in opsimsummary.HealpixelizedOpSim.obsHistIDForTile() and requires the opsim output as a dataframe as input.
2. Precomputed results of 1. stored in a database.

The idea is that there should be a common way of reading both of these and providing obsHistID for a tileID for both of these methods. Create a class MapTile2Pointing in healpix.py to do this.

The micro database created for examples only had the summary table. Now, we have started using the proposal table. Thus in order for our examples to work, we need to write out the proposal table as well.

fig1c = EnigmaDeepSummary.cadence_plot(fieldList[0], observedOnly=False, sql_query='night < 250 and night > 230', nightMax=250, nightMin=230, deltaT=1) as shown below. The xticklabels should have started from 230 instead of 0. 

The code that I think should do this is inserted, but commented out right now, because of an EOF error

While the HealpixelizedOpSim API allows one to name a vector column, addVec which is called by the init statement does not. This should be remedied.

Currently, we have a class method OpSimOut.fromOpSimDB which allos one to consturct and OpSimOut object from the database directly. It would be good if this could support a sql_query method to allow users to apply the sql query before the usual queries involved.

A use case is what is needed in Twinkles where we only care about a single fieldID. In this case it would be useful to be able to apply the sql query of SELECT * FROM Summary WHERE fieldID == 1427 in a pd.sql_query statement rather than a pd.read_sql_table statement.

We have had examples of how the hdf serialization of OpSim summary tables would work.

• It would be good to put this in a method of sumamrize_opsim.OpSimOutput so that it is easily accessible.
• We must add the proposal table to the hdf files so that it can work with the read in methods. Thus enigma, minion, opsim outputs must be redone
• While redoing, we must use the '_all' format that we introduced in the refactor, and then use selection in the hdf read method.

fig = EnigmaDeepSummary.cadence_plot(290, sql_query='night > 180 and night < 200', nightMax=200, nightMin=180, observedOnly=True, deltaT=5)
plt.savefig('cadence.pdf')

Gives an error. The error is due to trying to extend the matrix to a smaller region.

• Get set of pointings for healpix tiles.
• Write tests and example demo.

Do part of #54

• Include gzipped version of healpixelized database produced by script scripts/make_healpixeldb.py.
• add script in .travis installs to unzip the database in situ
• Update users how they can run on their machines.

This means users will have unzip this database themselves unless there is a install script as suggested in #37 . This probably means adding to install/readme instructions

Information about zerodithers for ddf and the subset used should be added to HealpixelizedOpSim output in the form of database in the table metadata