Create hdf5 grouping structure about jif HOT 5 CLOSED

Looks good. I agree with the separate branches for space and ground, even if we primarily use only one branch.

Will all groups fit in a rectangular window (as opposed to a more complicated shape)? We may also want to include any mask information that fits inside the group footprint so we know when sources are expected to be chopped off.

from jif.

We need to look at Erin Sheldon's MEDS DES formalism for grouping objects. This is what LSST DM will likely build off of (private communication Jim Bosch).

from jif.

After contacting Erin he responded

Hi Guys -

That meds library (https://github.com/esheldon/meds) is currently only for
reading the files. The code that creates them is a C++ code located here:

https://github.com/rmjarvis/deswl_shapelets/blob/master/src/MakeCutouts.cpp

It is C++ for historical reasons: we have a nice WCS class in C++ and I
leveraged that. I would do it in python now, and plan to rewrite it some
time soon.

Anyway, some documentation is at the top of that file. I will also be
describing it in great detail in our upcoming DES shear catalog paper, which
is about 85% complete.

I can send you a copy of that when it is cleared for release, or if you really
want to see it soon I could rip out that section and latex it up separately
(it is public information anyway, since you could figure it out from the code)

Let me know if you have any questions.
-e

from jif.

Here are some questions we asked Erin along with his responses:

Q: It looks like you are using fits file format. I am guessing it is because
you can have the WCS header easily attached to the cutout images. Did you
consider any other file structures like HDF5?

A: The reason is to remove a heavy dependency like HDF. Also with the
fitsio python package (not pyfits), we get fast random access within
the data structure (all postage stamps are stored internally in one
large 1-d array). Note we fpack the files, which reduced the size by
a factor of 5 or so.

Q: Is there are reason you only have square cutouts and of fixed sizes?

A: The box sizes are chosen to faciliate FFTs. The sizes are in 2N or 3*2N.

from jif.

Based on the previous information we've decide to diverge from the MEDS format for the following reasons:

1. HDF5 is not a "heavy" dependency in the frameworks we¹re already using.
2. We'll be extremely lucky if we ever get to a regime where we're IO
   limited!
3. A flat data structure sounds like a bookkeeping headache for the
   hierarchical source models we¹ve been discussing.
4. And, of course, we don't need to compute FFTs of our cutouts because we're
   forward modeling.

from jif.