panoptes / panoptes-pipeline Goto Github PK

expand the fake data generator to generate a network of test data - make it include multiple units, specify multiple targets, input reasonable observing times. so that a realistic cloud-based pipeline can be tested

Defined here: https://github.com/panoptes/PIAA/wiki/Metadata-for-Images-and-Postage-Stamp-Cubes

figure out how to format light curve metadata and add it to the JSON light curve entry. could be cloning the FITS file header of the corresponding postage stamp cube

needs to identify image sequence and PIC.

example: PSC_[PICID][SEQID]
but this gets really long because SEQID = [unitid][cameraid]_[start_time_of_sequence]...

The OBSERVATIONS_URL variable in metadata.search_observations points to a dead link: https://storage.googleapis.com/panoptes-exp.appspot.com/observations.csv

decide on the format of the combined master light curve for each object - also JSON to start. need to include metadata about the object and the data cubes that went into it

based on canonical RA and Dec? will be used to combine all light curves from the same star and included in PSC headers and light curve metadata

• upload PSCs to a GCS (Google Cloud Storage) bucket for archival purposes [[what is filesystem organization?]]
• upload individual light curves to a GCS bucket [[same bucket? same filesystem?]]
• run light curve analysis software on the cloud [[GCE instance?]]
• software outputs master light curves to another cloud storage location [[GCS bucket organized by star ids? CloudSQL?]]

@mimming

see https://docs.google.com/presentation/d/1LOBqTZBd1ACW3gVC4aTnYxSntBUuSiuK4QCtOLID-is/edit#slide=id.p and @joshwalawender's comment on the slide

when combining light curves from different units of the same target:

• if the midpoint of a frame (start time + half of exptime) falls within the midpoint extrema of another image sequence, they overlap?
• if multiple frames overlap, we average their fluxes as a starting approximation? and propagate the uncertainties?
• then the new timestamp for this combined flux is the midpoint between the midpoints of the overlapping frames, and this datapoint is added to master light curve?
  • with this method, best to recalculate master light curve each time new overlapping frames are added

I am installing panoptes-pipeline but always failed. My computer is mac, python 3.7. The installation stops at python3.7/site-packages/setuptools/_distutils/core.py : “_setup_distribution = dist = klass(attrs)“. I found the value of “script_args” parameter is empty, and caused “AttributeError: ‘Configuration’ object has no attribute ‘parentdir_prefix_version’ “.
I can install POCS, panoptes-data and -utils successfully. I do not know why I can not install pipeline (https://github.com/panoptes/panoptes-pipeline). I also tried

1. pip install panoptes-pipeline (no use)
2. python setup.py install, or python setup.py develop. (Fail)
3. pip install panoptes-pipeline (fail)
   Any solutions?

Let's put runnable scripts in a scripts folder and create a basic structure that is similar across repos.

/
/scripts/  # Runnable
/piaa/     # module level stuff
/docs/ 
/bin/      # Like scripts so probably won't need now.  
/tests/    # Testing always good
/examples/notebooks/ # Jupyter notebooks.
...