sdss / marvin Goto Github PK

Issue by albireox
Thursday Jul 07, 2016 at 06:46 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/5

Brian:

We should add convenience properties into the Nsa ModelClasses?. For starters, a good list could include broadband colors between FNugriz, log of stellar mass and similar quantities, and magnitudes of all the fluxes.

Here are examples for how to define something on both the class and instance side, such that the new property can be returned as a parameter, and used in a query filter for the postgresql query.
plate-ifu property in Cube modelclass

    @hybrid_property
    def plateifu(self):
        '''Returns parameter plate-ifu'''
        return '{0}-{1}'.format(self.plate, self.ifu.name)

    @plateifu.expression
    def plateifu(cls):
        return func.concat(Cube.plate, '-', IFUDesign.name)

old version of log stellar mass from the deprecated Sample modelclass. There might be a better way to do this to account for NULL and -9999.0 values of the stellar mass on the SQL side.

    @hybrid_property
    def nsa_logmstar(self):
        try:
            return math.log10(self.nsa_mstar)
        except ValueError:
            return -9999.0
        except TypeError:
            return None

    @nsa_logmstar.expression
    def nsa_logmstar(cls):
        return func.log(cls.nsa_mstar)

Issue by havok2063
Tuesday Oct 18, 2016 at 08:15 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/82

Marvin queries should be asynchronous. Ideally, users submit queries to the server which get queued into a system, run, and then return results or a notification when the results have finished. This may mean setting up a server-side user session which stores submitted queries and results.

At some level, this means that requests should be asynchronous, rather than the queries themselves. The Interaction class should not wait for a response from the server.

This all could go hand-in-hand with the move to Trac-based user authentication for the API, using oAuth, with user session and history, etc.

Issue by albireox
Thursday Jul 07, 2016 at 06:59 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/9

Triggered by #1, should we start considering compacting all the ModelClasses into Marvin (or some other product)? This would probably remove the dependence on SPM.

Issue by albireox
Thursday Oct 06, 2016 at 14:53 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/64

All Marvin files (cube, DAP maps, modelcubes, RSS) share a good number of similarities. It would simplify the code to have a MarvinFile class that gets instantiated with a filename and defines things like ._mplver, ._drver, .header, .wcs, .shape, etc. Then that MarvinFile can be used in the Marvin tools when data_origin='file'.

Issue by albireox
Thursday Jul 07, 2016 at 06:43 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/4

Reported by Brian:

Not sure if there is/was a related ticket for this already. The hybrid property plateifu is causing issues. I suspect this will be generic for all added properties.

On the instance side, plateifu is a string. On the class side, plateifu is the concat function. This means that when cube.plateifu is requested as a query return parameter, the param_to_form_lookup.mapToColumn method returns a concat function and not a ModelClass? attribute.

So when we create the join tables out of these parameters at line 276 in join_tables, it crashes but plateifu concat does not have the .class property.

AttributeError: Neither 'concat' object nor 'Comparator' object has an attribute 'class_'

For all of our hybrid properties, and perhaps normal added properties, on the ModelClasses?, how will we generate the ModelClass? to pass into the joins?

Assigning to you José, since it's related to how we are handling parameters in the _param_to_form_lookup, plus the query parameters, plus how we create the list of Join ModelClasses?, but really just opening this up for discussion.

Things will continue to work as long as we don't query on any of these parameters, and plateifu is a doozy of a parameter.

Issue by havok2063
Wednesday Oct 19, 2016 at 09:50 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/83

Ahead of DR14 and near to post-Beta, implement automated and extensive testing environment, with coverage. This includes using Travis-CI to automate all tests across multiple environments and using Coveralls.io to monitor testing coverage over time. Repos must be public. Can be set up for both Python and Javascript environments.

API tests - python + http requests
Tools tests - python
Web tests - python + flask + javascript

Python - can use coverage
Javascript - can use grunt-coveralls, mocha+istanbul

Add grunt-coveralls to Gruntfile to map coverage results to coveralls.io. Look into mocha+instanbul for building actual JS tests and see if we can merge that with grunt-coveralls to pipe output to coveralls.io.

I think this will potentially save us a load of trouble down the line post-beta trying to 1.) ensure robust quality control on web and api features, 2.) handle community-driven updates on tools, 3.) having to remember to run tests

https://travis-ci.org
https://coveralls.io
https://github.com/coagulant/coveralls-python
http://gruntjs.com/plugins - search coverage
https://github.com/pimterry/grunt-coveralls
https://mochajs.org/
https://github.com/gotwarlost/istanbul

Issue by albireox
Thursday Oct 06, 2016 at 14:57 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/65

Quite self-explanatory.

Issue by albireox
Thursday Jul 07, 2016 at 06:39 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/3

Implement this in Cube or in Spaxel.

Issue by bretthandrews
Thursday Jun 30, 2016 at 16:13 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/1

The keyword sample.nsa_logmstar is available to query on, but doesn't work.

Issue by havok2063
Thursday Oct 13, 2016 at 17:14 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/81

Sometimes the logging breaks with this error on many different occasions. Just noting here to fix at some point.

Traceback (most recent call last):
  File "/Users/Brian/anaconda2/lib/python2.7/logging/__init__.py", line 853, in emit
    msg = self.format(record)
  File "/Users/Brian/anaconda2/lib/python2.7/logging/__init__.py", line 726, in format
    return fmt.format(record)
  File "/Users/Brian/Work/github_projects/Marvin/python/marvin/extern/marvin_brain/python/brain/core/logger.py", line 84, in format
    result = logging.Formatter.format(self, record)
  File "/Users/Brian/anaconda2/lib/python2.7/logging/__init__.py", line 468, in format
    s = self._fmt % record.__dict__
KeyError: 'origin'
Logged from file logger.py, line 127

Issue by albireox
Thursday Jul 07, 2016 at 06:59 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/9

Triggered by #1, should we start considering compacting all the ModelClasses into Marvin (or some other product)? This would probably remove the dependence on SPM.

Issue by bretthandrews
Thursday Jun 30, 2016 at 16:13 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/1

The keyword sample.nsa_logmstar is available to query on, but doesn't work.

Issue by albireox
Thursday Jul 07, 2016 at 07:30 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/12

When I try to download a Cube or Maps file using Marvin tools I get a message such as

rsync: failed to open files-from file /tmp/sdss_access/20160616_004/sdss_access_01.txt: No such file or directory
rsync error: syntax or usage error (code 1) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(1333) [client=2.6.9]

This may be a problem with my rsync version, as other people do not seem to have this problem.

Issue by albireox
Thursday Jul 07, 2016 at 06:52 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/7

Add plot methods to display an spectrum (allow for different axis units, scales, and ranges) and a slice of a cube (in any of the three axis). For spectral ranges, the cube should be collapsable using mean, median, sum, std, etc. (this should probably coded as an independent method).

Issue by albireox
Thursday Jul 07, 2016 at 06:46 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/5

Brian:

We should add convenience properties into the Nsa ModelClasses?. For starters, a good list could include broadband colors between FNugriz, log of stellar mass and similar quantities, and magnitudes of all the fluxes.

Here are examples for how to define something on both the class and instance side, such that the new property can be returned as a parameter, and used in a query filter for the postgresql query.
plate-ifu property in Cube modelclass

    @hybrid_property
    def plateifu(self):
        '''Returns parameter plate-ifu'''
        return '{0}-{1}'.format(self.plate, self.ifu.name)

    @plateifu.expression
    def plateifu(cls):
        return func.concat(Cube.plate, '-', IFUDesign.name)

old version of log stellar mass from the deprecated Sample modelclass. There might be a better way to do this to account for NULL and -9999.0 values of the stellar mass on the SQL side.

    @hybrid_property
    def nsa_logmstar(self):
        try:
            return math.log10(self.nsa_mstar)
        except ValueError:
            return -9999.0
        except TypeError:
            return None

    @nsa_logmstar.expression
    def nsa_logmstar(cls):
        return func.log(cls.nsa_mstar)

Issue by albireox
Thursday Jul 07, 2016 at 06:54 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/8

Input parameters into the SQLalchemy query object must go in the right order, to ensure no crashes when other ModelClasses? are joined to it. Example case that crashes

session.query(datadb.Sample.nsa_redshift, datadb.Cube.plate).join(datadb.PipelineInfo,datadb.PipelineVersion).filter(datadb.PipelineVersion.version=='v1_5_1')

crashes with

InvalidRequestError: Could not find a FROM clause to join from.  Tried joining to <class 'sdss.internal.database.utah.mangadb.DataModelClasses.PipelineInfo'>, but got: Can't find any foreign key relationships between 'sample' and 'pipeline_info'.

Switching the order to Cube, then Sample will result in a successful join.

The instrumentedAttribute list is produced by Query._create_query_modelclasses, using the MarvinForm? _param_to_form_lookup.MapToColumn? method. The resulting list is passed into Query._createBaseQuery

Issue by albireox
Thursday Jul 07, 2016 at 06:43 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/4

Reported by Brian:

Not sure if there is/was a related ticket for this already. The hybrid property plateifu is causing issues. I suspect this will be generic for all added properties.

On the instance side, plateifu is a string. On the class side, plateifu is the concat function. This means that when cube.plateifu is requested as a query return parameter, the param_to_form_lookup.mapToColumn method returns a concat function and not a ModelClass? attribute.

So when we create the join tables out of these parameters at line 276 in join_tables, it crashes but plateifu concat does not have the .class property.

AttributeError: Neither 'concat' object nor 'Comparator' object has an attribute 'class_'

For all of our hybrid properties, and perhaps normal added properties, on the ModelClasses?, how will we generate the ModelClass? to pass into the joins?

Assigning to you José, since it's related to how we are handling parameters in the _param_to_form_lookup, plus the query parameters, plus how we create the list of Join ModelClasses?, but really just opening this up for discussion.

Things will continue to work as long as we don't query on any of these parameters, and plateifu is a doozy of a parameter.

Issue by albireox
Thursday Jul 07, 2016 at 06:43 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/4

Reported by Brian:

Not sure if there is/was a related ticket for this already. The hybrid property plateifu is causing issues. I suspect this will be generic for all added properties.

On the instance side, plateifu is a string. On the class side, plateifu is the concat function. This means that when cube.plateifu is requested as a query return parameter, the param_to_form_lookup.mapToColumn method returns a concat function and not a ModelClass? attribute.

So when we create the join tables out of these parameters at line 276 in join_tables, it crashes but plateifu concat does not have the .class property.

AttributeError: Neither 'concat' object nor 'Comparator' object has an attribute 'class_'

For all of our hybrid properties, and perhaps normal added properties, on the ModelClasses?, how will we generate the ModelClass? to pass into the joins?

Assigning to you José, since it's related to how we are handling parameters in the _param_to_form_lookup, plus the query parameters, plus how we create the list of Join ModelClasses?, but really just opening this up for discussion.

Things will continue to work as long as we don't query on any of these parameters, and plateifu is a doozy of a parameter.

Issue by albireox
Thursday Oct 06, 2016 at 21:47 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/67

As we approach the beta release, here is a list of things that need to happen before we are ready to release:

• - While we wait for the SDSS GitHub repo to be ready: [this step can be skipped if the repo is available soon]
  • - Make marvin and marvin_brain public.
  • - Using SubGit, sync sdss_access and tree and make them repos in marvin-manga. Make them public.
  • - Make marvin_refactor the master branch. This will require tagging the current marvin/trunk as the last marvin 1, plus a number of module-related things in the sas-vm.
  • - Move sqlalchemy-boolean-search and wtforms-alchemy to marvin-manga.
  • - Add tree, sdss_access, sqlalchemy-boolean-search, and wtforms-alchemy as external packages to marvin, using submodules.
  • - Make sdss_access work without modules by defining the necessary environment variables that tree and Path need when marvin gets imported. Check that sdss_sync works.
  • - Create setup.py for marvin. Setup Travis-CI and Coveralls.
  • - Make marvin pip-installable.
  • - Add pip-installable dependencies to setup.py.
  • - Tests in empty VM.
• - Move all repos to the SDSS GitHub team, and make them public. Make sure all previous steps still work.
• - Check Python 3 compatibility.
• - Update documentation.
  • - Add a What's new? and Known Issues. Reference them on the Marvin Web initial page.
  • - Update tutorials.
• - Tag a beta version.
• - Put beta release on Zenodo. Add Zenodo DOI and Bibtex entry on Marvin Docs front page
• - Check citations and acknowledgement instructions are ok in the Docs
• - Update sas-vm nginx server
  • - Keep Marvin 1 in one server pointing to the latest tag of Marvin 1.
  • - Marvin 2 to point to the beta tag.
  • - Add another nginx server running the master branch of Marvin 2.
• - Celebrate.

Issue by albireox
Thursday Sep 08, 2016 at 01:43 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/32

This has probably been discussed before, but right now it's almost possible to use the web locally without internet connection if you have a local DB. The only exception seems to be the preimaging for the point-and-click feature, which I think is downloaded from the SAS.

At ~ 200 Kb per image, loading the preimaging for 10k galaxies in the DB means about 2GB in the final release. It's not a huge amount of space, all things considered. Not sure if it's worth, though.

Issue by havok2063
Thursday Jul 07, 2016 at 14:33 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/16

Marvin query should use WTForm validation where it can. Move to natural language sql string makes this a bit difficult, especially since using WTForm-Alchemy auto generates the forms from sqlalchemy modelclasses.

Issue by albireox
Thursday Jul 07, 2016 at 07:15 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/10

This is probably part of a wider discussion about coordinate origin of reference, but right now there is an inconsistency between the coordinates shown in the __repr__ of Spaxel (which shows lower coordinates) and the default for getSpaxel, which is the centre of the Cube. Should we make the __repr__ match the centre of cube coords?

Issue by bretthandrews
Thursday Jun 30, 2016 at 16:13 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/1

The keyword sample.nsa_logmstar is available to query on, but doesn't work.

Issue by albireox
Tuesday Oct 04, 2016 at 20:54 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/55

During importing and while doing queries, Marvin outputs a good deal of warnings, debug print statements and other ways of verbosity. Before the beta release we should clean up this a bit and leave only the ones that are necessary. We can use the logging system to handle debug messages.

Issue by albireox
Wednesday Jul 06, 2016 at 18:07 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/2

For code reutilisation, let's change all my import matplotlib.pyplot to import matplotlib.pyplot as plt.

Issue by havok2063
Monday Oct 24, 2016 at 07:13 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/85

From a Marvin Cube, we should be able to get a list of maps associated with that cube. Either that are able to be loaded (locally from files or db), or potential maps (list all)? Often I forget what maps I have and I want to quickly find out from a Marvin Cube.

Issue by albireox
Sunday Oct 09, 2016 at 20:22 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/79

Now that marvin uses submodules and marvin_brain, sqlalchemy-boolean-search, and wtforms-alchemy are expected to live in extern/, checking out new versions of marvin and updating them at Utah becomes a bit more complicated. SubGit does not pushes the submodules, it just ignores them.

For now I have made a patchwork, manually cloning and linking versions at Utah, but this is not a sustainable solution. Some alternatives are:

• Stop using the SVN version altogether and clone the marvin repo, with its submodules. This can be done in /home/manga/software/svn.sdss.org/repo/manga/marvin or in a different directory reserved for git products. We can still clone each branch/tag to a different subdirectory so that modules will work with them. This seems like the best option.
• Keep the SVN and the submodules. For normal users that pip install marvin, marvin would use the submodules. For people who check out the SVN version (Utah, basically) would not have anything in the extern directory. Marvin would, in that case, default to using globally ignore versions. This would require changing the imports for brain, SABS, sdss_access, and wtforms-alchemy. Probably the easier way would be to add the paths in extern to the sys.path but with lower priority than the global versions, if they exists.

Issue by albireox
Thursday Jul 07, 2016 at 06:39 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/3

Implement this in Cube or in Spaxel.

Issue by albireox
Thursday Aug 11, 2016 at 23:50 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/27

Currently when one initialises a Cube or Maps it requires defining filename=, plateifu=, or mangaid=. If the keyword is ignored, there should be enough information in the input string for Marvin to decide whether it's a file path, a mangaid, or a plateifu and act accordingly.

Issue by albireox
Thursday Jul 07, 2016 at 06:50 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/6

Implement this functionality in Cube. It should be relatively simple now that getSpectrum is working.

Issue by havok2063
Tuesday Oct 04, 2016 at 20:57 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/56

API routes that crash on the server side return with a 500 status code, internal server error. We should add more detail here.

Issue by albireox
Tuesday Aug 09, 2016 at 19:11 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/23

As commented on a telecon with David and Kyle today, there are files that we may want to be able to access local and remotely via Marvin but that don't want to load in the DB (e.g., the DAP logcubes), as they would increase its size significantly and we don't plan on querying them.

To access those files remotely we would need to change how the local mode works so that, if there is a DB but it does not contain the data the user is requesting, it tries to use the files before failing.

Issue by albireox
Wednesday Jul 06, 2016 at 18:07 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/2

For code reutilisation, let's change all my import matplotlib.pyplot to import matplotlib.pyplot as plt.

Issue by havok2063
Friday Oct 07, 2016 at 18:59 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/71

We should better control for timeouts with the API calls to the server. And on the web. We can implement timeouts in requests, which gives us a bit more control on when and what we return to the user. Long term we need to assess why the server (nginx, postgres, flask) is timing out on queries and web page loads.

We have already adjusted some timeout on the Nginx side per this page
nginx timeouts
and changed
proxy_read_timeout = 300s (5 min)

Issue by havok2063
Saturday Oct 08, 2016 at 19:08 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/74

The Marvin Map tool should use the mask and inverse variance when .plot() is called to properly display the values, given some default setup.

Issue by albireox
Thursday Jul 07, 2016 at 06:54 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/8

Input parameters into the SQLalchemy query object must go in the right order, to ensure no crashes when other ModelClasses? are joined to it. Example case that crashes

session.query(datadb.Sample.nsa_redshift, datadb.Cube.plate).join(datadb.PipelineInfo,datadb.PipelineVersion).filter(datadb.PipelineVersion.version=='v1_5_1')

crashes with

InvalidRequestError: Could not find a FROM clause to join from.  Tried joining to <class 'sdss.internal.database.utah.mangadb.DataModelClasses.PipelineInfo'>, but got: Can't find any foreign key relationships between 'sample' and 'pipeline_info'.

Switching the order to Cube, then Sample will result in a successful join.

The instrumentedAttribute list is produced by Query._create_query_modelclasses, using the MarvinForm? _param_to_form_lookup.MapToColumn? method. The resulting list is passed into Query._createBaseQuery

Issue by albireox
Thursday Jul 07, 2016 at 06:46 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/5

Brian:

We should add convenience properties into the Nsa ModelClasses?. For starters, a good list could include broadband colors between FNugriz, log of stellar mass and similar quantities, and magnitudes of all the fluxes.

Here are examples for how to define something on both the class and instance side, such that the new property can be returned as a parameter, and used in a query filter for the postgresql query.
plate-ifu property in Cube modelclass

    @hybrid_property
    def plateifu(self):
        '''Returns parameter plate-ifu'''
        return '{0}-{1}'.format(self.plate, self.ifu.name)

    @plateifu.expression
    def plateifu(cls):
        return func.concat(Cube.plate, '-', IFUDesign.name)

old version of log stellar mass from the deprecated Sample modelclass. There might be a better way to do this to account for NULL and -9999.0 values of the stellar mass on the SQL side.

    @hybrid_property
    def nsa_logmstar(self):
        try:
            return math.log10(self.nsa_mstar)
        except ValueError:
            return -9999.0
        except TypeError:
            return None

    @nsa_logmstar.expression
    def nsa_logmstar(cls):
        return func.log(cls.nsa_mstar)

Issue by albireox
Saturday Aug 13, 2016 at 18:45 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/29

We should add a LaTeX-ready citation text to the documentation and, eventually, something like this for a potential Marvin paper.

Issue by havok2063
Thursday Jul 07, 2016 at 14:38 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/17

We should make a core Image class to handle manipulation of images, both the optical SAS images and MAP images. The Map class can subclass from this as well to benefit.

Should the current image utilities be migrated into this class?

Should we move the bundle.py and manga_cutouts.py from sdss_python_module into Marvin?

Issue by albireox
Thursday Jul 07, 2016 at 06:59 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/9

Triggered by #1, should we start considering compacting all the ModelClasses into Marvin (or some other product)? This would probably remove the dependence on SPM.

Issue by albireox
Thursday Jul 07, 2016 at 06:50 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/6

Implement this functionality in Cube. It should be relatively simple now that getSpectrum is working.

Issue by havok2063
Friday Oct 07, 2016 at 18:54 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/70

Marvin API calls needs authentication. We can use the authentication in requests to do this. The short term basic solution to implement the netrc approach with the standard sdss SAS authentication. Long term is to implement a session key using requests-oauth, and OAuth1 library, which is common for most modern APIs. Also move the login to Trac authentication, to track who is using marvin.

Issue by havok2063
Thursday Jul 07, 2016 at 14:27 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/13

José

If there is not DB one still gets this message

WARNING: DB connection failed with error: Error connecting to manga database: 'NoneType'
         object has no attribute 'query' Full traceback:   File "/Users/albireo/
         Documents/Repo/manga/marvin/python/marvin/utils/db/dbutils.py", line
         32, in testDbConnection     tmp =
         session.query(marvin.marvindb.datadb.PipelineVersion).first() .

because self.dapdb = None in marvindb.py. There is probably a cleaner way to do that check.

Issue by albireox
Thursday Jul 07, 2016 at 07:28 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/11

Add instruction on how to install iPython and recommend to use it.

Issue by albireox
Wednesday Jul 06, 2016 at 18:07 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/2

For code reutilisation, let's change all my import matplotlib.pyplot to import matplotlib.pyplot as plt.

Issue by havok2063
Thursday Jul 07, 2016 at 14:29 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/14

José

There may be a way of doing this, but so far I cannot see it. Right now I think the only way to disable the local DB and force Marvin to use the local files or the API is to set marvin.marvindb.session = None. This is not very self-evident. We should implement some switch to disable the DB easily. This is important for tests but also in case one has a partial copy of the DB.

Issue by albireox
Thursday Jul 07, 2016 at 06:54 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/8

Input parameters into the SQLalchemy query object must go in the right order, to ensure no crashes when other ModelClasses? are joined to it. Example case that crashes

session.query(datadb.Sample.nsa_redshift, datadb.Cube.plate).join(datadb.PipelineInfo,datadb.PipelineVersion).filter(datadb.PipelineVersion.version=='v1_5_1')

crashes with

InvalidRequestError: Could not find a FROM clause to join from.  Tried joining to <class 'sdss.internal.database.utah.mangadb.DataModelClasses.PipelineInfo'>, but got: Can't find any foreign key relationships between 'sample' and 'pipeline_info'.

Switching the order to Cube, then Sample will result in a successful join.

The instrumentedAttribute list is produced by Query._create_query_modelclasses, using the MarvinForm? _param_to_form_lookup.MapToColumn? method. The resulting list is passed into Query._createBaseQuery

Issue by havok2063
Monday Oct 24, 2016 at 07:08 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/84

Maps initialized from a cube with data_origin=file, should also try to first initialize from a file if it exists using the defaults for anything not specified. I think it can be a bit smarter on its initialization steps. I think this is ultimately related to #27 .

from marvin.tools.cube import Cube
filename ='/Users/Brian/Work/Manga/redux/v2_0_1/8485/stack/manga-8485-1901-LOGCUBE.fits.gz'
cube = Cube(filename=filename)
print(cube)  # returns data-origin=file
maps = cube.getMaps()
print(maps) # return data-origin=db (or remote)

Cube is a file. I have the corresponding maps file. With no input kwargs, I naively expect the above to produce a maps with data_origin=file if it is found. Otherwise try remote.

Issue by albireox
Thursday Sep 22, 2016 at 05:13 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/47

Currently RSS works, but its use is limited. To be really useful it needs to get some work done. At least, we should consider doing this:

• Lazy loading. Right now, when a RSS object is instantiated all its fibres are loaded. This is ok if the RSS is loaded from a file, not so fast if doing so from the DB, and quite likely not good if loading from the API. I haven't tried it with a large IFU, but with a 19-fibre one loading from DB takes several seconds. Ideally we want to create the RSSFiber objects, but without unpacking the flux/ivar/... information until it's accessed. This means not using Spectrum but some ad-hoc class.
• Load all the observing info, dispersion, specred, and x and y information. Right now only flux, ivar, mask, and wavelength are available.
• Load fiberid and allow to retrieve a fibre from the RSS based on fiberid. This is trivial from the DB, but not so much from a file. I think we'd need to include the metrology Yanny files in Marvin, although there may be another way of doing this that I don't know. Probably we also want to provide information about the fibre (harness, xpmm, ypmm, etc.)

Issue by albireox
Thursday Jul 07, 2016 at 06:52 GMT
Originally opened as https://github.com/marvin-manga/marvin/issues/7

Add plot methods to display an spectrum (allow for different axis units, scales, and ranges) and a slice of a cube (in any of the three axis). For spectral ranges, the cube should be collapsable using mean, median, sum, std, etc. (this should probably coded as an independent method).