aburgasser / splat Goto Github PK

Need a way to get list of available info from spectrum object such as nir_type, opt_type, ra, dec, etc.

Hello, I am trying to read in a bunch of L8 spectra however when I type the code below I get an error

import splat
specs = splat.getSpectrum(spt="L8.0")

A ValueError is raised and says this for me

C:\Users\Kyle\Programs\browndwarf\splat\splat\core.py in __init__(self, *args, **kwargs)
    241
    242 # populate information on source and spectrum from database
--> 243         if sdb != False:
    244             for k in sdb.keys():
    245                 setattr(self,k.lower(),str(sdb[k][0]))

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()

I think the issue lies with a duplicate entry for 10056 in source_data.txt. I can't seem to delete the duplicate entry for 10056 and have it read the table in properly

it looks like searchlibrary is doing a search on "shortname" even if that isn't set in database

splat.classifyByTemplate(sp,spt=['M7','L5'])

KeyError Traceback (most recent call last)
in ()
----> 1 splat.classifyByTemplate(sp,spt=['M7','L5'])

/Users/adam/projects/splat/code/splat/splat/core.py in classifyByTemplate(sp, *args, **kwargs)
2872 lib = searchLibrary(excludefile=excludefile,excludekey=excludekey,excludeshortname=excludeshortname,
2873 snr=snr,spt_type=spt_type,spt=spt,published=published,
-> 2874 giant=giant,companion=companion,young=young,binary=binary,spbinary=spbinary,output='all',logic='and')
2875
2876 # first search for the spectra desired - parameters are set by user

/Users/adam/projects/splat/code/splat/splat/core.py in searchLibrary(*args, **kwargs)
1697 sn = 'J'+sn
1698 # t = numpy.sum(source_db['SELECT'][numpy.where(source_db['SHORTNAME'] != sn)])
-> 1699 source_db['SELECT'][numpy.where(source_db['SHORTNAME'] != sn)] += 1
1700 # if numpy.sum(source_db['SELECT'][numpy.where(source_db['SHORTNAME'] != sn)]) > t:
1701 # print('rejected '+sn)

/Users/adam/anaconda3/lib/python3.5/site-packages/astropy/table/table.py in getitem(self, item)
1157 def getitem(self, item):
1158 if isinstance(item, six.string_types):
-> 1159 return self.columns[item]
1160 elif isinstance(item, (int, np.integer)):
1161 return self.Row(self, item)

/Users/adam/anaconda3/lib/python3.5/site-packages/astropy/table/table.py in getitem(self, item)
96 """
97 if isinstance(item, six.string_types):
---> 98 return OrderedDict.getitem(self, item)
99 elif isinstance(item, (int, np.integer)):
100 return self.values()[item]

KeyError: 'SHORTNAME'

stripping out header info and producing the spectrum db entry for a new spectrum

In [0]: sp1[0][0], absJmag1[0][0]
Out [0]: (SPEX_PRISM spectrum of SDSS J233224.38-005025, 15.31086633016845)
In [1]: print(sp1[0][0].fluxCalibrate('2MASS J',absJmag1[0][0],absolute=True))
Out [1]: None

something like Spectrum.save(filename), sensing whether a fits or ascii file is wanted

computer seems to have gone dark - communication issue at UCSD, or my machine?

keep the ascii versions but add sql versions for easy access

Right now the model reading takes about 0.4 seconds, which is fine but adds up for interpolated models. Suggest creating a global variable that contains N_param * 2 model spectrum objects that contain the corners of the most recent interpolated model call; if the next parameters are within that hypercube, just reinterpolate rather than read in

11335_10505.fits
11364_10806.fits
11181_10187.fits
10823_11422.fits
12004_10444.fits
10829_10104.fits
11182_10188.fits
10822_11283.fits
10824_11423.fits
10821_11058.fits
10107_10315.fits
11072_11527.fits
10600_10957.fits
10592_11111.fits
10675_11572.fits
10351_10583.fits
10375_10696.fits
10678_10105.fits
10115_11254.fits
10771_10871.fits
10767_10591.fits
10017_10945.fits
10034_10874.fits
10021_11106.fits
10200_11236.fits
10159_10513.fits
10126_10349.fits
11536_10509.fits

From Victor Garcia:

Ah right also (a possible?) bug report. It appears that the following code:

splat> splist = splat.getSpectrum(spt=['M0','M1'])

Retrieving 14 files

splat> print splist
[Spectrum of 2MASS J03444306+3137338, Spectrum of 2MASS J05363776+1000232, Spectrum of LHS 217, Spectrum of V838 Mon, Spectrum of LSPM J0734+5810, Spectrum of 2MASS J13032137+2351110, Spectrum of WISE J130740.45-463035.1, Spectrum of WISE J150711.06-344026.0, Spectrum of WISE J152915.47-451348.2, Spectrum of PM J17137-4535, Spectrum of 2MASS J17252029-0024508, Spectrum of 2MASS J18112466+3748513, Spectrum of WISE J183921.35-374431.0, Spectrum of V Z Cep]

Returns the spectrum of 2MASS J17252029-0024508, which SIMBAD says is an M5V (link below), but perhaps SIMBAD is wrong, and SpeX spectral type is better?

http://simbad.u-strasbg.fr/simbad/sim-basic?Ident=2MASS+J17252029-0024508&submit=SIMBAD+search

the check step for inserting spectra should included a pdf of all the uploaded spectra (those not already in the database), annotated with either the closest spectral standard or the repeat observation in the database (and some relevant information like spectral type, S/N, source name, etc.)

Line: splat.modelParameters(temperature=2821.301,age=5)

Error:

AttributeError Traceback (most recent call last)
in ()
----> 1 splat.modelParameters(temperature=2821.301, age=5)

AttributeError: 'module' object has no attribute 'modelParameters'

the fits filenames could be more intuitive (e.g., with source name, date, instrument), and headers need to be complete, also spectrum should be normalized to apparent magnitudes - this will require a full rework of how files are accessed

I downloaded the new version of splat but when I try to import splat, the following error is displayed:

**File "splat__init__.py", line 1, in
from splat import *

File "splat\splat.py", line 94, in
DB_SOURCES = fetchDatabase(DB_SOURCES_FILE)

File "splat\splat_db.py", line 343, in fetchDatabase
raise NameError('\nCould not find '+kwargs['folder']+' locally or on SPLAT website\n\n')

NameError:
Could not find /db/ locally or on SPLAT website**

It seems like /db/ folder can't be located even if it's clearly downloaded

classification by standard returns weird spectral type - returning average instead of best, would be best to do best

specfolder = '/Users/daniella/idl/spexbinaryfit/all/'
spc = splat.Spectrum(specfolder+'spex_prism_0001-0942W_121027.fits')
splat.classifyByStandard(spc,plot=True)

UnicodeDecodeError Traceback (most recent call last)
in ()
1 specfolder = '/Users/daniella/idl/spexbinaryfit/all/'
----> 2 spc = splat.Spectrum(specfolder+'spex_prism_0001-0942W_121027.fits')
3 splat.classifyByStandard(spc,plot=True)

/Users/daniella/Python/splat/splat.py in init(self, _args, *_kwargs)
372 sdb = keySpectrum(self.idkey)
373 if sdb != False:
--> 374 self.filename = sdb['DATA_FILE'][0]
375 elif self.model == False and self.filename != '':
376 kwargs['filename']=self.filename

/Users/daniella/Python/splat/splat.py in searchLibrary(_args, *_kwargs)
3287 L7.0IIp:
3288 >>> print splat.typeToNum(50)
-> 3289 Spectral type number must be between 0 (K0) and 49.0 (Y9)
3290 nan
3291 '''

/Users/daniella/miniconda3/lib/python3.5/site-packages/astropy/io/ascii/ui.py in read(table, guess, **kwargs)
285 try:
286 with get_readable_fileobj(table) as fileobj:
--> 287 table = fileobj.read()
288 except ValueError: # unreadable or invalid binary file
289 raise

/Users/daniella/miniconda3/lib/python3.5/codecs.py in decode(self, input, final)
319 # decode input (taking the buffer into account)
320 data = self.buffer + input
--> 321 (result, consumed) = self._buffer_decode(data, self.errors, final)
322 # keep undecoded input until the next call
323 self.buffer = data[consumed:]

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 57291: invalid continuation byte

loading 30 standards takes a while! possibly solution is to make an empty global variable and fill it when classifybystandard is called the first time

Implement emcee to build on the framework (should run faster than the current MCMC implementation)

would be nice to have a separate database that contains the many index - there needs to be a way of easily updating this as new spectra and new indices are added

It appears that there are some odd characters that Python3 doesn't like in the db/source_data.txt file. I receive the following error when doing splat.test():

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf3 in position 57291: invalid continuation byte

We may need to change the encoding of the text file or figure out what character within the file is giving us an error.

need to put up an easy-to-find link to the current documentation

In: spc = splat.getSpectrum(designation='J1632561+3505073')[0]

Out: Retrieving 1 file

In: print(splat.classifyByStandard(spc))

Out:
No spectra in the SPL database match the selection criteria
No spectra in the SPL database match the selection criteria
No spectra in the SPL database match the selection criteria
No spectra in the SPL database match the selection criteria
No spectra in the SPL database match the selection criteria
No spectra in the SPL database match the selection criteria
No spectra in the SPL database match the selection criteria
No spectra in the SPL database match the selection criteria
('L1.0', 0.5)

Need to include a bib file with all of the references in the table, and some kind of "processing" for these references - adding them to tables, providing links to ADS, etc.

Code line:
splat.classifyByTemplate(spc1,string=True,spt=['M5','M9'],force=True,snr=50,nbest=5,plot=True)

Error:

NameError Traceback (most recent call last)
in ()
----> 1 splat.classifyByTemplate(spc1,string=True,spt=['M5','M9'],force=True,snr=50,nbest=5,plot=True)
2 #splat.classifyByTemplate(spc1,force=True,snr=300,nbest=5,plot=True, select=('Mdwarfs'))

/Users/macbookpro/Documents/splat-master/splat.pyc in classifyByTemplate(sp, _args, *_kwargs)
2217 kwargs['legend'] = [sp.name,s.name]
2218 kwargs['colors'] = ['k','r','b']
-> 2219 plotSpectrum(sp,s,sp-s,**kwargs)
2220
2221 # string or not?

/Users/macbookpro/Documents/splat-master/splat_plot.pyc in plotSpectrum(_args, _kwargs)
508 xrange = kwargs.get('xrange',[0.85,2.42])
509 bound = xrange
--> 510 ymax = [s.fluxMax().value for s in sp]
511 yrng = kwargs.get('yrange',map(lambda x: xnumpy.nanmax(ymax)+numpy.nanmax(zeropoint),[-0.02,1.2]))
512 bound.extend(yrng)

/Users/macbookpro/Documents/splat-master/splat.pyc in fluxMax(self, *kwargs)
937
938 return numpy.nanmax(self.flux.value[numpy.where(
--> 939 numpy.logical_and(self.wave > nanmin(self.wave)+0.1(nanmax(self.wave)-nanmin(self.wave)),self.wave < nanmax(self.wave)-0.1_(nanmax(self.wave)-nanmin(self.wave))))])_self.funit
940
941

NameError: global name 'nanmin' is not defined

we're currently missing Y-band filters for SPLAT - need to add (courtesy Victor Garcia)

solved: "range" command behaves differently in python2 and python3; changed all instances of range to numpy.arange

From Adam Schneider:

First — Splat is just plain awesome. I’m only just getting into playing around with what’s available, but already I can tell that it will make my spectroscopic investigations simpler.

One thing I did note, however, is that the model-fitting isn’t set up for a spectrum that isn’t in the database (i.e., an uploaded spectrum). Looks like this is because the ‘splat_model.py’ script attempts to find the ‘shortname’ for any spectrum, which doesn’t automatically exist for an uploaded one.

I got around this by replacing this line:

filebase = kwargs.get('filebase', 'fit_'+spec.shortname+'_'+m_set)

with

try:
    filebase = kwargs.get('filebase', 'fit_'+spec.shortname+'_'+m_set)
except:
    filebase = kwargs.get('filebase', 'fit_'+spec.filename+'_'+m_set)

Maybe there’s a more elegant way — but this seemed to work. Good news is that the Splat model fitting finds a very similar Teff and log g to what I’ve found for an object in a paper I’m preparing. Nice to have some confirmation.

e.g., to set input wavelength to be something other than microns, input flux to be something other than erg/cm2/s/micron, etc.

Here is the error that happens during splat.test()

File "", line 1, in
File "/Users/ctheissen/repos/splat/splat.py", line 3572, in test
grav = classifyGravity(sp)
File "/Users/ctheissen/repos/splat/splat.py", line 1580, in classifyGravity
sptn, spt_e = classifyByIndex(sp,string=False,set='allers')
File "/Users/ctheissen/repos/splat/splat.py", line 1022, in classifyByIndex
indices = dict(i1.items() + i2.items() + i3.items())
TypeError: unsupported operand type(s) for +: 'dict_items' and 'dict_items'

The problem arises in line 1022 where
indices = dict(i1.items() + i2.items() + i3.items())

You cannot add dictionary items in Python3 since they are not lists.

might be useful for old fogies

From Victor Garcia

need a way to check if a specified keyword search parameter is actually in the spectral or source library

for source checking and source db input -> fill in all the source db information

filename= '11731_11594.fits'

need to automatically search through BANYAN7 paper, others?

Interpolations currently happen over all the possible values for Teff, logg, and z for given models. However, if you are looking at, say, low-Teff range where z = 0 and no other value, this should do something rather than try and fit other values of z that exist for higher-Teff models.

Changes to splat.filterInfo:

Says:
WFC3 F167N: WFC3 F160W
Should say:
WFC3 F167N: WFC3 F167N

Says:
NIRC2 LP: WIRC Fe II
Should say:
NIRC2 LP: NIRC2 L'-band

Says:
NIRC2 M: WIRC Fe II
Should say:
NIRC2 M: NIRC2 M-band

Changes to splat.filterProperties:

1. Change description:

Filter: WIRCAM CH4 OFF
Current description: CFHT WIRCAM CH4-on

Filter: WIRCAM CH4 ON
Current description: CFHT WIRCAM CH4-off

Filter: NIRC2 FE2
Current description: WIRC Fe II

2. File not found:

Filter: WFC3 F164N
splat looks for wfc_F164M.txt
True HST WFC3 filter is F164 Narrow.

3. Add zero points (although I couldn't find them online):

Filters:
WIRC BRGAMMA: WIRC H I Brackett Gamma
WIRC CH4L: WIRC CH4L
WIRC CH4S: WIRC CH4S
WIRC CO: WIRC CO
WIRC FE2: WIRC Fe II
WIRC H: WIRC H-cont
WIRC J: WIRC J-cont
WIRC K: WIRC K-cont
WIRC PABETA: WIRC H I Paschen Beta

4. Non-corresponding filter files:

Filter: WFC3 F167N
splat looks for wfc3_F160W.txt

5. Filter not available:

Filter: WFC3 F160W
Prints: WFC3 F160W not among the available filters: + list of filters from splat.filterInfo()

6. “ValueError: Not enough values to unpack (expected 2, got 0)”

All NIRC2 Filters:
NIRC2 J
NIRC2 H
NIRC2 K
NIRC2 KS
NIRC2 KP
NIRC2 KCONT
NIRC2 LP
NIRC2 M

7. “ValueError: too many values to unpack (expected 2)”

All IRAC Filters:
IRAC CH1
IRAC CH2
IRAC CH3
IRAC CH4

currently classifybytemplate has some hard coded "choices" - these should be placed in a kwargs to make the program more responsive to changes in searchlibrary

also, insert a "try" statement on reading in a file in case there is an issue (so whole sequence isn't disrupted)

import splat
splat.test()
Traceback (most recent call last):
File "", line 1, in
AttributeError: module 'splat' has no attribute 'test'

splat.classifyByStandard claims that the spectrum has no noise column, when in reality it does.

specfolder = '/Users/daniella/idl/spexbinaryfit/all/'
spc = splat.readSpectrum(specfolder+'spex_prism_0333+0014_081214.txt')
splat.classifyByStandard(spc,plot=True)

AttributeError Traceback (most recent call last)
in ()
1 specfolder = '/Users/daniella/idl/spexbinaryfit/all/'
2 spc = splat.readSpectrum(specfolder+'spex_prism_0333+0014_081214.txt')
----> 3 splat.classifyByStandard(spc,plot=True)

/Users/daniella/Python/splat/splat.py in classifyByStandard(sp, _args, *_kwargs)
1216
1217 # which comparison statistic to use
-> 1218 if numpy.isnan(numpy.median(sp.noise)):
1219 compstat = 'stddev_norm'
1220 else:

AttributeError: 'dict' object has no attribute 'noise'

sometimes calls to read in an online file don't work, even with access and online = True. seems to be sporadic. will need to figure out some way of addressing this and protecting looped calls in, e.g., classifyByTemplate

eliminate "mean of random" for measureindex and just report a straight measurement (MC only for uncertainty)

There seems to be a problem with reading in autofunction, autoclass, automodule, etc. The generated docs that are uploaded are OK, so not sure why these auto* are not working. need to do some research, may be related with where the programs are located

deciding whether to enter things into database - checking source and spectrum databases, and then making some decisions based on observing dates, keys, etc.