keplergo / pyke Goto Github PK
View Code? Open in Web Editor NEWEasy command line tools for Kepler, K2 & TESS data analysis.
Home Page: http://pyke.keplerscience.org
Easy command line tools for Kepler, K2 & TESS data analysis.
Home Page: http://pyke.keplerscience.org
ploterr
is not being used in kepdraw
.
plotgrid
was fixed in 311d745.
Thanks to @mirca's incredibly hard work, the master branch of PyKE is now compatible with Python 3 and no longer depends on PyRAF (cf #12), which were the key goals for "PyKe v3.0" aka "PyKE3". This means we can now start planning the first official release of PyKE3!
Let's start by releasing a "3.0.beta" and announce it to a small audience on social media, followed by an official "3.0.0" a few weeks later announced on the website.
Tasks to complete ahead of the release are likely to include the following:
kepdraw
tool [#18]kepsff
tool [#19]kepmask
+ kepextract
+ kepflatten
+ kepsff
+ kepdraw
(cf. https://keplerscience.arc.nasa.gov/PyKEprimerWalkthroughE.shtml)pyke
, pykep
, pykepler
are all taken by others). Let's use pyketools
?This warning
WARNING: VerifyWarning: Invalid keyword for column 4: Column null option (TNULLn) is invalid for binary table columns of type 'E' (got -1). The invalid value will be ignored for the purpose of formatting the data in this column. [astropy.io.fits.column]
appears all the time when using kepdraw
. I guess we should suppress it somehow.
thoughts? @barentsen
Hi,
I'm new to astronomy analysis, so maybe my need is fulfilled with another software, please let me know ๐
I would like to give a try analyzing Kepler time series with Warp10, an open source time series database that has huge analysis capabilities. For this, I need to export Kepler FITS time series to the Warp10 Warp10 Input Format.
I saw that there is a task called kepconvert, should I add the option there or create a new task for TS export? Other people might want to export to another standard.
It would be nice if the first release of PyKE3 can be used to create a detrended K2 lightcurve from the pixel data, i.e. using the steps explained in the original "Walkthrough E" at https://keplerscience.arc.nasa.gov/PyKEprimerWalkthroughE.shtml
The main thing missing to enable this is the kepsff
task. How much time would it cost to port this task to PyKE 3? @mrtommyb seems to think there is no difference between the code in martinsff
found in his Trappist-1 repo, and the original kepsff
that came with PyKE.
Geert's idea.
A possible implementation is already here: https://gist.github.com/barentsen/9b5a7b51f48e026e3af5
rtfd doesn't like our utils
module
ImportError: No module named 'pyke.utils'
As a consequence, numpydocs are not being processed appropriately.
I'm investigating.
In line with astropy, let's use --overwrite
instead of --clobber
in all commands and functions. It's a lot less obscure. I certainly could never remember the word clobber!
I was working on this one: ktwo220142950-c08_spd-targ.fits, when kepconvert failed with the following:
Exception: ERROR -- KEPCONVERT: error converting time to ktwo220142950-c08_spd-targ.lc.txt: format must be one of ['jd', 'mjd', 'decimalyear', 'unix', 'cxcsec', 'gps', 'plot_date', 'datetime', 'iso', 'isot', 'yday', 'fits', 'byear', 'jyear', 'byear_str', 'jyear_str']
Apparently units are "BJD-2454833" in the light curve file generated by kepextract instead of the ones above.
In a future version (>3.0), we should add a tool to overlay or compare the region of sky covered by a Target Pixel File against survey images (DSS, SDSS, 2MASS...). The use case is to enable a user to get a high-resolution view to e.g. check for blended stars.
This tool has been requested by several people, and an example implementation is available at https://github.com/stephtdouglas/k2-pix
We could call it kepsky
or similar.
Tutorial call to kepprfphot requires a directory of prf files. The tutorial needs to give instructions for downloading the files.
$ kepprfphot ktwo200071160-c91_lpd-targ-trimmed_3.fits ktwo200071160-c91_trimmed_3_prfphot --columns 1013 --rows 918 --fluxes 18000 --prfdir ~/kplr2011265_prf --background --clobber --verbose
The exception the routine throws when you try without it is also unhelpful:
KEPPRFPHOT started at: Thu Jun 29 16:51:12 2017
ERROR -- KEPIO.OPENFITS: cannot open ktwo200071160-c91_lpd-targ-trimmed_3.fits as a FITS file
ERROR -- KEPPRFPHOT: is ktwo200071160-c91_lpd-targ-trimmed_3.fits a Target Pixel File?
Traceback (most recent call last):
File "/usr/local/bin/kepprfphot", line 11, in
load_entry_point('pyke', 'console_scripts', 'kepprfphot')()
File "/Users/christina/K2/PyKE/pyke/kepprfphot.py", line 1157, in kepprfphot_main
args.overwrite, args.verbose, args.logfile)
File "/Users/christina/K2/PyKE/pyke/kepprfphot.py", line 244, in kepprfphot
kepmsg.err(logfile,message,verbose)
File "/Users/christina/K2/PyKE/pyke/kepmsg.py", line 16, in err
raise Exception(message)
Exception: ERROR -- KEPPRFPHOT: is ktwo200071160-c91_lpd-targ-trimmed_3.fits a Target Pixel File?
Right now it can only stitch together lightcurves, but we may want to look into stitching together pixels because of e.g. the K2 Campaigns that were split up in halves or revisited.
This is low priority.
kepconvert is supposed to be able to convert fits to ascii and ascii to fits. The fits to ascii works fine, but when you try to go the other way there is an error message.
kepconvert('kplr008462852-2011073133259_llc.fits','dip.ascii',overwrite=True,conversion='fits2asc', columns='TIME,SAP_FLUX,SAP_FLUX_ERR')
kepconvert('dip.ascii','kplr008462852-Q8.fits',overwrite=True,conversion='asc2fits',columns='TIME,SAP_FLUX,SEP_FLUX_ERR',verbose=True)
Produces the error
ERROR -- KEPCONVERT: cannot create light curve extension in kplr008462852-Q8.fits
Hi,
Please have a look at this part of PER2DET:
https://github.com/KeplerGO/PyKE/blob/master/pyke/kepfunc.py#L298#L313
All these lines are equivalent to:
for (j, y) in enumerate(DATy):
for (k, x) in enumerate(DATx):
xx = x - OBJx[i]
yy = y - OBJy[i]
Does anybody know why there are some many useless calculations? Is anybody sure that this code is correct?
CC: @barentsen @mirca
Hi @barentsen and @mirca,
It seems for me there's a bug in PRF2DET() function of kepfunc.py file. The second last line says:
PRFfit[j, k] += PRFfit[j,k] + splineInterpolation(dy * wy, dx * wx) * flux[i]
It like "x += x + dx". Previous function in the same file has:
PRFfit[j, k] = PRFfit[j, k] + splineInterpolation(dy, dx) * flux[i]
which looks normal. Can you comment on this?
The current PyKE doc lives in https://keplerscience.arc.nasa.gov/PyKE.shtml
The goal is to have it on sphinx (with a beautiful, elegant, minimalist theme).
I'm working on this also in #12
We may want to consider enforcing e.g. TkAgg as the backend.
Where we use data in a tutorial, let's make sure we provide an easy download link every time.
From [email protected]:
On Mon, Jun 5, 2017 at 8:53 AM, Mike Reed [email protected] wrote:
Hi Geert, Any update on this? Our 'big' issue is that kepextract used to work on a single pixel, but at some point there was an update so it now requires 7 pixels. Our kepmask file used to look like b5|||851|139|4,4, but now we have to list 7 pixels, which is too large for this crowded field, and not necessary for original Kepler data. We want to extract single pixels, find which pixels really have the star (from SC data, we know that it is only 2-3 pixels for each rotation) and then extract only those pixels.
When trying out the K2 aperture photometry tutorial I encounter the following exception:
$ kepsff ktwo202073445-c00_lpd-targ-kepextract-kepclip-kepflatten.fits --datacol DETSAP_FLUX --stepsize 5. --npoly_ardx 4 --sigma_dsdt 10. --overwrite
0%| | 0/7 [00:00<?, ?it/s]
Traceback (most recent call last):
File "/home/gb/bin/anaconda/bin/kepsff", line 11, in <module>
load_entry_point('pyketools', 'console_scripts', 'kepsff')()
File "/home/gb/dev/PyKE/pyke/kepsff.py", line 774, in kepsff_main
args.logfile)
File "/home/gb/dev/PyKE/pyke/kepsff.py", line 701, in kepsff
instr[1].data.field('SAP_FLUX')[t1:t2] /= cfac
ValueError: operands could not be broadcast together with shapes (244,) (8,) (244,)
PyKE2 included a tool called kepdraw
that was used to see what a lightcurve looks like. It was not ported to PyKE3 as part of #12, but I do think we should consider including this tool in the first release of PyKE3. This is because many users may not be comfortable with making a quick lightcurve plot from scratch, which may keep them from adopting PyKE3.
Hi,
I tried to rescale the TPF figure in kepmask
but manually entering imin and imax values had no effect. This is problematic when the automatic scaling runs into trouble for some TPFs, and need forced imin and imax to display properly (default scaling gives "ValueError: minvalue must be less than or equal to maxvalue"). One example is KTWO229228811-C08.
The default logarithmic scaling stays the same. When I tried with linear or squareroot with manual imin/imax values, it gives the following errors. Linear scaling:
line 171, in kepmask
zmin *= 1.0
TypeError: can't multiply sequence by non-int of type 'float'
Squareroot scaling:
line 168, in kepmask
zmin = math.sqrt(zmin)
TypeError: a float is required
When FLUX values have a negative value, they will display as white in kepmask
because it takes the log10 by default. To fix this, let's add a pedestal to make sure all values are above zero before taking the log.
I propose the following improvements to kepconvert
:
--help
string.Some very minor edits to docs:
install.html
after the make html
instructions, _build
โก๏ธ build
.install.html
directs users to pyke.readthedocs.io
; should it direct to pyke.keplerscience.org
instead?kepextract
docstring, consider: "The pipeline" โก๏ธ "The Kepler pipeline"kepextract
docstring, consider placing "Light curves for extended sources..." on its own line.kepextract
docstring: "one of three options::" โก๏ธ "one of three options:"kepextract
docstring: "pixels in within" -> "pixels within"kepextract
docstring: "warnings to the shell" -> "warnings print to the shell"invalid value encountered in double_scalars
while running kepextracttqdm
jumps a line when warnings appear. Is there an option to detect and enable Jupyter notebook support similar to astropy Progressbar?Aperture photometry...
) even in verbose=False
mode. Should it?Big props on how much faster kepextract is on short-cadence data with PyKE 3.0, thanks for that!
I've found another (less significant) bottleneck: kepclip takes A LOT longer than it should if it's just removing entries in a time series (up to a few minutes on a relatively new MacBook Pro). I wonder if you could investigate why kepclip is so SLOW on short-cadence data. kepstitch runs considerably faster for what should be essentially the same arithmetic. Thanks!
WARNING: VerifyWarning: Invalid keyword for column 4: Column null option (TNULLn) is invalid for binary table columns of type 'E' (got -1). The invalid value will be ignored for the purpose of formatting the data in this column. [astropy.io.fits.column]
. I have not in detail investigated the cause of this issue.It was pointed out to me by a beginning user that, e.g. in the first tutorial ( http://pyke.readthedocs.io/en/latest/tutorials/aperture-photometry-in-k2.html ), it is not clear that the wget
and kepmask
are command-line tools that need to be executed in the shell rather than a Python environment. Let's make sure this is always clear!
We need a convention on how PyKE should deal with printing feedback to the users and writing logfiles. The current system based on verbose
and logfile
arguments is not in line with the logging mechanisms used by major scientific packages. I propose the following:
print
statements anywhere.keplog.log(level, message)
function, which in turn shall make use of either Python's or AstroPy's built-in logging
framework, which support different levels of logging (ERROR
, WARNING
, INFO
, DEBUG
).verbose
arguments shall be removed in favor of the config mechanisms provided by the logging framework. A convenience function called keplog.set_level
can be added to help users set the log level (e.g. keplog.set_level(DEBUG)
). The command-line tools can have a user-friendly --silent
or --debug
option to set the log level to ERROR
or DEBUG
, respectively.logfile
arguments shall be removed in favor of the config mechanisms offered by the logging framework. A convenience function called e.g. keplog.set_logfile
can be added to help users divert output to a logfile.keplog.log(INFO, "Writing " + filename)
.desc
and disable
argument set, i.e.tqdm(iterable, desc="Informative description", disable=(keplog.get_level() <= INFO))
keplog.log(ERROR, msg)
.What do y'all think?
Hi!
I'm currently looking how to convert BJD time to UTC time. I wondering if it could be an option to kepconvert. It'll allow developers to easily import ASCII or CSV files without fearing the conversion in environment without Astropy ๐
function fitfunct_fmin
and the likes require date
as parameter, but don't do anything with it
Let's update our science center website to provide instructions on installing and using PyKE3.
This post intends to capture the list of tutorials we'd like to add to PyKE. Ideally, the tutorials would form a nice continuous story.This list is intended to be revised continuously. To make changes to this list, either comment below or edit this post.
kepmask
and keppixseries
.kepcotrend
?kepflatten
?kepconvert
?kepbls
?keptransit
? (or PyTransit or BATMAN)If you are a pro Windows user that came across this ticket, please feel free to respond and volunteer to help us figure out how to make PyKE accessible in Windows. The Kepler team tends to use OSX and Linux, but is interested in making their tools as widely accessible as possible.
Pypi only supports ReStructuredText, which is why pyketools
on pypi looks silly: https://pypi.python.org/pypi/pyketools
outfile
is currently required by many tasks. Let's help the user by setting a sensible default, e.g. infile + "-taskname.extension"
Minor notes on the CLI when you execute --help
.
kepwindow --help
"Name of input file" is too general. Should say some more concrete information, FITS only, or ascii too?keptrim --help
the command line argument --kepid
refers only to "the Kepler ID number from the Kepler Input Catalog". What about K2 EPIC numbers?kepstitch --help
, "Append multiple month short cadence and/or multiple quarter long cadence data" should have a qualifier that it's for the original Kepler data only. Also: --outfile OUTFILE
is an optional command line argument, which differs from most other CLI behaviors. (I actually prefer the default outfile name strategy.)kepstddev
output an outfile if it just calculates CDPP? Does it modify the header or something?kepsff
, repeated word "for":--npoly_arfl NPOLY_ARFL
Order of ploynomialforfor arclength-flux calibration
(default: 3)
kepsff
, the name --plotres
, does not seem to match the description:Save hardcopies of the plots?
keppca
must require the CBV files, right? This could be noted in the CLI.kephead
: "Snippet of keyword name" could be "full or partial keyword name"kepft
CLI "search period" is a bit misleading, since nothing is being "searched".kepfold
, many of the CLI optional arguments refer to binning properties, which are only needed if binning is selected. Can that be indicated somehow?kepflatten
: the description is wrong:Remove or replace data outliers from a time series
I believe it is important that the --help
strings show the default argument values.
For example, $ kepflatten --help | grep datacol
currently shows:
--datacol DATACOL Name of data column to plot
it would be helpful for users to show the default value, e.g.:
--datacol DATACOL Name of data column to plot (default: SAP_FLUX)
This is not a requirement for the first release however.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.