Code Monkey home page Code Monkey logo

psrqpy's Introduction

PSRQpy

This module aims to provide a python interface for querying the ATNF pulsar catalogue. It is an unofficial package and is not endorsed by or affiliated with the ATNF.

Full documentation of the module can be found here.

Any comments or suggestions are welcome.

Installation

To install the code from source, clone the git repository and run either:

python setup.py install --user

to install as a user, or

sudo python setup.py install

to install as root.

The module can also be installed using pip with:

pip install psrqpy

or in a Conda environment using:

conda install -c conda-forge psrqpy

Requirements

The requirements for installing the code are:

Examples

A simple query of the catalogue to, e.g., just return all pulsar frequencies, would be:

import psrqpy

q = psrqpy.QueryATNF(params='F0')

# get frequencies as an astropy table
t = q.table

print(t['F0'])

You can query multiple parameters, e.g.:

import psrqpy

q = psrqpy.QueryATNF(params=['F0', 'F1', 'RAJ', 'DecJ'])

# get values as an astropy table
t = q.table

print(t['F0'])

You can query specific pulsars, e.g.:

import psrqpy

q = psrqpy.QueryATNF(params=['F0', 'F1', 'RAJ', 'DecJ'], psrs=['J0534+2200', 'J0537-6910'])

# get values as an astropy table
t = q.table

# print the table
print(t)
  JNAME          F0       F0_ERR       F1      F1_ERR     RAJ      RAJ_ERR     DECJ     DECJ_ERR
                 Hz         Hz       1 / s2    1 / s2                                           
---------- ------------- ------- ------------- ------ ------------ ------- ------------ --------
J0534+2200     29.946923   1e-06  -3.77535e-10  2e-15 05:34:31.973   0.005 +22:00:52.06     0.06
J0537-6910 62.0261895958 1.3e-09 -1.992272e-10  4e-17 05:37:47.416    0.11 -69:10:19.88      0.6

You can set conditions for the searches, e.g.:

import psrqpy
q = psrqpy.QueryATNF(params=['Jname', 'f0'], condition='f0 > 100 && f0 < 200', assoc='GC')

where assoc=GC looks for all pulsars in globular clusters.

When a query is generated the entire catalogue is downloaded and stored in the QueryATNF object as a pandas DataFrame. The query can therefore be re-used to access data on different parameters, different pulsars, or using different conditions, without the need to re-download the catalogue. We may originally want to query pulsar frequencies using only frequencies greater than 10 Hz, with

import psrqpy
q = psrqpy.QueryATNF(params=['F0'], condition='F0 > 10')
freqs = q.table['F0']

Using the same QueryATNF object we could change to get frequency derivatives for pulsars with frequencies less than 10 Hz, with

q.condition = 'F0 < 10'
q.query_params = 'F1'

fdot = q.table['F1']

In these cases the whole catalogue (with no conditions applied and all available parameters) stored as a pandas DataFrame is accessible with

catalogue = q.catalogue

You can also generate a lovely period vs. period derivative plot based on the latest catalogue information, using just three lines of code, e.g.:

from psrqpy import QueryATNF
query = QueryATNF(params=['P0', 'P1', 'ASSOC', 'BINARY', 'TYPE', 'P1_I'])
query.ppdot(showSNRs=True, showtypes='all')

gives

PPdot

Development and Support

Code development is done via the package's GitHib repository. Any contributions can be made via a fork and pull request model from that repository, and must adhere to the MIT license. Any problems with the code or support requests can be submitted via the repository's Issue tracker.

Test suite

There are tests supplied that cover many of the functions within PSRQpy. These can be run from the base directory of the repository (after installing the pytest and pytest-socket modules, e.g., with pip) by just calling:

pytest

These tests are not included in the pip installed version of the code.

Copyright and referencing for the catalogue

Regarding the use of the catalogue and software behind it, the following statements apply:

PSRCAT is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version. PSRCAT is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

PSRCAT makes use of "evaluateExpression: A Simple Expression Evaluator". Copyright © 1996 - 1999 Parsifal Software, All Rights Reserved.

The programs and databases remain the property of the Australia Telescope National Facility, CSIRO, and are covered by the CSIRO Legal Notice and Disclaimer.

If you make use of information from the ATNF Pulsar Catalogue in a publication, we would appreciate acknowledgement by reference to the publication "The ATNF Pulsar Catalogue", R. N. Manchester, G. B. Hobbs, A. Teoh & M. Hobbs, Astronomical Journal, 129, 1993-2006 (2005) and by quoting the web address http://www.atnf.csiro.au/research/pulsar/psrcat for updated versions.

If making use of this code to access the catalogue, or produce plots, I would be grateful if (as well as citing the ATNF pulsar catalogue paper and URL given above) you consider citing the JOSS paper for this software:

@article{psrqpy,
  author = {{Pitkin}, M.},
   title = "{psrqpy: a python interface for querying the ATNF pulsar catalogue}",
  volume = 3,
  number = 22,
   pages = 538,
   month = feb,
    year = 2018,
 journal = "{Journal of Open Source Software}",
     doi = {10.21105/joss.00538},
     url = {https://doi.org/10.21105/joss.00538}
}

License

This code is licensed under the MIT License.

© Matt Pitkin, 2017

PyPI version Anaconda-Server Badge version Build Status codecov Documentation Status status DOI ASCL

psrqpy's People

Contributors

astrogewgaw avatar cbassa avatar chrisburr avatar david-mckenna avatar duncanmmacleod avatar gregoryashton avatar killian42 avatar mattpitkin avatar nickswainston avatar respiewak avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar

psrqpy's Issues

Versioned get_references

I want to use a specific version of the ANTF catalogue, which I see you have nice arguments to specify the version in the query function. I start running into problems when I try to get references for that version. Here is a dummy script to show the problem:

import psrqpy

query = psrqpy.QueryATNF(version='1.68').pandas
ref_dict  = psrqpy.get_references()

for pulsar in list(query['PSRJ']):
    query_id = list(query['PSRJ']).index(pulsar)
    for query_column in query.keys():
        if 'REF' in query_column and query_column.startswith('S'):
            #print(f"{pulsar}  {query_column}")
            ref_code = query[query_column][query_id]
            if type(ref_code) == str:
                ref = ref_dict[ref_code]

Which gives me the following error:

Traceback (most recent call last):
  File "/tmp/build/psrqpy_test.py", line 13, in <module>
    ref = ref_dict[ref_code]
KeyError: 'gac+20'

I've tried a few of the cache options but can't seem to get it to work. Is there a way to get a specific version of the references?

get_msp_catalogue returning None with warning 'Count not query the MSP table'

I'm seeing the following failure when running the test suite for psrqpy 1.2.2:

$ python -m pytest --pyargs psrqpy.tests
========================================= test session starts ==========================================
platform linux -- Python 3.10.4, pytest-7.1.2, pluggy-1.0.0
rootdir: /home/duncan/git/pesummary, configfile: pyproject.toml
plugins: socket-0.5.1
collected 36 items

query_test.py .............F......................                                               [100%]

=============================================== FAILURES ===============================================
____________________________________________ test_msp_table ____________________________________________

query = <Table length=2>
      F1       F1_ERR  F0_ERR       F0
    1 / s2     1 / s2    Hz         Hz
   float64   ...- ------- ------- -------------
 -3.77535e-10   2e-15   1e-06     29.946923
-1.992272e-10   4e-17 1.3e-09 62.0261895958

    def test_msp_table(query):
        """
        Try downloading the MSP table.
        """

        from psrqpy.utils import get_msp_catalogue

        table = get_msp_catalogue()

        # when added there were 400 pulsars in the table
>       assert len(table) >= 400
E       TypeError: object of type 'NoneType' has no len()

../../opt/mambaforge/envs/psrqpy/lib/python3.10/site-packages/psrqpy/tests/query_test.py:475: TypeError
----------------------------------------- Captured stderr call -----------------------------------------
UserWarning: Count not query the MSP table.
======================================= short test summary info ========================================
FAILED query_test.py::test_msp_table - TypeError: object of type 'NoneType' has no len()

To reproduce:

conda create -c conda-forge -n psrqpy -y psrqpy pytest pytest-socket
conda activate psrqpy
python -m pytest --pyargs psrqpy.tests
Conda package list
$ conda list
# packages in environment at /home/duncan/opt/mambaforge/envs/psrqpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
ads                       0.12.3                     py_2    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
astropy                   5.1             py310hde88566_0    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
brotli                    1.0.9                h166bdaf_7    conda-forge
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotlipy                  0.7.0           py310h5764c6d_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.5.18.1          ha878542_0    conda-forge
certifi                   2022.5.18.1     py310hff52083_0    conda-forge
cffi                      1.15.0          py310h0fdd8cc_0    conda-forge
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
cryptography              37.0.2          py310h597c629_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
fonttools                 4.33.3          py310h5764c6d_0    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        4.11.4          py310hff52083_0    conda-forge
iniconfig                 1.1.1              pyh9f0ad1d_0    conda-forge
jpeg                      9e                   h166bdaf_1    conda-forge
kiwisolver                1.4.2           py310hbf28c38_1    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
ld_impl_linux-64          2.36.1               hea4e1c9_2    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libblas                   3.9.0           14_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcblas                  3.9.0           14_linux64_openblas    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.1.0              h8d9b700_16    conda-forge
libgfortran-ng            12.1.0              h69a702a_16    conda-forge
libgfortran5              12.1.0              hdcd56e2_16    conda-forge
libgomp                   12.1.0              h8d9b700_16    conda-forge
liblapack                 3.9.0           14_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libstdcxx-ng              12.1.0              ha89aaad_16    conda-forge
libtiff                   4.3.0                h0fcbabc_4    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp                   1.2.2                h3452ae3_0    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libzlib                   1.2.11            h166bdaf_1014    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
matplotlib-base           3.5.2           py310h5701ce4_0    conda-forge
mock                      4.0.3           py310hff52083_3    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.22.4          py310h4ef5377_0    conda-forge
openjpeg                  2.4.0                hb52868f_1    conda-forge
openssl                   1.1.1o               h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pandas                    1.4.2           py310h769672d_2    conda-forge
pillow                    9.1.1           py310he619898_0    conda-forge
pip                       22.1.1             pyhd8ed1ab_0    conda-forge
pluggy                    1.0.0           py310hff52083_3    conda-forge
psrqpy                    1.2.2              pyhd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
py                        1.11.0             pyh6c4a22f_0    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyerfa                    2.0.0.1         py310hde88566_2    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1           py310hff52083_5    conda-forge
pytest                    7.1.2           py310hff52083_0    conda-forge
pytest-socket             0.5.1              pyhd8ed1ab_0    conda-forge
python                    3.10.4          h9a8a25e_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.10                    2_cp310    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py310h5764c6d_4    conda-forge
readline                  8.1                  h46c0cb4_0    conda-forge
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
scipy                     1.8.1           py310h7612f91_0    conda-forge
setuptools                62.3.2          py310hff52083_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
soupsieve                 2.3.1              pyhd8ed1ab_0    conda-forge
sqlite                    3.38.5               h4ff8645_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022a                h191b570_0    conda-forge
unicodedata2              14.0.0          py310h5764c6d_1    conda-forge
urllib3                   1.26.9             pyhd8ed1ab_0    conda-forge
werkzeug                  2.1.2              pyhd8ed1ab_1    conda-forge
wheel                     0.37.1             pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
zstd                      1.5.2                h8a70e8d_1    conda-forge

AstropyDeprecationWarning: Galactocentric frame default parameters

Hi @mattpitkin with psrqpy-1.0.7 and astropy-4.0.1.post1 I'm getting the following warning:

WARNING: AstropyDeprecationWarning: In v4.1 and later versions, the Galactocentric frame will adopt default parameters that may update with time. An updated default parameter set is already available through the astropy.coordinates.galactocentric_frame_defaults ScienceState object, as described in but the default is currently still set to the pre-v4.0 parameter defaults. The safest way to guard against changing default parameters in the future is to either (1) specify all Galactocentric frame attributes explicitly when using the frame, or (2) set the galactocentric_frame_defaults parameter set name explicitly. See http://docs.astropy.org/en/latest/coordinates/galactocentric.html for more information. [astropy.coordinates.builtin_frames.galactocentric]

can't find some PSRs in 1.0.11 output that I can see on the web table

Hi @mattpitkin , I've updated to psrqpy 1.0.11 and am doing

pulsars_atnf = psrqpy.QueryATNF(params=['name','jname','raj','decj','posepoch','rajd','decjd','F0','F1','F2','F3','pepoch','binary','dist','age']).table

as before. I then ran into the problem that J1706-4434 which got recently added to the glitch table cannot be matched against that main catalog table using a np.where loop I had been using.

So I went to the web catalog and searched manually, and actually there I can see it as entry 1322. I then went through the difference of the web-generated table against the one from psrqpy from the top and already number 6 (J0021-0909) from the web version does not show up in the psrqpy output either.

Any idea what's going on? Is there maybe some local cache I need to purge of old catalog versions?

Upper case parameters

In the QueryATNF method, the params are all coerced to upper case before the query is sent. However, for some strange reason not all the parameter names used by ATNF are upper case. Some examples:

ELong: Ecliptic longitude (degrees)
DecJD: Declination (J2000) (degrees)

Could you either have these as special cases, or not have the input be case sensitive? I'm happy to submit an MR, but maybe you have some preference on how that should be handled.

Parsing non-numeric values for parameters such as 'ASSOC'

Revised from Issue #42 -- "Parameters like 'assoc', 'type', and 'bincomp' are not easy to work with in their standard form, due to the inclusion of extra information, such as references. These could, however, be parsed to put important information (e.g., for 'assoc': 'AXP') into a new column for easy querying."

It would be best to split the information in those parameters ('type', 'assoc', 'bincomp', 'survey'?) for easier querying and sorting. For example: the 'type' "HE[cdt69,fhm+69,hjm+70]" (for the Crab pulsar) would be split into the 'type' value and reference(s).

Assert statement in Pulsar.__ne__()

Should the assert statement in the Pulsar class __ne__ def be there? I wouldn't think that should return an error... Or, if it should, why isn't something like it in the __eq__ def?

Add pulsar class

I would like to add a pulsar class that just contains information on a single pulsar. The plan would be for the QueryATNF class to have a method that returns a list of pulsar classes (if the JName parameter had not been set in the original query then it would be re-run with that set, so that each pulsar had a name). The pulsar class would have attributes for each of the allowed parameters, which would be set from the initial query. If attributes that were not in the initial query were requested then the class would run a new query including that parameter. The class could also run a query to return the complete ephemeris for a pulsar.

error message

Hi, I am using python3.6 (and I am sure that pandas has been installed), when I try to use the package, I keep getting the following error message:

~/.local/lib/python3.6/site-packages/psrqpy/search.py in parse_assoc(self)
988
989 ASSOCnew = self.catalogue['ASSOC'].copy()
--> 990 idxassoc = ~ASSOCnew.isna()
991
992 # Set references first

/opt/python/intelpython3/lib/python3.6/site-packages/pandas/core/generic.py in getattr(self, name)
3079 if name in self._info_axis:
3080 return self[name]
-> 3081 return object.getattribute(self, name)
3082
3083 def setattr(self, name, value):

AttributeError: 'Series' object has no attribute 'isna'

Producing ppdot diagram for a list of pulsars

Hello, I've noticed the latest versions of psrqpy only create ppdot diagrams for the full catalogue, rather than just the specific pulsars that are queried. I think this is because the query_table() function returns the entire catalogue, as opposed to just the pulsars passed to QueryATNF.

Is it possible to [re]introduce the ability to only plot pulsars of interest in the ppdot diagram?

Add support for multiple associations

As far as I can tell from the docs, there's no way to view the full entry for associations, only the first one. How difficult would it be to return the full ASSOC string? I could take a crack at this if it's not going to require a large amount of refactoring.

Do some parsing on parameters

Parameters like 'assoc', 'type', and 'bincomp' are not easy to work with in their standard form, due to the inclusion of extra information, such as references. These could, however, be parsed to put important information (e.g., for 'assoc': 'AXP') into a new column for easy querying.

Other parameters are not automatically populated when they can be easily derived, like 'P0' from 'F0' or 'PB' from 'FB0'. Functions should be added to the class to populate these classes (perhaps only when required?)

Add plotting functions

It would be good to have plotting functions to produce plots of one parameter vs another for any numerical parameters that had been queried.

In particular, for a start it would be nice to have a pvspdot function that automatically produces a P-Pdot diagram (this could include the pulsar death line, lines of characteristic age, lines of constant magnetic field, and it could have different markers for the different pulsar types).

Add proper position uncertainties

Right now the position uncertainties are hard to parse since they depend on the precision of the sexadecimal position. I've asked ATNF to add an error on RAJD/DECJD (or the other decimal coordinates), but meanwhile would it be possible to add the position precision/error parsing into this package?

ATNF typo causing QueryATNF to fail

The Pulsar J1623+5850 was recently updated in the ATNF catalog to include a mistake in the DECJ value:

+58:5015.77

This auses QueryATNF to fail and even when trying to apply conditions to filter out this problematic pulsar, it seems the query still fails. Likely it is reading in the entire table before filtering.

The error in the pulsar's data entry has been reported, but is there a work-around in the meantime?

A bunch of things to check

Here are a variety of things I need to check:

  • Are the parameters passed to the URL string case sensitive? If not, then I should covert the names to all uppercase
  • Find out if the order of requested parameters is reflected in the order of those output by the ATNF catalogue.
  • Find out what the sort order value for descending order is and only allow that our 'asc' to be input.
  • Set __len__ (as the number of pulsars) and __str__ (as the string output of the astropy table) methods of QueryATNF class

Maybe add more features to the condition parameter in QueryATNF?

You have already added support for querying the catalogue using logical conditions, but some things are still not supported, such as the RANGE function, or specifying a range of RA/DEC using the HMS/DMS functions or h/d keywords. It would be nice if this feature was expanded to include these functions, along with the SURVEY, DISCOVERY functions. All the functions I just mentioned are (as you probably know) described here, in section 2.3.

missing rescaling factors for glitch sizes

The rescaling factors of 1e-9 for DeltaF/F and 1e-3 for DeltaF1/F1 (and their respective error bars) from the Jodrell table are missing as of 0d6f225:

import psrqpy
glitches_jodrell = psrqpy.get_glitch_catalogue()
glitches_jodrell[0]['DeltaF/F']
554.0

Should yield 5.54e-7.

A pypi release after that change would be much appreciated.

Last astropy version doesn't build in python2

When building I bumped into an issue I had for another tool before: the current development version of astropy doesn't build with python 2 but is still the default being downloaded if you don't specify anything in the requirements.txt when using python setup.py.

A simple fix, for now at least, is to replace "astropy" by "astropy<3.0" as in that file and it builds like a charm. When using pip, either in the way you use in in travis, or as in the documentation works well. I woud actually advice you to make sure that the installation in travis and the installation instructions in the README are the same.

Database read issue for PSRCAT versions >=1.66

I've just been playing around with this and when calling the get_catalogue() function with versions >=1.66 get the following error:

In [8]: psrqpy.utils.get_catalogue(cache=False, version="1.66")
---------------------------------------------------------------------------
UnboundLocalError                         Traceback (most recent call last)
Cell In [8], line 1
----> 1 psrqpy.utils.get_catalogue(cache=False, version="1.66")

File ~/code/mwa/venv/lib/python3.8/site-packages/psrqpy/utils.py:93, in get_catalogue(path_to_db, cache, update, pandas, version)
     91 # get the tarball
     92 try:
---> 93     dbtarfile = download_atnf_tarball(
     94         atnftarball, usecache=cache, version=version
     95     )
     96 except RuntimeError as e:
     97     raise IOError(
     98         f"Problem accessing ATNF catalogue tarball: {atnftarball}\n{e}"
     99     )

File ~/code/mwa/venv/lib/python3.8/site-packages/psrqpy/utils.py:388, in download_atnf_tarball(url, usecache, version)
    385     raise RuntimeError("Error downloading pulsar catalogue: {}".format(str(e)))
    387 if tarfile.status_code != 200:
--> 388     raise RuntimeError("Error downloading pulsar catalogue: {}".format(str(e)))
    390 if not os.path.exists(CACHEDIR):
    391     try:

UnboundLocalError: local variable 'e' referenced before assignment

including when I leave it as the default latest where it highlights that there may well be a malformed entry in the catalogue itself that it making the package balk. e.g.,

In [9]: psrqpy.utils.get_catalogue(cache=False)
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
File ~/code/mwa/venv/lib/python3.8/site-packages/psrqpy/utils.py:161, in get_catalogue(path_to_db, cache, update, pandas, version)
    160 try:
--> 161     float(val)
    162 except ValueError:

ValueError: could not convert string to float: '56,451.27'

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
Cell In [9], line 1
----> 1 psrqpy.utils.get_catalogue(cache=False)

File ~/code/mwa/venv/lib/python3.8/site-packages/psrqpy/utils.py:163, in get_catalogue(path_to_db, cache, update, pandas, version)
    161     float(val)
    162 except ValueError:
--> 163     raise ValueError("Value with error is not convertable to a float")
    165 if dataline[2][0] == "-" or "." in dataline[2]:
    166     # negative errors or those with decimal points are absolute values
    167     scalefac = 1.0

ValueError: Value with error is not convertable to a floa

It looks like it's an error introduced into one or more entries in the catalogue, or perhaps an edge case?

For reference, I'm using the version downloaded from pip: v1.2.2

Extend __getitem__

I could extend the QueryATNF __getitem__ method to allow you to get individual pulsars, using the get_pulsar method.

Python 3 build failing

It seems that the current build is failing for Python 3 (but is fine for Python 2.7). The failure happens in test_derived_ecliptic for which the comparisons between the ecliptic longitude from the ATNF pulsar catalogue and those derived using astropy seem to be worse than in Python 2.7.

I'll look into why this is the case.

Add some more radio flux frequencies

Howdy. I noticed that S350 is missing, a shame since slr+14 and mss+20 have lots of measurements. I found that simply adding lines to config.py fixed my problem, so I did that locally. As long as I was at it, I added the other radio frequencies that now exist in psrcat but didn't when Matt created all of this. Below is an excerpt from my local version. Have a nice day, David.

PSR_TIMING['S400'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 400 MHz PSR_TIMING['S1400'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 1400 MHz PSR_TIMING['S2000'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 2000 MHz PSR_TIMING['S30'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 30 MHz added by Dave Smith, 23 August 2022 PSR_TIMING['S40'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 40 MHz PSR_TIMING['S50'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 50 MHz PSR_TIMING['S60'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 60 MHz PSR_TIMING['S80'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 80 MHz PSR_TIMING['S100'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 100 MHz PSR_TIMING['S150'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 150 MHz PSR_TIMING['S200'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 200 MHz PSR_TIMING['S300'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 300 MHz PSR_TIMING['S350'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 350 MHz added by Dave Smith, 23 August 2022 PSR_TIMING['S600'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 600 MHz PSR_TIMING['S700'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 700 MHz PSR_TIMING['S800'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 800 MHz PSR_TIMING['S900'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 900 MHz PSR_TIMING['S1600'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 1600 MHz PSR_TIMING['S3000'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 3000 MHz PSR_TIMING['S4000'] = {'ref': True, 'err': False, 'units': 'mJy'} # Mean flux at 4000 MHz PSR_TIMING['S5000'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 5000 MHz PSR_TIMING['S6000'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 6000 MHz PSR_TIMING['S8000'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 8000 MHz PSR_TIMING['S10G'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 10 GHz added by Dave Smith, 23 August 2022 PSR_TIMING['S20G'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 20 GHz added by Dave Smith, 23 August 2022 PSR_TIMING['S50G'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 50 GHz added by Dave Smith, 23 August 2022 PSR_TIMING['S100G'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 100 GHz PSR_TIMING['S150G'] = {'ref': True, 'err': True, 'units': 'mJy'} # Mean flux at 150 GHz PSR_TIMING['SPINDX'] = {'ref': True, 'err': True, 'units': None} # Radio spectral index

Test suite failing to fetch catalogue

I am seeing reproducible failures of the psqrpy 1.2.4 test suite when it attempts to download the catalogue. This can be reproduced simply by

$ python -c "import psrqpy; print(psrqpy.QueryATNF())"
Traceback (most recent call last):
  File "/home/duncan.macleod/.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/utils.py", line 161, in get_catalogue
    float(val)
ValueError: could not convert string to float: '*'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/duncan.macleod/.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/search.py", line 351, in get_catalogue
    dbtable = get_catalogue(path_to_db=path_to_db, cache=cache,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/duncan.macleod/.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/utils.py", line 163, in get_catalogue
    raise ValueError("Value with error is not convertable to a float")
ValueError: Value with error is not convertable to a float

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/duncan.macleod/.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/search.py", line 216, in __init__
    _ = self.get_catalogue(path_to_db=loadfromdb, cache=cache,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/duncan.macleod/.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/search.py", line 355, in get_catalogue
    raise RuntimeError("Problem getting catalogue: {}".format(str(e)))
RuntimeError: Problem getting catalogue: Value with error is not convertable to a float
Environment info
$ conda list
# packages in environment at /home/duncan.macleod/.conda/envs/psrqpy:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       2_gnu    conda-forge
ads                       0.12.3                     py_2    conda-forge
appdirs                   1.4.4              pyh9f0ad1d_0    conda-forge
astropy                   5.2.1           py311hcb2cf0a_0    conda-forge
attrs                     22.2.0             pyh71513ae_0    conda-forge
beautifulsoup4            4.11.1             pyha770c72_0    conda-forge
brotli                    1.0.9                h166bdaf_8    conda-forge
brotli-bin                1.0.9                h166bdaf_8    conda-forge
brotlipy                  0.7.0           py311hd4cff14_1005    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
ca-certificates           2022.12.7            ha878542_0    conda-forge
certifi                   2022.12.7          pyhd8ed1ab_0    conda-forge
cffi                      1.15.1          py311h409f033_3    conda-forge
charset-normalizer        2.1.1              pyhd8ed1ab_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_0    conda-forge
contourpy                 1.0.6           py311h4dd048b_0    conda-forge
cryptography              39.0.0          py311h9b4c7bb_0    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
exceptiongroup            1.1.0              pyhd8ed1ab_0    conda-forge
fonttools                 4.38.0          py311hd4cff14_1    conda-forge
freetype                  2.12.1               hca18f0e_1    conda-forge
idna                      3.4                pyhd8ed1ab_0    conda-forge
importlib-metadata        6.0.0              pyha770c72_0    conda-forge
iniconfig                 2.0.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h166bdaf_2    conda-forge
kiwisolver                1.4.4           py311h4dd048b_1    conda-forge
lcms2                     2.14                 hfd0df8a_1    conda-forge
ld_impl_linux-64          2.39                 hcc3a1bd_1    conda-forge
lerc                      4.0.0                h27087fc_0    conda-forge
libblas                   3.9.0           16_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_8    conda-forge
libbrotlidec              1.0.9                h166bdaf_8    conda-forge
libbrotlienc              1.0.9                h166bdaf_8    conda-forge
libcblas                  3.9.0           16_linux64_openblas    conda-forge
libdeflate                1.14                 h166bdaf_0    conda-forge
libffi                    3.4.2                h7f98852_5    conda-forge
libgcc-ng                 12.2.0              h65d4601_19    conda-forge
libgfortran-ng            12.2.0              h69a702a_19    conda-forge
libgfortran5              12.2.0              h337968e_19    conda-forge
libgomp                   12.2.0              h65d4601_19    conda-forge
libjpeg-turbo             2.1.4                h166bdaf_0    conda-forge
liblapack                 3.9.0           16_linux64_openblas    conda-forge
libnsl                    2.0.0                h7f98852_0    conda-forge
libopenblas               0.3.21          pthreads_h78a6416_3    conda-forge
libpng                    1.6.39               h753d276_0    conda-forge
libsqlite                 3.40.0               h753d276_0    conda-forge
libstdcxx-ng              12.2.0              h46fd767_19    conda-forge
libtiff                   4.5.0                h82bc61c_0    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp-base              1.2.4                h166bdaf_0    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libzlib                   1.2.13               h166bdaf_4    conda-forge
markupsafe                2.1.1           py311hd4cff14_2    conda-forge
matplotlib-base           3.6.2           py311he728205_0    conda-forge
mock                      5.0.1              pyhd8ed1ab_0    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
ncurses                   6.3                  h27087fc_1    conda-forge
numpy                     1.24.1          py311hbde0eaa_0    conda-forge
openjpeg                  2.5.0                hfec8fc6_2    conda-forge
openssl                   3.0.7                h0b41bf4_1    conda-forge
packaging                 23.0               pyhd8ed1ab_0    conda-forge
pandas                    1.5.2           py311h2872171_2    conda-forge
pillow                    9.4.0           py311h104bd61_0    conda-forge
pluggy                    1.0.0              pyhd8ed1ab_5    conda-forge
pooch                     1.6.0              pyhd8ed1ab_0    conda-forge
psrqpy                    1.2.4              pyhd8ed1ab_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pyerfa                    2.0.0.1         py311h4c7f6c3_3    conda-forge
pyopenssl                 23.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.9              pyhd8ed1ab_0    conda-forge
pysocks                   1.7.1              pyha2e5f31_6    conda-forge
pytest                    7.2.0              pyhd8ed1ab_2    conda-forge
pytest-socket             0.5.1              pyhd8ed1ab_0    conda-forge
python                    3.11.0          ha86cf86_0_cpython    conda-forge
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python_abi                3.11                    3_cp311    conda-forge
pytz                      2022.7             pyhd8ed1ab_0    conda-forge
pyyaml                    6.0             py311hd4cff14_5    conda-forge
readline                  8.1.2                h0f457ee_0    conda-forge
requests                  2.28.1             pyhd8ed1ab_1    conda-forge
scipy                     1.10.0          py311h8e6699e_0    conda-forge
six                       1.16.0             pyh6c4a22f_0    conda-forge
soupsieve                 2.3.2.post1        pyhd8ed1ab_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tomli                     2.0.1              pyhd8ed1ab_0    conda-forge
tzdata                    2022g                h191b570_0    conda-forge
urllib3                   1.26.14            pyhd8ed1ab_0    conda-forge
werkzeug                  2.2.2              pyhd8ed1ab_0    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.6                h166bdaf_0    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
zipp                      3.11.0             pyhd8ed1ab_0    conda-forge
zstd                      1.5.2                h6239696_4    conda-forge
PyTest traceback
$ python -m pytest --pyargs psrqpy.tests -ra -v -x
======================================== test session starts =========================================
platform linux -- Python 3.11.0, pytest-7.2.0, pluggy-1.0.0 -- /home/duncan.macleod/.conda/envs/psrqpy/bin/python
cachedir: .pytest_cache
rootdir: /home/duncan.macleod
plugins: socket-0.5.1
collected 36 items

.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/tests/query_test.py::test_crab ERROR    [  2%]

=============================================== ERRORS ===============================================
____________________________________ ERROR at setup of test_crab _____________________________________

path_to_db = None, cache = True, update = False, pandas = True, version = '1.69'

    def get_catalogue(
        path_to_db=None, cache=True, update=False, pandas=False, version="latest"
    ):
        """
        This function will attempt to download and cache the entire ATNF Pulsar
        Catalogue database `tarball
        <http://www.atnf.csiro.au/people/pulsar/psrcat/downloads/psrcat_pkg.tar.gz>`_,
        or read in database file from a provided path. The database will be
        converted into an :class:`astropy.table.Table` or
        :class:`pandas.DataFrame`. This was originally based on the method in the
        `ATNF.ipynb
        <https://github.com/astrophysically/ATNF-Pulsar-Cat/blob/master/ATNF.ipynb>`_
        notebook by Joshua Tan (`@astrophysically
        <https://github.com/astrophysically/>`_).

        Args:
            path_to_db (str): if the path to a local version of the database file
                is given then that will be read in rather than attempting to
                download the file (defaults to None).
            cache (bool): cache the downloaded ATNF Pulsar Catalogue file. Defaults
                to True. This is ignored if `path_to_db` is given. The cache file
                path uses the specification in the `appdirs
                <https://github.com/ActiveState/appdirs>`_ package to set the
                location.
            update (bool): if True the ATNF Pulsar Catalogue will be
                re-downloaded and cached if there has been a change compared to the
                currently cached version. This is ignored if `path_to_db` is given.
            pandas (bool): if True the catalogue will be returned as a
                :class:`pandas.DataFrame` rather than the default of an
                :class:`~astropy.table.Table`.
            version (str): the version string (without the leading "v") of the ATNF
                catalogue version to download. This defaults to "latest" to get the
                most up-to-date version.

        Returns:
            :class:`~astropy.table.Table` or :class:`~pandas.DataFrame`: a table
            containing the entire catalogue.

        """

        if path_to_db is None:
            # remove any cached file if requested
            if update:
                if check_update():
                    os.remove(os.path.join(CACHEDIR, DEFAULTCACHENAME))

            if version == "latest":
                atnftarball = ATNF_TARBALL
            else:
                atnftarball = ATNF_VERSION_TARBALL.format(version)

            # get the tarball
            try:
                dbtarfile = download_atnf_tarball(
                    atnftarball, usecache=cache, version=version
                )
            except RuntimeError as e:
                raise IOError(
                    f"Problem accessing ATNF catalogue tarball: {atnftarball}\n{e}"
                )

            try:
                # open tarball
                pulsargz = tarfile.open(dbtarfile, mode="r:gz")

                # extract the database file
                dbfile = pulsargz.extractfile("psrcat_tar/psrcat.db")
            except IOError:
                raise IOError("Problem extracting the database file")
        else:
            try:
                dbfile = open(path_to_db, "r")
            except IOError:
                raise IOError("Error loading given database file")

        breakstring = "@"  # break between each pulsar
        commentstring = "#"  # specifies line is a comment

        # create list of dictionaries - one for each pulsar
        psrlist = [{}]

        version = None  # catalogue version

        # loop through lines in dbfile
        for line in dbfile.readlines():
            if isinstance(line, str):
                dataline = line.split()
            else:
                dataline = line.decode().split()  # Splits on whitespace

            if dataline[0][0] == commentstring:
                # get catalogue version (should be in first comment string)
                if dataline[0] == "#CATALOGUE" and len(dataline) == 2:
                    version = dataline[1]
                continue

            if dataline[0][0] == breakstring:
                # First break comes at the end of the first object and so forth
                psrlist.append({})  # New object!
                continue

            try:
                psrlist[-1][dataline[0]] = float(dataline[1])
            except ValueError:
                psrlist[-1][dataline[0]] = dataline[1]

            if len(dataline) > 2:
                # check whether 3rd value is a float (so its an error value) or not
                try:
                    float(dataline[2])
                    isfloat = True
                except ValueError:
                    isfloat = False

                if isfloat:
                    # error values are last digit errors, so convert to actual
                    # errors by finding the number of decimal places after the
                    # '.' in the value string
                    val = dataline[1].split(":")[-1]  # account for RA and DEC strings

                    try:
>                       float(val)
E                       ValueError: could not convert string to float: '*'

.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/utils.py:161: ValueError

During handling of the above exception, another exception occurred:

self = <[KeyError("Sorting by attribute 'JNAME' is not possible as it is not in the table") raised in repr()] QueryATNF object at 0x7f23de7042d0>
path_to_db = None, cache = True, update = False, overwrite = True, version = 'latest'

    def get_catalogue(self, path_to_db=None, cache=True, update=False,
                      overwrite=True, version="latest"):
        """
        Call the :func:`psrqpy.utils.get_catalogue` function to download the
        ATNF Pulsar Catalogue, or load a given catalogue path.

        Args:
            path_to_db (str): if the path to a local version of the database
                file is given then that will be read in rather than attempting to
                download the file (defaults to None).
            cache (bool): cache the downloaded ATNF Pulsar Catalogue file. Defaults
                to True. This is ignored if `path_to_db` is given.
            update (bool): if True the ATNF Pulsar Catalogue will be
                re-downloaded and cached if there has been a change compared to the
                currently cached version. This is ignored if `path_to_db` is given.
            overwrite (bool): if True the returned catalogue will overwrite the
                catalogue currently contained within the :class:`~psrqpy.QueryATNF`
                class. If False then a new :class:`~psrqpy.QueryATNF` copy of the
                catalogue will be returned.
            version (str): the version string (without the leading "v") of the ATNF
                catalogue version to download. This defaults to "latest" to get the
                most up-to-date version.

        Returns:
            :class:`psrqpy.QueryATNF`: a table containing the catalogue.
        """
        from .utils import get_catalogue

        cachefile = os.path.join(CACHEDIR, f"query_{version}.pkl")

        if cache and not update and path_to_db is None:
            # check if cache file exists
            try:
                self.load(cachefile, cleardfonly=True)
                return self
            except IOError:
                pass

        try:
>           dbtable = get_catalogue(path_to_db=path_to_db, cache=cache,
                                    update=update, pandas=True,
                                    version=version)

.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/search.py:351:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path_to_db = None, cache = True, update = False, pandas = True, version = '1.69'

    def get_catalogue(
        path_to_db=None, cache=True, update=False, pandas=False, version="latest"
    ):
        """
        This function will attempt to download and cache the entire ATNF Pulsar
        Catalogue database `tarball
        <http://www.atnf.csiro.au/people/pulsar/psrcat/downloads/psrcat_pkg.tar.gz>`_,
        or read in database file from a provided path. The database will be
        converted into an :class:`astropy.table.Table` or
        :class:`pandas.DataFrame`. This was originally based on the method in the
        `ATNF.ipynb
        <https://github.com/astrophysically/ATNF-Pulsar-Cat/blob/master/ATNF.ipynb>`_
        notebook by Joshua Tan (`@astrophysically
        <https://github.com/astrophysically/>`_).

        Args:
            path_to_db (str): if the path to a local version of the database file
                is given then that will be read in rather than attempting to
                download the file (defaults to None).
            cache (bool): cache the downloaded ATNF Pulsar Catalogue file. Defaults
                to True. This is ignored if `path_to_db` is given. The cache file
                path uses the specification in the `appdirs
                <https://github.com/ActiveState/appdirs>`_ package to set the
                location.
            update (bool): if True the ATNF Pulsar Catalogue will be
                re-downloaded and cached if there has been a change compared to the
                currently cached version. This is ignored if `path_to_db` is given.
            pandas (bool): if True the catalogue will be returned as a
                :class:`pandas.DataFrame` rather than the default of an
                :class:`~astropy.table.Table`.
            version (str): the version string (without the leading "v") of the ATNF
                catalogue version to download. This defaults to "latest" to get the
                most up-to-date version.

        Returns:
            :class:`~astropy.table.Table` or :class:`~pandas.DataFrame`: a table
            containing the entire catalogue.

        """

        if path_to_db is None:
            # remove any cached file if requested
            if update:
                if check_update():
                    os.remove(os.path.join(CACHEDIR, DEFAULTCACHENAME))

            if version == "latest":
                atnftarball = ATNF_TARBALL
            else:
                atnftarball = ATNF_VERSION_TARBALL.format(version)

            # get the tarball
            try:
                dbtarfile = download_atnf_tarball(
                    atnftarball, usecache=cache, version=version
                )
            except RuntimeError as e:
                raise IOError(
                    f"Problem accessing ATNF catalogue tarball: {atnftarball}\n{e}"
                )

            try:
                # open tarball
                pulsargz = tarfile.open(dbtarfile, mode="r:gz")

                # extract the database file
                dbfile = pulsargz.extractfile("psrcat_tar/psrcat.db")
            except IOError:
                raise IOError("Problem extracting the database file")
        else:
            try:
                dbfile = open(path_to_db, "r")
            except IOError:
                raise IOError("Error loading given database file")

        breakstring = "@"  # break between each pulsar
        commentstring = "#"  # specifies line is a comment

        # create list of dictionaries - one for each pulsar
        psrlist = [{}]

        version = None  # catalogue version

        # loop through lines in dbfile
        for line in dbfile.readlines():
            if isinstance(line, str):
                dataline = line.split()
            else:
                dataline = line.decode().split()  # Splits on whitespace

            if dataline[0][0] == commentstring:
                # get catalogue version (should be in first comment string)
                if dataline[0] == "#CATALOGUE" and len(dataline) == 2:
                    version = dataline[1]
                continue

            if dataline[0][0] == breakstring:
                # First break comes at the end of the first object and so forth
                psrlist.append({})  # New object!
                continue

            try:
                psrlist[-1][dataline[0]] = float(dataline[1])
            except ValueError:
                psrlist[-1][dataline[0]] = dataline[1]

            if len(dataline) > 2:
                # check whether 3rd value is a float (so its an error value) or not
                try:
                    float(dataline[2])
                    isfloat = True
                except ValueError:
                    isfloat = False

                if isfloat:
                    # error values are last digit errors, so convert to actual
                    # errors by finding the number of decimal places after the
                    # '.' in the value string
                    val = dataline[1].split(":")[-1]  # account for RA and DEC strings

                    try:
                        float(val)
                    except ValueError:
>                       raise ValueError("Value with error is not convertable to a float")
E                       ValueError: Value with error is not convertable to a float

.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/utils.py:163: ValueError

During handling of the above exception, another exception occurred:

    @pytest.fixture(scope="module")
    def query():
>       return QueryATNF()

.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/tests/conftest.py:11:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/search.py:216: in __init__
    _ = self.get_catalogue(path_to_db=loadfromdb, cache=cache,
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <[KeyError("Sorting by attribute 'JNAME' is not possible as it is not in the table") raised in repr()] QueryATNF object at 0x7f23de7042d0>
path_to_db = None, cache = True, update = False, overwrite = True, version = 'latest'

    def get_catalogue(self, path_to_db=None, cache=True, update=False,
                      overwrite=True, version="latest"):
        """
        Call the :func:`psrqpy.utils.get_catalogue` function to download the
        ATNF Pulsar Catalogue, or load a given catalogue path.

        Args:
            path_to_db (str): if the path to a local version of the database
                file is given then that will be read in rather than attempting to
                download the file (defaults to None).
            cache (bool): cache the downloaded ATNF Pulsar Catalogue file. Defaults
                to True. This is ignored if `path_to_db` is given.
            update (bool): if True the ATNF Pulsar Catalogue will be
                re-downloaded and cached if there has been a change compared to the
                currently cached version. This is ignored if `path_to_db` is given.
            overwrite (bool): if True the returned catalogue will overwrite the
                catalogue currently contained within the :class:`~psrqpy.QueryATNF`
                class. If False then a new :class:`~psrqpy.QueryATNF` copy of the
                catalogue will be returned.
            version (str): the version string (without the leading "v") of the ATNF
                catalogue version to download. This defaults to "latest" to get the
                most up-to-date version.

        Returns:
            :class:`psrqpy.QueryATNF`: a table containing the catalogue.
        """
        from .utils import get_catalogue

        cachefile = os.path.join(CACHEDIR, f"query_{version}.pkl")

        if cache and not update and path_to_db is None:
            # check if cache file exists
            try:
                self.load(cachefile, cleardfonly=True)
                return self
            except IOError:
                pass

        try:
            dbtable = get_catalogue(path_to_db=path_to_db, cache=cache,
                                    update=update, pandas=True,
                                    version=version)
        except Exception as e:
>           raise RuntimeError("Problem getting catalogue: {}".format(str(e)))
E           RuntimeError: Problem getting catalogue: Value with error is not convertable to a float

.conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/search.py:355: RuntimeError
====================================== short test summary info =======================================
ERROR .conda/envs/psrqpy/lib/python3.11/site-packages/psrqpy/tests/query_test.py::test_crab - RuntimeError: Problem getting catalogue: Value with error is not convertable to a float
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
========================================== 1 error in 4.76s ==========================================

Getting an unexpected error from derived_binary

I've produced a psrcat-style db_file from some ephemerides for my project, which have been verified to work with tools like tempo2 and psrchive, but I get an error when trying to read in the db_file with psrqpy. This is the relevant part of the traceback.

  File "/fred/oz002/users/rspiewak/code/psrqpy/psrqpy/search.py", line 198, in __init__
    update=checkupdate)
  File "/fred/oz002/users/rspiewak/code/psrqpy/psrqpy/search.py", line 347, in get_catalogue
    self.set_derived()
  File "/fred/oz002/users/rspiewak/code/psrqpy/psrqpy/search.py", line 1111, in set_derived
    self.derived_binary()      # derived binary parameters
  File "/fred/oz002/users/rspiewak/code/psrqpy/psrqpy/search.py", line 1554, in derived_binary
    self.update(ECCnew, name='ECC')
  File "/fred/oz002/users/rspiewak/code/psrqpy/psrqpy/search.py", line 394, in update
    raise ValueError("Could not update table with "
ValueError: Could not update table with supplied column

It seems as if it would be due to EPS1/2 parameters being incorrectly converted. Some of the ephemerides have one or both of those parameters set to 0, but I tried changing them to, eg, 1e-9 with no luck. Any ideas?

Offline functionality?

I was thinking it might be useful to be able to maintain functionality offline, for example with calls to the psrcat software or psrcat.db files. Is this already supported, is it something that has been considered, or is it something that would require a lot of reworking?

Make `get_catalogue` into a class?

After #36, we may want to re-work `get_catalogue into a class, and make it possible to add to the table with data from additional .db files (e.g., private updated timing parameters).

Add gravitational wave derived parameters

As a gravitational wave person it would be nice to have utility functions to output some of our standard values:

  • gravitational wave luminosity (assuming l=m=2 mode and braking index of 5)
  • spin-down limit on the gravitational wave amplitude
  • spin-down limit on the l=m=2 mass quadrupole
  • spin-down limit on the fiducial ellipticity

References no longer work

It's been found that the references from the ATNF catalogue not longer work. It looks like there has been a change to the formating of the reference file in the catalogue tarball, which breaks things. This will need to be fixed (note that the current file also seems to contain unnecessary natexlab{abc...} strings appended to the years, which will need to be removed!)

pplot fails if no latex on system, default to usetex=False?

When running python -m pytest --pyargs psrqpy.tests on a system that doesn't have texlive pre-installed, one ends up with the following error:

______________________________ test_ppdot_diagram ______________________________
.
.
.
>       fig = query.ppdot(showtypes='ALL', showGCs=True, intrinsicpdot=True)
../../../envs/igwn-py38-testing/lib/python3.8/site-packages/psrqpy/tests/query_test.py:433: 
.
.
.
E           RuntimeError: Failed to process string with tex because latex could not be found

I think this is fundamentally because psrqpy defaults to text.usetex = True in matplotlib:

psrqpy/psrqpy/search.py

Lines 2847 to 2848 in ba9119c

rcparams['text.usetex'] = rcparams['text.usetex'] if \
'text.usetex' in rcparams else True

If possible, I think it would be a positive change to default to False such that this code runs without (major) issue on systems that don't pre-install texlive (i.e. the IGWN HTCondor Pool). Matplotlib's native text rendering is much better than it used to be, so can probably render most/all of the necessary symbols without relying on latex.

missing SNR label in legend?

Hi Matt, this is really nice! Makes plotting pulsar populations very easy. I just cloned and installed per your instructions, and everything ran swimmingly except for one issue: I tried to recreate one of your plots that showed SNRs, and got a KeyError with 'SNR.' I dug around search.py and found the issue: Starting at L940 you define legend strings, but 'SNR' doesn't make it in, while it does get added into nshowtypes. Here is the full error message (without 'showSNRs' there is no error):

In [24]: query.ppdot(Bfield=[], showtypes=['BINARY'], showSNRs=True)
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
<ipython-input-24-d331258e9612> in <module>()
----> 1 query.ppdot(Bfield=[], showtypes=['BINARY'],showSNRs=True)

//anaconda/lib/python2.7/site-packages/psrqpy-0.4.12-py2.7.egg/psrqpy/search.pyc in ppdot(self, intrinsicpdot, excludeGCs, showtypes, showGCs, showSNRs, markertypes, deathline, deathmodel, filldeath, filldeathtype, showtau, brakingidx, tau, showB, Bfield, pdotlims, periodlims, rcparams)
    978                     markertypes[thistype]['linestyle'] = 'none'
    979                 typehandle, = ax.loglog(periods[typeidx], pdots[typeidx],
--> 980                                         label=typelegstring[thistype],
    981                                         **markertypes[thistype])
    982                 if thistype in typelegstring:

KeyError: 'SNR'

Error retrieving ATNF tarball

I am getting a 503 service unavailable error when I try to download and query the database. Pasting the error below. I googled around and didn't find any fixes. Wondering if you have any idea what is going on? I am using the following package versions:

python                    3.9.1                h88f2d9e_2
astropy                   4.2.1            py39he3068b8_1
psrqpy                    1.1.1              pyhd8ed1ab_0    conda-forge

The error:

>>> query = QueryATNF()
Traceback (most recent call last):
  File "/Users/cherryng/miniconda3/lib/python3.9/site-packages/psrqpy/utils.py", line 76, in get_catalogue
    dbtarfile = download_file(ATNF_TARBALL, cache=cache)
  File "/Users/cherryng/miniconda3/lib/python3.9/site-packages/astropy/utils/data.py", line 1257, in download_file
    raise errors[sources[0]]
  File "/Users/cherryng/miniconda3/lib/python3.9/site-packages/astropy/utils/data.py", line 1222, in download_file
    f_name = _download_file_from_source(
  File "/Users/cherryng/miniconda3/lib/python3.9/site-packages/astropy/utils/data.py", line 1051, in _download_file_from_source
    with urlopener.open(req, timeout=timeout) as remote:
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 555, in error
    result = self._call_chain(*args)
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 747, in http_error_302
    return self.parent.open(new, timeout=req.timeout)
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 523, in open
    response = meth(req, response)
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 632, in http_response
    response = self.parent.error(
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 561, in error
    return self._call_chain(*args)
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/Users/cherryng/miniconda3/lib/python3.9/urllib/request.py", line 641, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 503: Service Unavailable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/cherryng/miniconda3/lib/python3.9/site-packages/psrqpy/search.py", line 336, in get_catalogue
    dbtable = get_catalogue(path_to_db=path_to_db, cache=cache,
  File "/Users/cherryng/miniconda3/lib/python3.9/site-packages/psrqpy/utils.py", line 78, in get_catalogue
    raise IOError("Problem accessing ATNF catalogue tarball")
OSError: Problem accessing ATNF catalogue tarball

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/cherryng/miniconda3/lib/python3.9/site-packages/psrqpy/search.py", line 214, in __init__
    _ = self.get_catalogue(path_to_db=loadfromdb, cache=cache,
  File "/Users/cherryng/miniconda3/lib/python3.9/site-packages/psrqpy/search.py", line 339, in get_catalogue
    raise RuntimeError("Problem getting catalogue: {}".format(str(e)))
RuntimeError: Problem getting catalogue: Problem accessing ATNF catalogue tarball

Plotting .ppdot gives two plots in Jupyter notebook

I am running a Jupyter Notebook with Python3 and with (problem also without) the %matplotlib notebook before all my imports. My code is

psrcatParams = ['name', 'jname', 'posepoch', 'elong', 'elat', 'pmelong', 'pmelat', 'gl', 'gb', 'p0', 'p1', 
                'dist', 'assoc']
qq = QueryATNF(params=psrcatParams, condition='P0 < 30.0e-3')

And in the next cell
qq.ppdot( showSNRs=True, showtypes='all', rcparams={'figure.figsize': (8, 7), 'figure.dpi': 111} )
Which gives two plots, one after the other. They look identical to me and are what I expect from the manual. Oh the defaults on my laptop are also huge plots, hence the rcparams above.

By the way, thanks for the code. It is a great way to get an ATNF query into pandas.
bill e.g.
ps--Running on Fedora 29 with bleeding edge everything, I would guess.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.