Code Monkey home page Code Monkey logo

pydl's Introduction

PyDL

Powered by Astropy Badge License DOI: 10.5281/zenodo.2575873 PyPI Badge GitHub Actions CI Status Test Coverage Status Documentation Status

Description

This package consists of Python replacements for functions that are part of the IDL® built-in library or part of astronomical IDL® libraries. The emphasis is on reproducing results of the astronomical library functions. Only the bare minimum of IDL® built-in functions are implemented to support this.

There are four astronomical libraries targeted:

This package affiliated with the astropy project and is registered with PyPI.

Full Documentation

Please visit PyDL on Read the Docs

History

This package was initially developed on the SDSS-III svn repository. It was moved to the new GitHub repository on 2013-03-06. The present location of the repository is http://github.com/weaverba137/pydl .

License

PyDL is free software licensed under a 3-clause BSD-style license. For details see the LICENSE.rst file.

Legal

pydl's People

Contributors

albireox avatar astrofrog avatar bsipocz avatar cdeil avatar eteq avatar kbarbary avatar larrybradley avatar mwcraig avatar neutrinoceros avatar weaverba137 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

pydl's Issues

Incorrect Euclidean division in Python 3

In Python 3, the Euclidean division needs to be done using // instead of /.
This causes the test of the median to fail with the following backtrace:

        from scipy.signal import medfilt, medfilt2d
        if width is None:
            if axis is None:
                f = array.flatten()
                if f.size % 2 == 1 or even:
                    return np.median(array)
                else:
                    i = f.argsort()
>                   return f[i[f.size/2]]
E                   IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
 
pydl/median.py:53: IndexError

[1.0.0rc2] CI failure with astropy-6.0: test_munu_to_radec fails with ConvertError

When running the CI tests with Astropy-6.0, I get the following failure:

______________________________ test_munu_to_radec ______________________________

    def test_munu_to_radec():
        munu = SDSSMuNu(mu=0.0*u.deg, nu=0.0*u.deg, stripe=10)
>       radec = munu.transform_to(ICRS)

/usr/lib/python3/dist-packages/pydl/pydlutils/tests/test_coord.py:17: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <SDSSMuNu Coordinate (stripe=10, node=95.0 deg): (mu, nu) in deg
    (0., 0.)>
new_frame = <class 'astropy.coordinates.builtin_frames.icrs.ICRS'>

    def transform_to(self, new_frame):
        """…"""
        from .errors import ConvertError
    
        if self._data is None:
            raise ValueError("Cannot transform a frame with no data")
    
        if (
            getattr(self.data, "differentials", None)
            and hasattr(self, "obstime")
            and hasattr(new_frame, "obstime")
            and np.any(self.obstime != new_frame.obstime)
        ):
            raise NotImplementedError(
                "You cannot transform a frame that has velocities to another frame at a"
                " different obstime. If you think this should (or should not) be"
                " possible, please comment at"
                " https://github.com/astropy/astropy/issues/6280"
            )
    
        if hasattr(new_frame, "_sky_coord_frame"):
            # Input new_frame is not a frame instance or class and is most
            # likely a SkyCoord object.
            new_frame = new_frame._sky_coord_frame
    
        trans = frame_transform_graph.get_transform(self.__class__, new_frame.__class__)
        if trans is None:
            if new_frame is self.__class__:
                # no special transform needed, but should update frame info
                return new_frame.realize_frame(self.data)
            msg = "Cannot transform from {0} to {1}"
>           raise ConvertError(msg.format(self.__class__, new_frame.__class__))
E           astropy.coordinates.errors.ConvertError: Cannot transform from <class 'pydl.pydlutils.coord.SDSSMuNu'> to <class 'abc.ABCMeta'>

/usr/lib/python3/dist-packages/astropy/coordinates/baseframe.py:1272: ConvertError

This is a regression; it did not happen with astropy-5.3.2.
Fuill test log here.

REBIN exactness

Hello,

I am looking for an identical REBIN implementation in Python. To try things out, I tried the following,

seed = 100
array = FIX(RANDOMN(seed, 100, 200) * 100, TYPE=1)
array_rebin = REBIN(array, 50, 100)

The results of using the REBIN function provided by this package did not exactly match , though the differences are no greater than 1 in any given pixel.

Citable record?

We're planning to reference and cite all Astropy affiliated packages in the upcoming Astropy v2.0 paper. I couldn't find a citable record for this package - are you interested in creating one so that we can cite it? For example, you can create a zenodo record and just comment back here to let me know that it's done. Please also let me know if you'd like to opt out of a citation and we'll just link to the package. Thanks!

[1.0.0rc2] test_yanny fails on big endian machines

On a Debian system with s90x architecture (IBM zSystems, big endian, 64 bit), we get the following test failure:

_____________________________ TestYanny.test_yanny _____________________________

self = <pydl.pydlutils.tests.test_yanny.TestYanny object at 0x3ff377b37c0>

        def test_yanny(self):
            """Used to test the yanny class.
            """
            #[…]
            for t in par.tables():
>               assert (par.dtype(t) ==
                        self.json2dtype(struct_data[t]['dtype']))
E                       AssertionError: assert dtype([('mag', '>f4', (5,)), ('b', 'S33', (5,)), ('foo', 'S25'), ('c', '>f8'), ('flags', '>i4', (2,)), ('new_flag', 'S5')]) == dtype([('mag', '<f4', (5,)), ('b', 'S33', (5,)), ('foo', 'S25'), ('c', '<f8'), ('flags', '<i4', (2,)), ('new_flag', 'S5')])
E                        +  where dtype([('mag', '>f4', (5,)), ('b', 'S33', (5,)), ('foo', 'S25'), ('c', '>f8'), ('flags', '>i4', (2,)), ('new_flag', 'S5')]) = <bound method yanny.dtype of #%yanny\n# test.par\n#\n# FTCL/Yanny file for testing the perl & python yanny readers.\n#\n# T...ESS 2008-06-22T00:27:33\nSTATUS_UPDATE FAILURE 2008-06-22T01:27:33\n#\nold {1.2 2.3 3.4} hello\nold {4.3 3.2 2.1} { { } }\n>('MYSTRUCT')
E                        +    where <bound method yanny.dtype of #%yanny\n# test.par\n#\n# FTCL/Yanny file for testing the perl & python yanny readers.\n#\n# T...ESS 2008-06-22T00:27:33\nSTATUS_UPDATE FAILURE 2008-06-22T01:27:33\n#\nold {1.2 2.3 3.4} hello\nold {4.3 3.2 2.1} { { } }\n> = #%yanny\n# test.par\n#\n# FTCL/Yanny file for testing the perl & python yanny readers.\n#\n# This file is much more complic...CESS 2008-06-22T00:27:33\nSTATUS_UPDATE FAILURE 2008-06-22T01:27:33\n#\nold {1.2 2.3 3.4} hello\nold {4.3 3.2 2.1} { { } }\n.dtype
E                        +  and   dtype([('mag', '<f4', (5,)), ('b', 'S33', (5,)), ('foo', 'S25'), ('c', '<f8'), ('flags', '<i4', (2,)), ('new_flag', 'S5')]) = <bound method YannyTestCase.json2dtype of <pydl.pydlutils.tests.test_yanny.TestYanny object at 0x3ff377b37c0>>([['mag', '<f4', [5]], ['b', 'S33', [5]], ['foo', 'S25'], ['c', '<f8'], ['flags', '<i4', [2]], ['new_flag', 'S5']])
E                        +    where <bound method YannyTestCase.json2dtype of <pydl.pydlutils.tests.test_yanny.TestYanny object at 0x3ff377b37c0>> = <pydl.pydlutils.tests.test_yanny.TestYanny object at 0x3ff377b37c0>.json2dtype

/usr/lib/python3/dist-packages/pydl/pydlutils/tests/test_yanny.py:300: AssertionError

Full test log here.
I think this is a wrong endianness mark somewhere.

Astropy Affiliated Package Review

This package has been re-reviewed by the Astropy coordination committee in relation to the Astropy affiliated package ecosystem.

We have adopted a review process for affiliated package that includes assigning quantitative ‘scores’ (red/orange/green) for different categories of review. You can read up more about this process here. (This document, currently in Google Docs, will be moved to the documentation in the near future.) For each of the categories below we have listed the score and have included some comments when the score is not green.

Functionality/ScopeGeneral%20package
No further comments
Integration with Astropy ecosystemGood
No further comments
DocumentationNeeds%20work
The documentation at the moment just consists of API docs, but it really should include a minimum of narrative/introduction, and some examples of usage
TestingPartial
Test coverage is currently 62%, which could be improved
Development statusGood
No further comments
Python 3 compatibilityGood
No further comments

Summary/Decision: There are some areas in which this package did not score well with the new review process (in this case the documentation). As per the review guidelines, these areas should be fixed – if review areas stay in the red for too long, we may consider removing the package from the affiliated package list. However, in this case, simply adding some narrative docs would bring the score for the docs up to at least orange, so it should be an easy fix.

If you agree with the above review, please feel free to close this issue. If you have any follow-up questions or disagree with any of the comments above, leave a comment and we can discuss it here. At any point in future you can request a re-review of the package if you believe any of the scores should be updated - contact the coordination committee, and we’ll do a new review. Note that we are in the process of redesigning the http://affiliated.astropy.org page to show these scores (but not the comments). Finally, please keep the title of this issue as-is (“Astropy Affiliated Package Review”) to make it easy to search for affiliated package reviews in future.

Unexpected iaxis argument in call to djs_maskinterp()

Hi,
I noticed a likely typo here: https://github.com/weaverba137/pydl/blob/main/pydl/pydlspec2d/spec2d.py#L439
Within the function filter_thru(), djs_maskinterp() is called with argument 'iaxis', but expects 'axis' instead:
https://github.com/weaverba137/pydl/blob/main/pydl/pydlutils/image.py#L63
Simply renaming the argument (iaxis->axis) allows the function to run to completion...but expert eyes are needed to tell if this is the correct behaviour.
Many thanks!

additional pydl Mangle bugs

I've found a few more apparent issues in the mangle codes, when using is_in_polygon. I've attached some test files - a rectangular lune polygon (with 4 caps) and a single circular polygon (1 cap), both in both polygon and fits formats, along with random points in the circular region (fits file with tags ra and dec). These were generated with idlutils and/or mangle itself to illustrate.

test_files.zip

import numpy as np
import pydl.pydlutils.mangle as mangle
from astropy.table import Table

poly1=mangle.read_fits_polygons('lune.fits')
poly2=mangle.read_fits_polygons('circ.fits')
rand=Table.read('rand_circ.fits')
pos=np.transpose(np.array([rand['RA'],rand['DEC']]))

isin1=mangle.is_in_polygon(poly1,pos)

First, note the need to transpose the position data table - it might be nice to adjust the mangle code to do this transpose internally, or avoid it by rearranging some dimensions.

Second, the is_in_polygon call will crash because of a dimension problem:

in is_in_polygon(polygon, points, caps)

    512         if is_cap_used(polygon.use_caps, cap):

    513             in_polygon &= is_in_cap(polygon.x[icap, :],
--> 514                                     polygon.cm[icap], points)
    515     return in_polygon
    516 

/Users/Mike/computing/python/anaconda/lib/python2.7/site-packages/pydl/pydlutils/mangle.py in is_in_cap(x, cm, points)
    482         A boolean vector giving the result for each point.
    483     """
--> 484     return cap_distance(x, cm, points) >= 0.0
    485 
    486 

/Users/Mike/computing/python/anaconda/lib/python2.7/site-packages/pydl/pydlutils/mangle.py in cap_distance(x, cm, points)
    402     else:
    403         raise ValueError("Inappropriate shape for point!")
--> 404     dotprod = np.dot(xyz, x)
    405     cdist = np.degrees(np.arccos(1.0 - np.abs(cm)) - np.arccos(dotard))I
    406     if cm < 0:

ValueError: shapes (100,3) and (4,3) not aligned: 3 (dim 1) != 4 (dim 0)

I tracked this down to an extra dimension of length 1 in the polygon cap parameters (x and cm). I was able to fix this (though there might be other ways) adjusting line 513 from:

in_polygon &= is_in_cap(polygon.x[icap, :],
                                    polygon.cm[icap], points)

to

in_polygon &= is_in_cap(np.squeeze(polygon.x)[icap, :],
                                    np.squeeze(polygon.cm)[icap], points)

This works, and gives correct results (compared to idlutils) for any random set.

However, if instead I'm using the circular polygon with a single cap, I get an indexing error

isin1=mangle.is_in_polygon(poly2,pos)
is_in_polygon(polygon, points, ncaps)
    511     for icap in range(usencaps):
    512         if is_cap_used(polygon.use_caps, icap):
--> 513             in_polygon &= is_in_cap(np.squeeze(polygon.x)[icap, :],
    514                                     np.squeeze(polygon.cm)[icap], points)
    515     return in_polygon

IndexError: too many indices for array

I was able to circumvent this with a conditional in is_in_polygon based on the number of caps, but was unable to get it to return correct results.

Finally, I noticed that if I try reading in these polygons in Mangle format using read_mangle_polygons, none of this works because the polygons don't have the appropriate attributes (e.g. .x or .cm). This seems to be a bug in read_mangle_polygons.

bspline.__init__() got an unexpected keyword argument 'groupbadpix'

Hi,

Thanks for your wonderful code. I found an error when I'm trying to run pydlspec2d.spec2d.combine1fiber function. The traceback is as follow:

stack_flux, stack_ivar = spec2d.combine1fiber(loglam_cube, flux_cube, loglam_ref, ivar_cube)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/site-packages/pydl/pydlspec2d/spec2d.py", line 232, in combine1fiber
sset, bmask = iterfit(inloglam_r[ss],
^^^^^^^^^^^^^^^^^^^^^^^
File "/site-packages/pydl/pydlutils/bspline.py", line 608, in iterfit
sset = bspline(xdata[xsort[maskwork]], **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: bspline.init() got an unexpected keyword argument 'groupbadpix'

To me, it is quite straightforward so I don't provide examples for this error. Probably, due to some update in bspline class, you update the parameter list in the bspline.__init__ but forgot the argument groupbadpix.

Let me know if you need more information about this issue.

Cheers

IOError when running tests with a non-writable PyDL location

When I run tests using pydl.test(), the following tests fail because the installed PyDL is not writable.
I suggest that a temporary location is used instead.

__________________________ TestMisc.test_struct_print __________________________

self = <pydl.pydlutils.tests.test_misc.TestMisc object at 0x7fd44c6c0390>

    def test_struct_print(self):
        slist = np.zeros((5,), dtype=[('a', 'c16'), ('b', np.bool)])
        with raises(PydlutilsException):
            lines, css = struct_print(slist, silent=True)
        slist = np.array([(1, 2.34, 'five'),
                          (2, 3.456, 'seven'),
                          (3, -4.5678, 'nine')],
                          dtype=[('a', 'i4'), ('bb', 'f4'), ('ccc', 'S5')])
        lines, css = struct_print(slist, silent=True)
        assert lines[0] == 'a bb           ccc  '
        assert lines[1] == '- ------------ -----'
        assert lines[2] == '1         2.34 five '
        assert lines[3] == '2        3.456 seven'
        assert lines[4] == '3      -4.5678 nine '
        assert len(css) == 0
        lines, css = struct_print(slist, silent=True, alias={'ccc': 'c'})
        assert lines[0] == 'a bb           c    '
        assert lines[1] == '- ------------ -----'
        assert lines[2] == '1         2.34 five '
        assert lines[3] == '2        3.456 seven'
        assert lines[4] == '3      -4.5678 nine '
        assert len(css) == 0
        lines, css = struct_print(slist, silent=True,
                                  formatcodes={'a': '{0:02d}'})
        assert lines[0] == 'a  bb           ccc  '
        assert lines[1] == '-- ------------ -----'
        assert lines[2] == '01         2.34 five '
        assert lines[3] == '02        3.456 seven'
        assert lines[4] == '03      -4.5678 nine '
        assert len(css) == 0
        lines, css = struct_print(slist, silent=True, fdigit=3)
        assert lines[0] == 'a bb         ccc  '
        assert lines[1] == '- ---------- -----'
        assert lines[2] == '1       2.34 five '
        assert lines[3] == '2       3.46 seven'
        assert lines[4] == '3      -4.57 nine '
        assert len(css) == 0
        lines, css = struct_print(slist, silent=True, html=True)
        assert lines[0] == '<table>'
        assert lines[1] == '<tr><th>a</th><th>bb</th><th>ccc</th></tr>'
        assert lines[2] == '<tr><td>1</td><td>        2.34</td><td>five </td></tr>'
        assert lines[3] == '<tr><td>2</td><td>       3.456</td><td>seven</td></tr>'
        assert lines[4] == '<tr><td>3</td><td>     -4.5678</td><td>nine </td></tr>'
        assert lines[5] == '</table>'
        assert css[0] == '<style type="text/css">'
        assert css[1] == 'table {'
        assert css[2] == '    border-collapse: collapse;'
        assert css[3] == '}'
        assert css[4] == 'th {'
        assert css[5] == '    padding: 2px;'
        assert css[6] == '    text-align: right;'
        assert css[7] == '    border: 1px solid black;'
        assert css[8] == '    font-weight: bold;'
        assert css[9] == '}'
        assert css[10] == 'td {'
        assert css[11] == '    padding: 2px;'
        assert css[12] == '    text-align: right;'
        assert css[13] == '    border: 1px solid black;'
        assert css[14] == '}'
        assert css[15] == '</style>'
        slist = np.array([(1, 2.34, 'five'),
                          (2, 3.456, 'seven'),
                          (3, -4.5678, 'nine')],
                          dtype=[('a', 'i4'), ('bb', 'f8'), ('ccc', 'S5')])
        lines, css = struct_print(slist, silent=True, ddigit=3)
        assert lines[0] == 'a bb         ccc  '
        assert lines[1] == '- ---------- -----'
        assert lines[2] == '1       2.34 five '
        assert lines[3] == '2       3.46 seven'
        assert lines[4] == '3      -4.57 nine '
        assert len(css) == 0
        lines, css = struct_print(slist, silent=True,
>           filename=self.struct_print_file)

/usr/lib/python2.7/dist-packages/pydl/pydlutils/tests/test_misc.py:188: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

array = array([(1, 2.34, 'five'), (2, 3.456, 'seven'), (3, -4.5678, 'nine')], 
      dtype=[('a', '<i4'), ('bb', '<f8'), ('ccc', 'S5')])
filename = '/usr/lib/python2.7/dist-packages/pydl/pydlutils/tests/t/struct_print.txt'
formatcodes = {'a': '{0:1d}', 'bb': '{1:14.7g}', 'ccc': '{2:5s}'}
alias = {'a': 'a', 'bb': 'bb', 'ccc': 'ccc'}, fdigit = 5, ddigit = 7
html = False, no_head = False, silent = True

    def struct_print(array, filename=None, formatcodes=None, alias=None,
                     fdigit=5, ddigit=7, html=False, no_head=False,
                     silent=False):
        """Print a NumPy record array (analogous to an IDL structure) in a
        nice way.
    
        Parameters
        ----------
        array : :class:`numpy.ndarray`
            A record array to print.
        filename : :class:`str` or file-like, optional
            If supplied, write to this file.
        formatcodes : :class:`dict`, optional
            If supplied, use explicit format for certain columns.
        alias : :class:`dict`, optional
            If supplied, use this mapping of record array column names to printed
            column names.
        fdigit : :class:`int`, optional
            Width of 32-bit floating point columns, default 5.
        ddigit : :class:`int`, optional
            Width of 64-bit floating point columns, default 7.
        html : :class:`bool`, optional
            If ``True``, print an html table.
        no_head : :class:`bool`, optional
            If ``True``, *don't* print a header line.
        silent : :class:`bool`, optional
            If ``True``, do not print the table, just return it.
    
        Returns
        -------
        :func:`tuple`
            A tuple containing a list of the lines in the table.  If `html` is
            ``True``, also returns a list of lines of CSS for formatting the
            table.
    
        Examples
        --------
        >>> import numpy as np
        >>> from pydl.pydlutils.misc import struct_print
        >>> struct_print(np.array([(1,2.34,'five'),(2,3.456,'seven'),(3,4.5678,'nine')],dtype=[('a','i4'),('bb','f4'),('ccc','S5')]),silent=True)
        (['a bb          ccc  ', '- ----------- -----', '1        2.34 five ', '2       3.456 seven', '3      4.5678 nine '], [])
        """
        if html:
            headstart = '<tr><th>'
            headsep = '</th><th>'
            headend = '</th></tr>'
            colstart = '<tr><td>'
            colsep = '</td><td>'
            colend = '</td></tr>'
            css = ['<style type="text/css">',
                'table {',
                '    border-collapse: collapse;',
                '}',
                'th {',
                '    padding: 2px;',
                '    text-align: right;',
                '    border: 1px solid black;',
                '    font-weight: bold;',
                '}',
                'td {',
                '    padding: 2px;',
                '    text-align: right;',
                '    border: 1px solid black;',
                '}',
                '</style>']
        else:
            headstart = ''
            headsep = ' '
            headend = ''
            colstart = ''
            colsep = ' '
            colend = ''
            css = list()
        #
        # Alias should be a dictionary that maps structure names to column names
        #
        if alias is None:
            #
            # Create a dummy alias dictionary
            #
            alias = dict(list(zip(array.dtype.names, array.dtype.names)))
        else:
            #
            # Fill in any missing values of the alias dictionary
            #
            for tag in array.dtype.names:
                if tag not in alias:
                    alias[tag] = tag
        #
        # Formatcodes allows an override for certain columns.
        #
        if formatcodes is None:
            formatcodes = dict()
        #
        # This dictionary will hold the number of characters in each column
        #
        nchar = dict()
        #
        # Construct format codes for each column
        #
        for k, tag in enumerate(array.dtype.names):
            if tag in formatcodes:
                thiscode = formatcodes[tag]
                thisn = len(thiscode.format(array[tag][0]))
            else:
                d = array.dtype.fields[tag][0]
                if d.kind == 'i' or d.kind == 'u':
                    thisn = max(max(len(str(array[tag].min())),
                                len(str(array[tag].max()))), len(tag))
                    thiscode = "{{{0:d}:{1:d}d}}".format(k, thisn)
                elif d.kind == 'f':
                    if d.itemsize == 8:
                        prec = ddigit
                    else:
                        prec = fdigit
                    thisn = prec + 6
                    if array[tag].min() < 0:
                        thisn += 1
                    thiscode = "{{{0:d}:{1:d}.{2:d}g}}".format(k, thisn, prec)
                elif d.kind == 'S' or d.kind == 'U':
                    thisn = max(d.itemsize, len(tag))
                    thiscode = "{{{0:d}:{1:d}s}}".format(k, thisn)
                else:
                    raise PydlutilsException(
                            "Unsupported kind: {0}".format(d.kind))
                formatcodes[tag] = thiscode
            nchar[tag] = thisn
        #
        # Start building an array of lines
        #
        lines = list()
        #
        # Construct header lines
        #
        if html:
            lines.append('<table>')
            hdr1 = (headstart + headsep.join([alias[tag]
                    for tag in array.dtype.names]) + headend)
            lines.append(hdr1)
        else:
            if not no_head:
                hdr1 = (headstart + headsep.join([("{{0:{0:d}s}}".format(
                        nchar[tag])).format(alias[tag])
                        for tag in array.dtype.names]) + headend)
                hdr2 = (headstart + headsep.join(['-' * nchar[tag]
                        for tag in array.dtype.names]) + headend)
                lines.append(hdr1)
                lines.append(hdr2)
        #
        # Create a format string for the data from the individual format codes
        #
        rowformat = (colstart + colsep.join([formatcodes[tag]
                    for tag in array.dtype.names]) + colend)
        for k in range(array.size):
            lines.append(rowformat.format(
                        *([decode_mixed(l) for l in array[k].tolist()])))
        if html:
            lines.append('</table>')
        f = None   # This variable will store a file handle
        close_file = False
        if filename is not None:
            if hasattr(filename, 'write'):
                f = filename
            else:
>               f = open(filename, 'w')
E               IOError: [Errno 13] Permission denied: '/usr/lib/python2.7/dist-packages/pydl/pydlutils/tests/t/struct_print.txt'

/usr/lib/python2.7/dist-packages/pydl/pydlutils/misc.py:390: IOError

and

_______________________ TestYanny.test_write_table_yanny _______________________

self = <pydl.pydlutils.tests.test_yanny.TestYanny object at 0x7fd44c6aed10>

    def test_write_table_yanny(self):
        """Test writing an astropy Table to yanny.
            """
        filename = self.data('table.par')
        a = [1, 4, 5]
        b = [2.0, 5.0, 8.2]
        c = [b'x', b'y', b'z']
        t = Table([a, b, c], names=('a', 'b', 'c'),
                  meta={'name': 'first table'})
>       t.write(filename, tablename='test')

/usr/lib/python2.7/dist-packages/pydl/pydlutils/tests/test_yanny.py:126: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/usr/lib/python2.7/dist-packages/astropy/table/table.py:2349: in write
    io_registry.write(self, *args, **kwargs)
/usr/lib/python2.7/dist-packages/astropy/io/registry.py:403: in write
    writer(data, *args, **kwargs)
/usr/lib/python2.7/dist-packages/pydl/pydlutils/yanny.py:1339: in write_table_yanny
    hdr=hdr, comments='Table')
/usr/lib/python2.7/dist-packages/pydl/pydlutils/yanny.py:1231: in write_ndarray_to_yanny
    par.write(filename, comments=comments)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = 
newfile = '/usr/lib/python2.7/dist-packages/pydl/pydlutils/tests/t/table.par'
comments = '# Table\n'

    def write(self, newfile=None, comments=None):
        """Write a yanny object to a file.
    
            This assumes that the filename used to create the object was not that
            of a pre-existing file.  If a file of the same name is detected,
            this method will *not* attempt to overwrite it, but will print a warning.
            This also assumes that the special 'symbols' key has been properly
            created.  This will not necessarily make the file very human-readable,
            especially if the data lines are long.  If the name of a new file is
            given, it will write to the new file (assuming it doesn't exist).
            If the writing is successful, the data in the object will be updated.
    
            Parameters
            ----------
            newfile : :class:`str`, optional
                The name of the file to write.
            comments : :class:`str` or :class:`list` of :class:`str`, optional
                Comments that will be placed at the head of the file.  If a
                single string is passed, it will be written out verbatim, although
                a '#' character will be added if it does not already have one.
                If a list of strings is passed, comment characters will be added
                and the strings will be joined together.
            """
        if newfile is None:
            if len(self.filename) > 0:
                newfile = self.filename
            else:
                raise ValueError("No filename specified!")
        if os.access(newfile, os.F_OK):
            raise PydlutilsException(
                  "{0} exists, aborting write!".format(newfile))
        if comments is None:
            basefile = os.path.basename(newfile)
            timestamp = datetime.datetime.utcnow().strftime(
                        '%Y-%m-%d %H:%M:%S UTC')
            comments = "#\n# {0}\n#\n# Created by pydl.pydlutils.yanny.yanny\n#\n# {1}\n#\n".format(basefile, timestamp)
        else:
            if isinstance(comments, six.string_types):
                if not comments.startswith('#'):
                    comments = '# ' + comments
                if not comments.endswith('\n'):
                    comments += '\n'
            else:
                comments = ("\n".join(["# {0}".format(c) for c in comments]) +
                            "\n")
        contents = "#%yanny\n" + comments
        #
        # Print any key/value pairs
        #
        for key in self.pairs():
            contents += "{0} {1}\n".format(key, self[key])
        #
        # Print out enum definitions
        #
        if len(self._symbols['enum']) > 0:
            contents += "\n" + "\n\n".join(self._symbols['enum']) + "\n"
        #
        # Print out structure definitions
        #
        if len(self._symbols['struct']) > 0:
            contents += "\n" + "\n\n".join(self._symbols['struct']) + "\n"
        contents += "\n"
        #
        # Print out the data tables
        #
        for sym in self.tables():
            columns = self.columns(sym)
            for k in range(self.size(sym)):
                line = list()
                line.append(sym)
                for col in columns:
                    if self.isarray(sym, col):
                        datum = ('{' + ' '.join([self.protect(x)
                                 for x in self[sym][col][k]]) + '}')
                    else:
                        datum = self.protect(self[sym][col][k])
                    line.append(datum)
                contents += "{0}\n".format(' '.join(line))
        #
        # Actually write the data to file
        #
>       with open(newfile, 'w') as f:
E       IOError: [Errno 13] Permission denied: '/usr/lib/python2.7/dist-packages/pydl/pydlutils/tests/t/table.par'

/usr/lib/python2.7/dist-packages/pydl/pydlutils/yanny.py:926: IOError

spheregroup failing simple example

The spheregroup module in pydl is crashing at multiple places. The first one (based on the master branch version of the repository) is reproduced below, which can be easily fixed by changing "warning" to "warnings". However, the subsequent crashes (with the same example) happen at increasingly deeper parts of the code. Any help would be greatly appreciated!

import numpy as np
from pydl.pydlutils.spheregroup import spheregroup
ra = np.random.uniform(0,1,100)
dec = np.random.uniform(0,1,100)
gg = spheregroup(ra,dec,10.0/3600.0)
---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-7-91139cb89eb3> in <module>()
----> 1 gg = spheregroup(ra,dec,10.0/3600.0)

/usr/local/lib/python2.7/site-packages/pydl-0.4.0.dev377-py2.7.egg/pydl/pydlutils/spheregroup/spheregroup.pyc in spheregroup(ra, dec, linklength, chunksize)
     37     from .. import PydlutilsUserWarning
     38     from numpy import zeros
---> 39     from warning import warn
     40     npoints = ra.size
     41     if npoints == 1:

ImportError: No module named warning

Custom sdist command is not fully compatible with build

The setup.py file declares a custom sdist command, which is fine when using python setup.py sdist, but not when using python -m build. The net result is that setup.py is sometimes left out of the distribution file(s). Probably adding setup.py to MANIFEST.in would fix this, but this needs to be tested.

Infrastructure updates notice from Astropy Project

Hello from Astropy Project!

The following updates to your package might be necessary to ensure compatibility with the latest stable version of Astropy:

  • MPLBACKEND is now set to Agg in ci-helpers, packages expecting interactive plotting should override it in .travis.yml
  • Astropy 3.1 is not compatible with Numpy <1.13 versions. If you want to keep testing these older Numpy versions, please use ASTROPY_VERSION=3.0 or ASTROPY_VERSION=LTS in your Travis CI matrix.
  • Add sphinx-astropy as a package dependency if you are using astropy-helpers 3.1 or later. Otherwise, your documentation build (e.g., on ReadTheDocs) might fail.
  • If you are using six that is bundled with Astropy, please consider using the standalone six package instead.

If these are no longer applicable to you, please close the issue.

This is an automated issue for packages that opted in for automated astropy-helpers update. If this is opened in error, please let @pllim know.

xref astropy/astropy-tools#108

combine1fiber memory issue

The combine1fiber function is failing with a MemoryError when combining the spectra of two fibers:

import numpy as np
import pyfits as pf 
from pydl.pydlspec2d.spec2d import combine1fiber

#in this case, fiberFile1='spec-3683-55178-0016.fits' and fiberFile2='spec-3683-55178-0024.fits'

fluxArray=pf.open(fiberFile1)[1].data.field('flux')
loglamArray=pf.open(fiberFile1)[1].data.field('loglam')
ivarArray = pf.open(fiberFile1)[1].data.field('ivar')


nextFlux = pf.open(fiberFile2)[1].data.field('flux')
nextLoglam = pf.open(fiberFile2)[1].data.field('loglam')
nextIvar = pf.open(fiberFile2)[1].data.field('ivar')


newloglam=3.775+np.arange(4000)*1e-4
inloglam=np.array([loglamArray,nextLoglam])
objflux=np.array([fluxArray, nextFlux])
objivar=np.array([ivarArray, nextIvar])


fluxArray, ivarArray = combine1fiber(inloglam=inloglam, objflux=objflux, newloglam=newloglam, objivar=objivar, verbose=True)

---------------------------------------------------------------------------------------------------------------------------------
Traceback (most recent call last):
  File "plt_plate.py", line 53, in <module>
    fluxArray, ivarArray = combine1fiber(inloglam=loglams, objflux=fluxes, newloglam=loglamArray, objivar=ivars, verbose=True)
  File "/global/common/edison/contrib/desi/edison/hpcports_gnu-8.2/pydl-0.3.0_f30e8006-8.2/lib/python2.7/site-packages/pydl-0.3.0-py2.7.egg/pydl/pydlspec2d/spec2d/co
mbine1fiber.py", line 105, in combine1fiber
    wavesort = inloglam[isort]
MemoryError
srun: error: nid00076: task 0: Exited with exit code 1
srun: Terminating job step 881290.0

inloglam shape = (2, 4620)
objfluxes shape = (2, 4620)
objvar shape = (2, 4620)

I didn't think the arrays involved were large enough to cause memory issues, but I can't get the function to coadd any two fiber spectra. Any help would be appreciated! Thank you.

I/O shape conflict in pydl.goddard.astro.airtovac

When providing airtovac with a multi-dimensional array, I get the following shape casting conflict:

from pydl.goddard.astro import airtovac
import numpy
wave = numpy.arange(1000,dtype=float).reshape(2,500)+5000
airtovac(wave)

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/pydl/goddard/astro.py", line 29, in airtovac
vacuum = zeros(air.size, dtype=air.dtype) + air
ValueError: operands could not be broadcast together with shapes (1000,) (2,500)

I think this can be solved by changing air.size to air.shape

Failure on 32-bits architecture with Python 3

On a 32-bits architecture, with Python 3.5.2, I obtain the following failure:

=================================== FAILURES ===================================
___________________________ [doctest] pydl.uniq.uniq ___________________________
031 
032     References
033     ----------
034     http://www.exelisvis.com/docs/UNIQ.html
035 
036     Examples
037     --------
038     >>> import numpy as np
039     >>> from pydl import uniq
040     >>> uniq(np.sort(np.array([ 1, 2, 3, 1, 5, 6, 1, 7, 3, 2, 5, 9, 11, 1 ])))
Expected:
    array([ 3,  5,  7,  9, 10, 11, 12, 13])
Got:
    array([ 3,  5,  7,  9, 10, 11, 12, 13], dtype=int32)

Some tests fail on 32 bits architectures

When running tests on a 32-bits architecture, some tests fail with the following trace:

=================================== FAILURES ===================================
__________________ [doctest] pydl.pydlutils.misc.djs_laxisgen __________________
050 
051     Notes
052     -----
053     For two or more dimensions, there is no difference between this routine
054     and :func:`~pydl.pydlutils.misc.djs_laxisnum`.
055 
056     Examples
057     --------
058     >>> from pydl.pydlutils.misc import djs_laxisgen
059     >>> djs_laxisgen([4,4])
Differences (unified diff with -expected +actual):
    @@ -2,3 +2,3 @@
            [1, 1, 1, 1],
            [2, 2, 2, 2],
    -       [3, 3, 3, 3]], dtype=int32)
    +       [3, 3, 3, 3]])

pydl/pydlutils/misc.py:59: DocTestFailure
__________________ [doctest] pydl.pydlutils.misc.djs_laxisnum __________________
092 
093     Notes
094     -----
095     For two or more dimensions, there is no difference between this routine
096     and :func:`~pydl.pydlutils.misc.djs_laxisgen`.
097 
098     Examples
099     --------
100     >>> from pydl.pydlutils.misc import djs_laxisnum
101     >>> djs_laxisnum([4,4])
Differences (unified diff with -expected +actual):
    @@ -2,3 +2,3 @@
            [1, 1, 1, 1],
            [2, 2, 2, 2],
    -       [3, 3, 3, 3]], dtype=int32)
    +       [3, 3, 3, 3]])

pydl/pydlutils/misc.py:101: DocTestFailure
___________________ [doctest] pydl.pydlutils.sdss.sdss_objid ___________________
326         out of bounds.
327 
328     Notes
329     -----
330     firstField flag never set.
331 
332     Examples
333     --------
334     >>> from pydl.pydlutils.sdss import sdss_objid
335     >>> sdss_objid(3704,3,91,146)
Expected:
    array([1237661382772195474])
Got:
    array([1237661382772195474], dtype=int64)

pydl/pydlutils/sdss.py:335: DocTestFailure
___________________________ TestSDSS.test_sdss_objid ___________________________

self = <pydl.pydlutils.tests.test_sdss.TestSDSS object at 0xb398fa50>

    def test_sdss_objid(self):
        assert sdss_objid(3704, 3, 91, 146) == 1237661382772195474
        run = np.array([3704, 1000])
        camcol = np.array([3, 6])
        field = np.array([91, 77])
        obj = np.array([146, 123])
>       assert (np.array([1237661382772195474, 1237649770790322299]) ==
                sdss_objid(run, camcol, field, obj)).all()
E       assert <built-in method all of numpy.ndarray object at 0xb3b00ef8>()
E        +  where <built-in method all of numpy.ndarray object at 0xb3b00ef8> = array([123766..., dtype=int64) == array([1237645..., dtype=int64)
E           Use -v to get the full diff.all

pydl/pydlutils/tests/test_sdss.py:132: AssertionError
_______________________ TestYanny.test_write_table_yanny _______________________

self = <pydl.pydlutils.tests.test_yanny.TestYanny object at 0xb3960490>

    def test_write_table_yanny(self):
        """Test writing an astropy Table to yanny.
            """
        filename = self.data('table.par')
        a = [1, 4, 5]
        b = [2.0, 5.0, 8.2]
        c = [b'x', b'y', b'z']
        t = Table([a, b, c], names=('a', 'b', 'c'),
                  meta={'name': 'first table'})
        t.write(filename, tablename='test')
        par1 = yanny(filename)
        par2 = yanny(self.data('test_table.par'))
>       assert par1 == par2
E       assert #%yanny\n# Tab...\nTEST 5 8.2 z\n == #%yanny\n# Tabl...\nTEST 5 8.2 z\n
E         Omitting 1 identical items, use -v to show
E         Differing items:
E         {'TEST': rec.array([(1, 2.0, 'x'), (4, 5.0, 'y'), (5, 8.2, 'z')], \n          dtype=[('a', '<i4'), ('b', '<f8'), ('c', 'S1')])} != {'TEST': rec.array([(1L, 2.0, 'x'), (4L, 5.0, 'y'), (5L, 8.2, 'z')], \n          dtype=[('a', '<i8'), ('b', '<f8'), ('c', 'S1')])}
E         Use -v to get the full diff

pydl/pydlutils/tests/test_yanny.py:129: AssertionError

pydl Mangle issue(s)

There seem to be a few issues in pydl.pydlutils.mangle, and I think I've traced it to cap_distance.

In cap_distance, if you supply xyz points (ncol = 3), line 401 is:
xyz = point
when it should be:
xyz = points

In the next steps, when the dot product is taken between the positions and the cap centers, it should be taken with the xyz positions, and not points. Line 404 reads:
dotprod = np.dot(points, x)
when it should be:
dotprod = np.dot(xyz, x)

(Apologies if my understanding is incorrect and the errors are due to misuse rather than real issues, I'm fairly new to python!)

objid not unique?

Hi,

Should the ObjID unwrap to a unique run, rerun, etc? I've run into some examples where it doesn't. I'm specifically working with ObjIDs from DR7. Here's an example where I get the same unwrap result with different objIDs.

from pydl.photoop.photoobj import unwrap_objid
print(unwrap_objid(numpy.array([587722983912112587], dtype=int))[0])
print(unwrap_objid(numpy.array([587722984180548043], dtype=int))[0])

(1, 40, 752, 5, 618, 459)
(1, 40, 752, 5, 618, 459)

Thanks,
Kyle

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.