ipums / ipumspy Goto Github PK

I haven't been able to get ipums.wait_for_extract() to work when passing a UsaExtract object (just sits there forever 5 minutes, but you know, that's basically forever), but it does work when I pass an extract number and collection.

To replicate:

from ipumspy import IpumsApiClient, UsaExtract

ipums = IpumsApiClient(USER_API_KEY)


extract = UsaExtract(["us2012b", "us2014b"], ["age", "sex", "farm"],)

ipums.submit_extract(extract=extract)
ipums.wait_for_extract(extract)

I'm trying to install ipumspy. I did it in the past without problem. Are you awhare of this issue?

Collecting ipumspy
  Using cached ipumspy-0.4.0-py3-none-any.whl (68 kB)
Requirement already satisfied: requests[use-chardet-on-py3]<3.0.0,>=2.26.0 in /bbkinghome/mbarrera/miniconda3/lib/python3.10/site-packages (from ipumspy) (2.28.1)
Collecting PyYAML<6.0.0,>=5.4.1
  Using cached PyYAML-5.4.1.tar.gz (175 kB)
  Installing build dependencies ...   Getting requirements to build wheel ... error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [62 lines of output]
      /tmp/pip-build-env-td9_64m_/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
      !!
      
              ********************************************************************************
              The license_file parameter is deprecated, use license_files instead.
      
              By 2023-Oct-30, you need to update your project and remove deprecated calls
              or your builds will no longer be supported.
      
              See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
              ********************************************************************************

Thanks for this package. I am finding it super useful.

I have one small suggestion on this page: https://ipumspy.readthedocs.io/en/latest/variables.html

In this example:

from ipumspy import readers

# read ddi and data
ddi_codebook = readers.read_ipums_ddi([ddi xml file path])
ipums_df = readers.read_microdata(ddi_codebook, [data file path])

The square brackets around [ddi xml file path] and [data file path] make it seem like these functions expect a list. I'd suggest:

ddi_xml_file_path = ...
data_file_path = ...
ddi_codebook = readers.read_ipums_ddi(ddi_xml_file_path)
ipums_df = readers.read_microdata(ddi_codebook, data_file_path)

MVP is to be able to take inputs from the user and end up with downloaded ddi xml codebooks and data files that can be read into pandas using functions in readers.py.

I am having issues downloading ipumspy and all of its associated dependencies with Python 3.11. Most of the required dependencies are outdated, including numpy which is seemingly giving me the most trouble. Any suggestions how I can resolve this conflict in dependencies?

This is not an API function, but a way to read traditional extracts in hierarchical format.

This seems to be a known issue.

Please disregard.

For purged or failed extracts, there will be no files to download. Add functionality to download_extract() to check for that and return a helpful error message

Currently the test to for extract_to_dict() relies on a pickle object in the test fixtures rather than creating an extract object every time. As the pickle format is sometimes updated and older formats may eventually no longer be readable by the pickle module, this may become a sustainability issue in the future. An area for improvement.

In the Get Started documentation for ipumspy, there is a typo in the DOWNLOAD_DIR variable:

ddi_file = list(DOWLOAD_DIR.glob("*.xml"))[0]

DOWNLOAD is missing the "N".

Additionally, the two lines directly below should change "ipumsy" to "readers" as ipumsy does not work and readers is imported at the top of the documentation:

ddi = ipumspy.read_ipums_ddi(ddi_file)
ipums_df = ipumspy.read_microdata(ddi, DOWNLOAD_DIR / ddi.file_description.filename)

This will also make that block consistent with the code block in the Read an IPUMS Extract section.

• Tested on last master branch commit
• Tested on last pypi package version https://pypi.org/project/ipumspy/0.2.1/

Installation of ipumspy requires dependency click<8.0.0,>=7.0.0. This dependency conflicts with recent versions of dtale, jupyter-plotly and maybe with other common data science packages.

It can be troublesome in data science virtual environments to fix this issue post installation.

Would it be possible to update pyproject.toml click = "^7.0.0" dependency to:

[15] click = ">=7.0.0"

Joe pointed out that you can pass a non-usa collection name to UsaExtract() and not get an error, which is fine, but we should probably have a warning or something that clarifies that.

While extract definitions persist indefinitely, actual extract files do not. It would be good to be able to allow a user to re-constitute a previously-created extract and re-download it after IPUMS has stopped retaining the extract data file.

If the user tries to request a non-existent extract number, the API will return:

{
    "status": 404,
    "error": "Not Found"
}

ipumspy currently doesn't correctly detect this and so tries to parse this as if it were an actual extract payload.

Ideally it would detect this and relay the error message to the user.