ipums / ipumspy Goto Github PK
View Code? Open in Web Editor NEWLicense: Mozilla Public License 2.0
License: Mozilla Public License 2.0
I haven't been able to get ipums.wait_for_extract()
to work when passing a UsaExtract object (just sits there forever 5 minutes, but you know, that's basically forever), but it does work when I pass an extract number and collection.
To replicate:
from ipumspy import IpumsApiClient, UsaExtract
ipums = IpumsApiClient(USER_API_KEY)
extract = UsaExtract(["us2012b", "us2014b"], ["age", "sex", "farm"],)
ipums.submit_extract(extract=extract)
ipums.wait_for_extract(extract)
I'm trying to install ipumspy. I did it in the past without problem. Are you awhare of this issue?
Collecting ipumspy
Using cached ipumspy-0.4.0-py3-none-any.whl (68 kB)
Requirement already satisfied: requests[use-chardet-on-py3]<3.0.0,>=2.26.0 in /bbkinghome/mbarrera/miniconda3/lib/python3.10/site-packages (from ipumspy) (2.28.1)
Collecting PyYAML<6.0.0,>=5.4.1
Using cached PyYAML-5.4.1.tar.gz (175 kB)
Installing build dependencies ... Getting requirements to build wheel ... error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [62 lines of output]
/tmp/pip-build-env-td9_64m_/overlay/lib/python3.10/site-packages/setuptools/config/setupcfg.py:293: _DeprecatedConfig: Deprecated config in `setup.cfg`
!!
********************************************************************************
The license_file parameter is deprecated, use license_files instead.
By 2023-Oct-30, you need to update your project and remove deprecated calls
or your builds will no longer be supported.
See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
********************************************************************************
Thanks for this package. I am finding it super useful.
I have one small suggestion on this page: https://ipumspy.readthedocs.io/en/latest/variables.html
In this example:
from ipumspy import readers
# read ddi and data
ddi_codebook = readers.read_ipums_ddi([ddi xml file path])
ipums_df = readers.read_microdata(ddi_codebook, [data file path])
The square brackets around [ddi xml file path]
and [data file path]
make it seem like these functions expect a list. I'd suggest:
ddi_xml_file_path = ...
data_file_path = ...
ddi_codebook = readers.read_ipums_ddi(ddi_xml_file_path)
ipums_df = readers.read_microdata(ddi_codebook, data_file_path)
MVP is to be able to take inputs from the user and end up with downloaded ddi xml codebooks and data files that can be read into pandas using functions in readers.py
.
I am having issues downloading ipumspy and all of its associated dependencies with Python 3.11. Most of the required dependencies are outdated, including numpy which is seemingly giving me the most trouble. Any suggestions how I can resolve this conflict in dependencies?
This is not an API function, but a way to read traditional extracts in hierarchical format.
This seems to be a known issue.
Please disregard.
For purged or failed extracts, there will be no files to download. Add functionality to download_extract()
to check for that and return a helpful error message
Currently the test to for extract_to_dict()
relies on a pickle object in the test fixtures rather than creating an extract object every time. As the pickle format is sometimes updated and older formats may eventually no longer be readable by the pickle
module, this may become a sustainability issue in the future. An area for improvement.
In the Get Started documentation for ipumspy, there is a typo in the DOWNLOAD_DIR variable:
ddi_file = list(DOWLOAD_DIR.glob("*.xml"))[0]
DOWNLOAD is missing the "N".
Additionally, the two lines directly below should change "ipumsy" to "readers" as ipumsy does not work and readers is imported at the top of the documentation:
ddi = ipumspy.read_ipums_ddi(ddi_file)
ipums_df = ipumspy.read_microdata(ddi, DOWNLOAD_DIR / ddi.file_description.filename)
This will also make that block consistent with the code block in the Read an IPUMS Extract section.
Installation of ipumspy
requires dependency click<8.0.0,>=7.0.0
. This dependency conflicts with recent versions of dtale, jupyter-plotly and maybe with other common data science packages.
It can be troublesome in data science virtual environments to fix this issue post installation.
Would it be possible to update pyproject.toml click = "^7.0.0"
dependency to:
[15] click = ">=7.0.0"
Joe pointed out that you can pass a non-usa collection name to UsaExtract()
and not get an error, which is fine, but we should probably have a warning or something that clarifies that.
While extract definitions persist indefinitely, actual extract files do not. It would be good to be able to allow a user to re-constitute a previously-created extract and re-download it after IPUMS has stopped retaining the extract data file.
If the user tries to request a non-existent extract number, the API will return:
{
"status": 404,
"error": "Not Found"
}
ipumspy currently doesn't correctly detect this and so tries to parse this as if it were an actual extract payload.
Ideally it would detect this and relay the error message to the user.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.