lorae / roundup Goto Github PK

The function append_data_to_historic within the HistoricDataComparer class contains a try-except block. The except is specifically a FileNotFoundError, which can introduce issues if the file is found but does not comport to the data standards assumed in the try block.

Create more graceful, generic error handling and a specific version of error handling when the file exists but does not comport with data standards. Once this is complete, remove the print statements (e.g. "existing_df read", "column order mapped", etc).

Fed_Boston.py is not able to navigate to 2024 publications. Adjust the web scraper accordingly. Explore API options.

Change the way entries display so that hyperlinked title is above a posted pubdate and estimate pubdate, authors, and abstract.

Current ECB 'Abstract' field contains keyword data, like this:

Ensure ECBScraper class gathers correct Abstract data field, and edit existing database for already collected entries.

I suspect it has to do with Selenium.

Should be based on whether the current month is January.

Current scrapers have a mix of logic, but have been migrating toward always scraping the current and former year's entries. This may mean overuse of computing resources.

Abstract data Xpath index is not working. Revisit this script and potentially use an API instead

The following text appeared incorrectly on the StreamLit app:

"Two in five Americans have medical debt, nearly half of whom owe at least $2,500. Concerned by this burden, governments and private donors have undertaken large, high-profile efforts to relieve medical debt. We partnered with RIP Medical Debt to conduct two randomized experiments that relieved medical debt with a face value of $169 million for 83,401 people between 2018 and 2020. We track outcomes using credit reports, collections account data, and a multimodal survey. There are three sets of results. First, we find no impact of debt relief on credit access, utilization, and financial distress on average. Second, we estimate that debt relief causes a moderate but statistically significant reduction in payment of existing medical bills. Third, we find no effect of medical debt relief on mental health on average, with detrimental effects for some groups in pre-registered heterogeneity analysis."

Something is happening to make it italicized unintentionally.

Script appears to not be collecting data. Issue may relate to Chromium driver.

It appears to be cutting date string off - e.g. "Fri, 19 Jan 2" rather than "Fri, 19 Jan 2024"

appears to now have issues collecting author names from website

Add to a new directory called "docs" or something similar. Walk through the process of resolving a broken script and describe common issues.

It would be especially cool if the notebook could show code from specific scripts within the project without copy-pasting the code (e.g. dynamically pulling code from a different file). I am not sure if this concept has an already built-out method, but it would be nice, especially when producing call-outs to explain how the GenericScraper ABC relates to the concrete scraper classes.

Hasn't found a new working paper since Sept 2023

Currently, there are three lists of scraper IDs, each contained in compare.py, streamlit_app.py, or runall.py, which look like the following:

source_order = ['NBER', 'FED-BOARD', 'FED-BOARD-NOTES', 'FED-ATLANTA', 'FED-BOSTON', 'FED-CHICAGO', 'FED-CLEVELAND', 'FED-DALLAS', 'FED-KANSASCITY', 'FED-NEWYORK', 'FED-PHILADELPHIA', 'FED-RICHMOND', 'FED-SANFRANCISCO', 'FED-STLOUIS', 'BEA', 'BFI', 'BIS', 'BOE', 'ECB', 'IMF']

Integrating a new scraper module in the project involves updating all three lists, which is unintuitive and prone to producing bugs.

Resolving this issue would involve creating a global order of sources that can be called by all 3 scripts.

does not have any 2024 entries yet