View it here: https://roundup.streamlit.app/

The purpose of this project is regularly track and present the most recent working papers in economics. ('Working papers', also known as 'pre-print' papers, present academic research that has not yet been peer-reviewed.) Remotely run via GitHub Actions once daily, this project scrapes data from working paper publishers at 6:40 AM EST, compares newly collected data to a historic database of working papers, and presents only the most recent ones on the project dashboard. The dashboard may be of use for those interested in understanding the most recent active areas of economics research, such as economists, policy-oriented researchers, and students. As of April 2024, the project incorporates data from 21 different sources.

The primary purpose of this repository is to maintain and improve the project dashboard. A detailed summary of the web scraping, GitHub Actions, and StreamLit components of this project follow.

The web scrapers in this project gather eight pieces of information on each economics working paper:

• Title
• Author(s)
• Abstract: A one-paragraph long summary of the paper.
• Date published: As reported by the website of origin. If the paper was posted and re-posted, the most recent date of publication is used.
• estPubDate: Our best estimate for when the paper was posted: Sometimes this may differ from the official publication date by a few days. We define estPubDate as the first date that the paper was encountered by our web scrapers.
• URL: URLs to working paper landing pages, rather than direct links to PDFs, are preferred. However, with some sources, this is not possible, and PDF URLs are provided instead.
• Paper ID number: According to each website's own numbering system.
• Source: Name of the website where the paper was published.

A variety of methods are used to access web-based data, including PDF rendering, direct requests to hidden APIs, use of Selenium to read Java-rendered content, HTML parsing, and more. A high-level summary of the methods used to scrape each website can be found in the Data Sources section of this document.

This project is maintained in an object-oriented format. Each website has a bespoke method, called fetch_data(), designed to scrape it and defined in a website-specific class located in src/scraper/sites. To impose order on the 20+ web scrapers involved in this project, each website-specific scraper class is a child of the GenericScraper abstract base class, defined in src/scraper/generic_scraper.py.

All 20+ websites can be scraped by running the run_scraper.py script, located in the root directory. It instantiates each scraper class sequentially and saves the collected data in uniformly-structured Pandas data frames. It also records whether the scrape was successful or unsuccessful in streamlit/scraper_status.txt. This list of scraped data frames is then combined into one resultant data frame of all scraped working paper entries in a given run.

The data frame of web scrape results is then passed to various methods in the HistoricDataComparer class from src/scraper/data_comparer.py. These methods identify novel (versus previously encountered) data by comparing the newly-scraped paper identifiers to old identifiers. All those that are truly novel are assigned an estimated publication date of the day that they were first identified, and appended to the data/historic-wp-data.csv and data/historic-wp-ids.txt files, which are used in identifying and maintaining a record of data encountered so far.

The web scraping activity in this repository is fully automated through a GitHub Actions project workflow. When .github/workflows/main.yml is activated at 6:40 AM Eastern Standard Time every day, it runs the main script of the project - run_scraper.py - which cycles through each of the 20+ web scraping modules, and then through data comparison modules that identify and store newly-encountered data. The project workflow concludes by committing its changes to data/historic-wp-data.csv and data/historic-wp-ids.txt under the alias actions-user. These commits are given the description, "run (DD/MM/YYYY)", with "DD/MM/YYYY" populated by the day, month, and year the action was initiated.

The streamlit_app.py script produces the project website, which is a user-friendly aggregation of the most recent economics research. The app draws primarily from the data/historic-wp-data.csv file to populate itself with information. StreamLit automatically refreshes its data display roughly every 5 minutes. Thus, within 5 minutes of a daily commit made by GitHub Actions, changes to data/historic-wp-data.csv should be reflected on the website. The app also uses information from streamlit/scraper_status.txt to display active and inactive scrapers in the sidebar.

GitHub Actions are initiated at 6:40 AM EST and take roughly 4-6 minutes to run. With an additional 5 minutes of time built in for StreamLit to update, all newly scraped information should be reflected on the project dashboard by 7:00 AM EST daily.

Websites that are scraped for data, as of April 2024, are:

The web scrapers are run remotely at 6:40 AM EST daily via the project GitHub Actions workflow located in .github/workflows/main.yml. No additional action is required to initiate this process.

However, the web scrapers may also be operated on your local machine. See below for instructions on how to run the project for the first time on your local machine and any subsequent time.

1. Clone the repository:

   git clone https://github.com/lorae/roundup

2. Set your working directory in the repository:

   cd your/path/to/roundup

3. Create a virtual environment:

   python -m venv .venv

4. Source the virtual environment:

   • Using bash/zsh:

     source .venv/bin/activate

   • Using Windows PowerShell:

     .venv/Scripts/activate

5. Install dependencies

   python -m pip install -r requirements.txt

6. Start the script

   python run_scraper.py

7. View results:

   Local unique results will be stored in data/local_scrape_outcomes. Three files are created with each run:

   • YYYY-MM-DD-HHMM-data.csv: Contains all novel working paper entries with metadata.

   • YYYY-MM-DD-HHMM-dashboard.html: Browser-viewable dashboard file with clickable links for each entry.

   • YYYY-MM-DD-HHMM-ids.txt: Unique identifiers of the novel entries, formatted as a Python set.

   Note that "YYYY-MM-DD-HHMM" will be populated with the day, hour and minute that you instantiated the HistoricDataComparer class, which contains the save_results method that produces these files.

   Results will also be appended to data/historic-wp-data.csv and data/historic-wp-ids.txt with each run of run_scraper.py. However, these files are more difficult to comfortably view directly, as they contain all working papers ever encountered by the web scrapers.

   Note that the results in data/local_scrape_outcomes/ are local only, and ignored by GitHub (.gitignore). The interactive project dashboard on https://roundup.streamlit.app will not be updated with results from your local run unless you commit your changes to data/historic-wp-data.csv to the main branch of the repository.

1. Set your working directory into the repository:

   cd your/path/to/roundup

2. Source the virtual environment:

   • Using bash/zsh:

     source .venv/bin/activate

   • Using Windows PowerShell:

     .venv/Scripts/activate

3. Start the script

   python runall.py

4. View results:

   As above.

The schematic below illustrates the basic file structure of the project.

roundup/
│
├── .gitignore
├── README.md # The file you are currently reading
├── run_scraper.py # Main project script
├── requirements.txt # Project dependencies
│
├── src/
│ ├── data_comparer.py # Defines `HistoricDataComparer` class for data comparison and saving
│ └── scraper/ # Web scraping modules
│   ├── generic_scraper.py # Defines `GenericScraper` class
│   ├── external_requests.py # Methods for accessing and parsing remote data
│   └── sites/ # Website-specific scraping modules
|     ├── bea_scraper.py
|     ├── bfi_scraper.py
|     ├── bis_scraper.py
|     ├── boe_scraper.py
|     ├── ecb_scraper.py
|     └── ...
│
├── data/
│ ├── wp_ids.txt # Set of unique IDs for previously encountered data
│ ├── wp_data.csv # Table of previously encountered data
│ └── local_scrape_outcomes/ # Holds results of local scrapes
│   ├── README.md
│   ├── YYYY-MM-DD-HHMM-dashboard.html
│   ├── YYYY-MM-DD-HHMM-data.csv
│   ├── YYYY-MM-DD-HHMM-ids.txt
│   └── ...
│
├──.github/workflows/
│ └── main.yml # Runs `Daily Run` GitHub Actions workflow
│
└── streamlit
  ├── app.py # Creates Streamlit website
  └── scraper_status.txt # Tracks active and inactive web scrapers

This code accesses remote data via network requests. The scripts are programmed to make minimal and spaced-out requests to avoid putting undue load on the servers of the data sources. Our goal is to collect data responsibly without disrupting the services of the websites we access. The collected data is intended for academic and research purposes only.

All efforts have been made to ensure that the scraping activities comply with the terms of service and /robots.txt directives of the websites from which data is being collected. However, terms of service can change, and it is the sole responsibility of users of this code to ensure ongoing compliance. The developers of this project accept no liability for any misuse of the data or any damages that might arise from the use of this code.

By using this code, you agree to abide by these principles and to regularly check the terms of service of the websites you are scraping to ensure compliance.