This is an implementation of an Asynchronous Pubmed Scraper. It allows for controlled concurrency and saves files in .parquet format, optimizing data storage efficiency.

git clone https://github.com/FourierMourier/async_pubmed_scraper.git

conda create --name {env} python=3.10

conda activate {env}

python -m venv venv

windows

venv\Scripts\activate.bat

posix

source venv/bin/activate

cd async_pubmed_scraper
pip install -r "requirements.txt"

There you go

Scraping User Agents List: Before starting the scraping process, you'll need a list of user agents. You can download the user agents list, for example, here or anywhere else you'd like.

To use it, please copy the example config file main.example.yaml to main.yaml, and adjust your paths:

user_agents_list_path: path/to/your/agents_list.txt
# sets the directory path where you'll get the results from pubmed
output_dir: collected_data

Then, you can run main.py.

Optimal Implementation Approach: When it comes to implementation, you might be tempted to use a straightforward approach where you create an asyncio.Task for each individual task, such as checking URLs, and call these tasks as soon as you reach them. For instance, when processing a single page, you might create multiple tasks to process individual URLs representing article pages. However, using this approach, you'll be sharing the same aiohttp.ClientSession across all tasks, and any server error could potentially disrupt the page-wise pipeline:

Alternative Approach for Enhanced Clarity: Alternatively, you can wait for all tasks corresponding to a specific real task (e.g., collecting all pages to process) before moving on to the next step (collecting all URLs to be parsed). In this scenario, the client session object will be associated with the same group of tasks and won't be nested within other tasks, leading to a clearer process flow.

By the way,

1. without the control of concurrency using asyncio.Semaphore it's too likely for you to get banned by pubmed.
2. asyncio.Lock object for the safe session reopening
3. some pages indeed do NOT have abstract content so you might see smth like "...doesn't have abstract-content selected!" in the terminal

This project is licensed under the Apache 2.0 License - see the LICENSE file for details.

This README was created with the assistance of OpenAI's ChatGPT (September 25 Version), a large language model. You can learn more about it here