Small learning project in how to create a document search / RAG system for articles that are published in arxiv. I decided to everything from scratch for my own curiousity.

There are some environment variables that need to be set for the system to work.

• ARXIV_RAW_DIR - where to store the raw data from the extraction step.
• ARXIV_CLEAN_FP - where to store the cleaned file from the load step.
• ARXIV_DOC_FP - where to store the documents from the transform step.

Raw data is fetched from kaggle using the Kaggle Public API. You will need to get an API token store it at $HOME/.kaggle/kaggle.json. For more information see here under Getting Started: Installation & Authentication.

Code is in data/extraction directory. To run this step, run the data/extraction/extraction.sh script.

The raw JSON downloaded in the previous stepped is transformed into a tabular format and cleaned up, to later be stored as a parquet file.

The relevant articles are filtered out and a column body is created in the format "{title}, written by {authors}, {abstract}" is created. This column is what is used for indexing later.

The vector database implementation can be found in db. This only really implements functionality for a vector database, storage is handled by the client.