trisix / billionaire-data-pipeline Goto Github PK

A small Apache Airflow project that will be used to collect data about billionaires from Bloomberg and Forbes. While Bloomberg are total bros and their data is in the site header in JSON format, Forbes is a totally different beast: they really don't want to give me their data but Playwright and BeautifulSoup do the job, although it's probably overkill. Both DAGs will periodically scrape the data, clean it with Polars, save the raw data as a Feather/Apache Arrow file and then write all new or updated data into a PostgreSQL database. Still a work in progress and at some point it's also planned to actually do something with the data.