This repository serves as documentation for a Formula1 Racing project created using Azure services for data engineering. The project aims to process Formula 1 racing data, create an automated data pipeline, and make the data available for presentation and analysis purposes.

The Formula 1 racing project involves processing data for both driver champions and constructor champions. Points are awarded based on the finishing position in each race, and there are separate championships for drivers and constructors. Pole positions are determined through qualifying races.

The data is sourced from the Ergest Developer API, providing tables such as circuits, races, constructors, drivers, results, pitstops, lap times, and qualifying.

ER Diagram

1. Azure Databricks: For compute using Python & SQL.
2. Azure Data Lake Gen2 (Delta Lake / LakeHouse): For hierarchical storage and utilizing delta tables.
3. Azure Data Factory: For pipeline orchestration for Databricks Notebook.
4. Azure Key Vault: For storing secrets for ADLS credentials used in the Databricks Notebook.

Project Architecture

• Created Azure Data Lake Storage with three containers: raw, processed, presentation.

• Utilized Azure Databricks with a specific cluster configuration.
• Mounted Azure Data Lake using Service Principal for secure access.

• Ingested eight types of files and folders from the raw container.
• Created separate notebooks for ingestion and converted raw data into processed data.

• Used processed data to perform further transformation and analysis.
• Created notebooks for race results, driver standings, constructor standings, and calculated race results.

• Stored data generated from processed notebooks in the presentation container.
• Analyzed and visualized dominant drivers and teams.

• Created Ingest Pipeline, Transform Pipeline, and Process Pipeline.
• Used Azure Trigger for automation.
• Orchestration ensures automated processing of raw and processed containers.

• Utilized tumbling window trigger for triggering based on past date data.
• Customized triggers for specific date ranges.

• Total cost incurred: Rs. 3200, surpassing the Rs. 3000 budget.
• Major costs from Azure Databricks for compute and ADLS for storage.
• Cluster timeout management helped in cost control.