Executive summary: My goal for this project was to collect data from various chocolate makers and distributors in France, and eventually build a comprehensive dataset about the chocolate online market in France.
Methodology:
- Web scraping: I scraped product data from several French e-commerce websites (chocolate section), using Python scrapy and requests libraries.
- Python: I build an additional script to:
- do some minor transformations to the .csv files resulting from web scraping (data cleaning)
- automatically load the resulting dataframes in BigQuery using the bigquery.client
- GCP BigQuery: I created an account and an empty project in GCP.
- dbt: I created a dbt Cloud project that is connected to my BigQuery project. In this dbt project, I created 3 sections:
- Bronze (staging): raw data, with few modifications.
- Silver (transformations): the raw data with some transformations (new columns, filters, etc)
- Gold (final): final datasets that will be used for analysis. Thanks to dbt, I could 'export' the SQL views and tables to GCP BigQuery. I also managed to create the data lineage, QoD tests and documentation. All dbt modifications were saved thanks to dbt-Github.
- PowerBI: Finally, I could plug my Gold final datasets to Power BI to create a visual overview of the market.
Final output
- A comprehensive dataset about the French chocolates online market
- A visual Power BI dashboard
Go further To learn more about the project, you may read my non technical article here: https://margot-marchais-maurice.webflow.io/chocolate-french-market
Technical learnings: I did this project to help me acquire new skills such as: build a datawharehouse in GCP Bigquery, automatically feed this DWH with scraped data thanks to a Python script, learn how to use dbt (data transformations, tests and docs generation),... It also made me refresh my web scraping skills (scrapy and requests libraries).
Aperçu du dashboard:
![2024-05-06_11h58_36](https://private-user-images.githubusercontent.com/116331323/328144412-8afef720-91af-49b1-bdbd-2d606fff29b1.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTcyMDYxMTEsIm5iZiI6MTcxNzIwNTgxMSwicGF0aCI6Ii8xMTYzMzEzMjMvMzI4MTQ0NDEyLThhZmVmNzIwLTkxYWYtNDliMS1iZGJkLTJkNjA2ZmZmMjliMS5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjAxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYwMVQwMTM2NTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1lYWFhMDg5NDBmMzc4M2FiMTFjMGM1YTYzZTMyMzEwOTQ0NmIzZWMyYmZhNmU3OWQ1NTU5ZGQzY2ZkMzViNmU4JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.nHPRNybzf_ui6DlDW5IJ5r2tpyj7xZQy87myuiqtnJI)
![2024-05-06_11h58_45](https://private-user-images.githubusercontent.com/116331323/328144433-d09f8884-0728-45a4-a6f6-54ff96ddae48.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTcyMDYxMTEsIm5iZiI6MTcxNzIwNTgxMSwicGF0aCI6Ii8xMTYzMzEzMjMvMzI4MTQ0NDMzLWQwOWY4ODg0LTA3MjgtNDVhNC1hNmY2LTU0ZmY5NmRkYWU0OC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjAxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYwMVQwMTM2NTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT01MTRiYzQ2N2ZkNmFjYWM1NDRkMzJhMzQ5YjFkNTJhNWU2OGZhY2FjYWU1NjY1OGQ0ZTYzMTk4OThiMTRhYzcyJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.4ckxVJCiO8ixdz7xNiGSeURDCtssXoLmPlWwti819y0)
![2024-05-06_12h39_09](https://private-user-images.githubusercontent.com/116331323/328153592-9b4aad28-33bc-4190-8543-211e1949cb60.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTcyMDYxMTEsIm5iZiI6MTcxNzIwNTgxMSwicGF0aCI6Ii8xMTYzMzEzMjMvMzI4MTUzNTkyLTliNGFhZDI4LTMzYmMtNDE5MC04NTQzLTIxMWUxOTQ5Y2I2MC5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjAxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYwMVQwMTM2NTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT0xZDk2NzNmNjVlY2I3MjI1YzBiZDMwZDc0MmZlMDFiNGU2ZTZhNmQ5OTY5MWYyMGVhMDUwNDU3MzlkMjNjNWNjJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.PkhJNfkQHNZKtAywF3Npa_9pnTjPwycRqKIcy5j9o-8)
Brands positioning:
![2024-05-06_12h39_36](https://private-user-images.githubusercontent.com/116331323/328153671-66df8493-448f-46a8-96c0-5f7b0c15c862.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTcyMDYxMTEsIm5iZiI6MTcxNzIwNTgxMSwicGF0aCI6Ii8xMTYzMzEzMjMvMzI4MTUzNjcxLTY2ZGY4NDkzLTQ0OGYtNDZhOC05NmMwLTVmN2IwYzE1Yzg2Mi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjAxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYwMVQwMTM2NTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1iMDE5OTZiYjI1Y2Q2ZjJjZjVlYzIxNzA5OGNkYzc3YjA1YzYxNWZmNWIyOTQzNmIzNzhjOTRiNGE2OWIyMzRlJlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.k9LaHZ_FkL-cIEDl6fcKahkzGUkCHHmkFPZBok1j538)
![2024-05-06_11h59_27](https://private-user-images.githubusercontent.com/116331323/328144596-49dbc481-3ce8-4508-86c2-2d33599c6fef.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTcyMDYxMTEsIm5iZiI6MTcxNzIwNTgxMSwicGF0aCI6Ii8xMTYzMzEzMjMvMzI4MTQ0NTk2LTQ5ZGJjNDgxLTNjZTgtNDUwOC04NmMyLTJkMzM1OTljNmZlZi5wbmc_WC1BbXotQWxnb3JpdGhtPUFXUzQtSE1BQy1TSEEyNTYmWC1BbXotQ3JlZGVudGlhbD1BS0lBVkNPRFlMU0E1M1BRSzRaQSUyRjIwMjQwNjAxJTJGdXMtZWFzdC0xJTJGczMlMkZhd3M0X3JlcXVlc3QmWC1BbXotRGF0ZT0yMDI0MDYwMVQwMTM2NTFaJlgtQW16LUV4cGlyZXM9MzAwJlgtQW16LVNpZ25hdHVyZT1jM2M1NjBkZWMwZWI0YmRiMjgzZjMxNTgwMjZjZjMyNjBiNjg4MTg1M2E2NjE2NzA3NzBlNjcyYWQ5YTc2NTc1JlgtQW16LVNpZ25lZEhlYWRlcnM9aG9zdCZhY3Rvcl9pZD0wJmtleV9pZD0wJnJlcG9faWQ9MCJ9.3kaeEUconfQOcWN9nq_ijDqjLMyBl9VGker3OzlCM5Y)