Data Engineering Workshop 3
One Day workshop on web scraping, extractors and debugging a program.
What will you learn by the end of this workshop?
- By the end of this workshop you will learn how to scrap a website using python
- You will learn how to save the scrapped data in a database.
- You will learn how to run a date range and incremental extractors.
- You will learn how to debug a python program.
- You will learn time profiling and memory profiling.
Schedule
Time | Topics |
---|---|
09:00 - 10:00 | Webscrapping using python |
10:00 - 12:00 | Storing the srapped data in Postgres DB. |
12:00 - 01:00 | Creating Django view to integrate the script |
01:00 - 02:00 | Break |
02:00 - 03:00 | Creating Date Range and Incremental Extractors |
03:00 - 04:30 | Python debugging and profiling |
04:30 - 04:45 | Q & A |
04:45 - 05:00 | Wrapping Up |
Things to Note:
-
Make sure Workshop 2 is completed and you have a complete working project that was build in Workshop 2.
-
If the working project is not ready you may copy the project myworld from the DataEngineering-Workshop2 repository which you have cloned for the previous workshop and paste it in the current directory which you are going to work on for this workshop.
-
You will have to clone DataEngineering-Workshop3 repository for today's workshop. But make sure to create a new folder outside that repository and work on it instead of making all the changes to DataEngineering-Workshop3 directly.