soumyadeep-github Goto Github PK

adf-spark-notebook-pipeline

A simple pipeline to transform data within Azure Data Factory using Azure Databricks. Although it is written in Scala the same can be replicated in Python.

airflow-tutorial

Tutorial like code for how to deploy airflow using docker and how to use the DockerOperator.

amazon_reviews_scraper

Get all product reviews for a product on Amazon.

assignment-1

This contains an attempt towards analyzing the Black Friday data set from Kaggle.

assignment-2

This repository contains files which were used to transform a set of .txt files such that the date column can be shifted to the front of the given files.

azuredatabricksnotebook

bellings-and-ready-wear

This repository contains a .pbix (power bi desktop) and excel file. The excel file contains sales reports of Bellings and Ready Ware, two retails chains running throughout Australia while the power bi file contains some analysis about the same.

calcite-exploration

calciteparsingexploration

data-analysis

The aim of this project was to analyse given data set and find out if there exists any trends. The data is produced by a tool similar to Google analytics and the dataset is about a website which is an online repository for books.

data-ingestion

The aim of this project is automate data ingestion from flat files like CSV and compressed files GZIP into a database like Postgres. The entire setup is automated using Docker and is pretty fast too as multiprocessing is being used.

data_wrangling_analysis_and_ab_testing_with_sql

Data Wrangling, Analysis and AB Testing with SQL

datastructures_custommodule

dbt-project

django-docker-demo

Basic setup for Django with Docker

ecommerce_scraper_project

Have you ever needed to go through multiple e-commerce websites simultaneously to check the price of the same product? Prices can differ but having a list side by side would be a bonus, wouldn't it? This Flask App can do exactly that.

fastai

The fastai deep learning library, plus lessons and tutorials

fibonacci_dockerized

flipkart_webscraper

Just a barebones program to get into Flipkart scraping.

hands-on-machine-learning

My ML Learning

hashtagcashtag

My Insight Data Engineering Fellowship project. I implemented a big data processing pipeline based on ​lambda architecture​, that aggregates Twitter and US stock market data for user sentiment analysis using open source tools - ​Apache Kafka ​for data ingestions, Apache Spark ​& ​Spark Streaming ​for batch & real-time processing, ​Apache Cassandra f​ or storage, ​Flask​, ​Bootstrap and ​HighCharts f​ or frontend.

hazelcast-platform-training

hive-metastore-docker

Example for article Running Spark 3 with standalone Hive Metastore 3.0

javaprojectscraping

A simple Java project to fetch top 250 movies from IMDB into a CSV file.

learningspark

A repository for learning Apache Spark using Scala.

linkedin_scraper

Scraping data from LinkedIn. The aim was to scrape data off of LinkedIn. The scraping project was solely performed as an experiment and has no other intentions.

loans-data-visualization

This repository contains a loans dataset and a Power BI file to visualize this data.

mining-internet-search

public-finance-researcher-assignment

Assignment for Public Finance Researcher position CivicDataLab

spark-assignment

The aim of this project is to perform analysis on some (car crash) data using PySpark and make the entire process deployable using Docker.