Code Monkey home page Code Monkey logo

insurance-company-database-with-jdbc-and-spark-rdd's Introduction

A JDBC client Program / Project

This is a jdbc client project from my early days as a Computer Science student

This programm was created for the fifth semester class Database Management Systems and is the one of the final projects for the class

Description of project

A jdbc application that runs queries in pgAdmin to simulate the functionality of an insurance company's database using Apache Spark RDD for query implementation.

Impementation and Instructions of project

  1. Tip 1: I record the second runtime in order for the buffers to initialize
  2. Tip 2: I use the EXPLAIN command for the execution plan
  3. Tip 3: I run the queries in sequence so that the changes to buffers, parallelism, etc. continue to be active for the next query
  4. Insert the data from the given dataset using proper queries and the command VACUUM FULL
  5. Implement some functionality using proper queries:

a. Find the customer who had the greatest mileage to a specific timestamp
b. Find the average mileage for the last month from all the data
c. Find the monthly sum of mileage for each customer
d. Find the average distance traveled for each customer
e. Find the average mileage per region.

  1. Tip 4: Configure PostgreSQL to use more buffer than your computer's RAM with the command ALTER SYSTEM SET shared_buffers TO '256MB';
  2. Run queries again
  3. Tip 5: Configure PostgreSQL to use all the processing power of your computer with the command max_parallel_workers_per_gather
  4. Create the appropriate indexes in the db to run the queries above faster
  5. Break the dataset into shards / partitions using inheritance between tables using range method
  6. Connect databse with JDBC client in Java using the proper library
  7. Write a Java console application that will calculate some statistics, execute a query in Spark and will use the partitioning

About this project

  • There is a predetermined dataset used for this project
  • The comments to make the code understandable, are within the archives
  • This project was written in Eclipse Java IDE
  • This repository was created to show the variety of the work I did and experience I gained as a student

insurance-company-database-with-jdbc-and-spark-rdd's People

Contributors

peteprattis avatar

Stargazers

 avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.