• Introduction
• Features
• Demo
• Technologies
• Installation
• File Structure
• Architecture

Welcome to Astronomic Big Data, a groundbreaking project that brings together cutting-edge technologies such as data transfer, web scraping, UI design, Kafka, cloud infrastructure, Docker, socket IO, database management, and API integration. Our ambitious goal is to create a sophisticated data alerting and analytics system that harnesses the power of NoSQL databases and a hybrid cloud computing environment. By providing real-time event monitoring and advanced data exploration capabilities, we empower scientists and researchers to gain valuable insights from the vast realm of astronomical data. Join us in this transformative journey as we push the boundaries of big data processing and revolutionize the way we perceive and utilize astronomical information.

• Data Collection: Astronomic Big Data leverages a sophisticated data collection mechanism that integrates with various astronomical sources, including the NASA API and private databases. Through seamless integration with Kafka and Redis on Upstash (cloud), ElasticSearch on Docker, MongoDB (local), OpenWeather API, and more, the project ensures a comprehensive and up-to-date repository of astronomical information.

• Data Processing: Our project boasts robust data processing pipelines powered by versatile libraries like Axios, Cheerio, and Puppeteer. These pipelines efficiently clean, filter, and transform the collected raw data, ensuring it is organized into a structured and usable format for further analysis.

• Data Visualization: TWith a focus on user-friendly exploration, Astronomic Big Data offers interactive visualizations that unlock the mysteries of astronomical data. Through a thoughtfully designed dashboard, users can seamlessly navigate through different menus like Events, Asteroids, Sun, and Weather. Built upon a solid MVC architecture, the visualizations enable users to gain valuable insights into celestial objects, events, and phenomena.

• Dashboard page
• User profile page
• Events page
• Asteroids page
• Sun page
• Weather page

Dashboard page	Events page	Asteroids page
]

Dashboard page	Sun page	Weather page

The Astronomic Big Data project is primarily developed using JavaScript, EJS, and CSS with the power of Node.js and Express.

• Kafka Apache with Upstash (cloud): The project harnesses Kafka Apache combined with Upstash, a cloud-based Redis service, to facilitate the transfer of random messages generated by the Simulator (producer/consumer). This powerful combination ensures efficient and reliable data streaming between components.

• Elastic Search and Kibana (Docker): Leveraging Docker, the project utilizes Elastic Search and Kibana to manage and visualize astronomical data. The client consumes messages from Kafka and efficiently indexes them into Elastic Search. Kibana provides a user-friendly interface for exploring and analyzing the stored data.

• Redis with Upstash (cloud): To store the Bring Star Catalog (BSC.js) containing the titles and locations of random astronomical messages, Astronomic Big Data employs Redis through Upstash. This lightning-fast in-memory database enhances data retrieval and access speed.

• MongoDB with MongoDB Atlas (cloud): The project integrates MongoDB, coupled with MongoDB Atlas on the cloud, to store weather data fetched through the OpenWeather API. This allows for seamless storage and retrieval of weather-related information.

• NASA API: For fetching up-to-date asteroids data, the project interacts with the NASA API. This integration ensures a rich and comprehensive repository of information about asteroids.

• Web Scraping: To gather HTML data from TheSkyLive website for the sun, Astronomic Big Data uses web scraping techniques. This allows the project to access valuable astronomical data and include it in the dataset for further analysis.

To use Astronomic Big Data, follow these installation steps:

1. Clone the repository:

    git clone -b master https://github.com/AnthonyAssayah/AstronomicBigData
   

2. Install all the required modules

   npm install
   

3. Build and run the docker-compose

   docker-compose -f elasticsearch-kibana-docker-compose.yml -d
   

4. Run the required js files

   node ./Client_ES.js
   node ./simulator.js
   node ./server.js
   

Within the download you'll find the following directories and files:

DASHBOARDWITHWS
  ├── node_modules
  ├── model
  |     ├── asteroids.js
  |     ├── events.js
  |     ├── sunScrapping.js
  |     └── weather.js   
  ├── public
  │   ├── css
  │   ├── fonts
  │   ├── images
  │   └── js
  ├── views
  │   ├── pages
  │   └── partials     
  ├── pages
  ├── docker-compose.yml
  ├── server.js
  ├── package.json
  └── package-lock.json