Light

alexdseo / visualization-app-on-world-events Goto Github PK

View Code? Open in Web Editor NEW

0.0 1.0 1.0 1.5 GB

Jupyter Notebook 6.74% HTML 93.26%

visualization-app-on-world-events's Introduction

Visualization App on World Events

Making some cool maps about what is happening in Ukraine right now.

#SupportUkraine

Workflow

View Google Drive

1. Preprocess images

image_preprocessing.ipynb

Requires:
- Pillow: pip3 install PyMuPDF Pillow
Input: data (see Google Drive for all files; sample on GitHub)
Output: data_resized (see Google Drive for all files; sample on GitHub)

2. Extract text with Optical Character Recognition

text_extraction.ipynb

Requires:
- Pillow: pip3 install PyMuPDF Pillow
- PyTesseract: pip install pytesseract
- Tesseract: brew install tesseract
Input: data_resized (see Google Drive for all files; sample on GitHub)
Output: raw_text (see Google Drive for all files; sample on GitHub)

3. Process text

text_processing.ipynb

Input: raw_text (see Google Drive for all files; sample on GitHub)
Output: cleaned_text (see Google Drive for all files; sample on GitHub)

4. Name Entities Recognition and analysis

Run Named Entity Recognition on GPU:
- Input: cleaned_text (see Google Drive for all files; sample on GitHub)
- Output: charts in Notebook and on website

5. Geolocation

Run Geoparsing on GPU:
- Input: cleaned_text (see Google Drive for all files; sample on GitHub)
- Output: geoparse.csv
Run Dynamic Mapping on GPU:
- Requires Mapbox API token
- Input: geoparse_clean.csv
- Output: world map and Ukraine map in Notebook and on website

6. Web page

See our live website here and interact with our maps!

visualization-app-on-world-events's People

Contributors

Watchers

Forkers

candysan-alive

visualization-app-on-world-events's Issues

4. Named Entities Recognition and Analysis (2)

Plot a distribution of top 20 mentioned PERSON, NORP, ORG, GPE and LOC in the entire dataset.

Create a simple bar plot illustrating the number of mentions.

1. Image preprocessing:

Download the screenshots from:
https://drive.google.com/file/d/1XhV7EUPRAKlzKHYpjR_LN1W3IjP8l-5U/vie
w?usp=sharing

Split and organize the images if needed.

6. Create Dynamic Map

Make two dynamic maps that display the geolocated entities over time (day-by-day):
a) The World map, where we can observe the
activities/mentions of the larger entities, such as countries
b) Ukraine map, to show the smaller entities that appear in the news such as cities.

4. Named Entities Recognition and Analysis (4)

Look at the distributions. Are they different? Can they tell us something about the differences in reporting between the three outlets? Write about this.

7. Host a Webpage

Showcase our work including all entities distribution, dynamical maps, as well as the additional analysis and explanations.

5. Geo-parsing

Perform the geoparsing on GPE and LOC entities. Use Python GeoPy library. Be cognizant of the potential API limits and make sure to optimize for the number of queries.

2. Optical Character Recognition

Use the OCR software/package to perform the OCR on the extracted images.

4. Named Entities Recognition and Analysis (1)

Perform the Named Entity Recognition on the text you extracted. Use SpaCy or NLTK

Get all the: (when using SpaCy)
PERSON, NORP, FAC, ORG, GPE, LOC, PRODUCT, EVENT, DATE and TIME from the text

4. Named Entities Recognition and Analysis (3)

Compare the distributions of the entities between the news sources.

**Note: The Fox News dataset is missing days between March 1st and March 11th. When comparing the news sources, consider this and exclude those dates from CNN and Aljazeera. At the end, we should get 15 bar charts, one for each required entity type and each news source.

3. Text preprocessing

Save the extracted text into a preferable format that will allow you to fetch the
text again when needed. It can be saved as a JSON, or as a DataFrame and then
pickled, or in the set of textual files properly named and organized in the folders, so you know the date and the news source of each text.

Make sure to showcase that script can remove the unnecessary extra text from the menus or the ads.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.