Go to Installation
Go to Libraries/Frameworks Used
Go to Application Structure
Team Members: Joanna Huang (joannah2), Drshika Asher (drshika2), Brooke Novosad (novosad3), Rainy Yan (yuzheng9)
Team Captain: Brooke Novosad (novosad3)
Team Name: Autumn Lovers
Presentation: Presentation Video
Project Description: Our application allows you to enter any CS410 related search query to find the 10 most relevant course materials (CS410 Lecture Transcripts and Slides) with blurbs to match your query.
-
Have python3 and pip installed: tutorial
-
Clone Repository Locally
HTTPS
$ git clone https://github.com/drshika/CS410-Final-Project.git
SSH
$ git clone [email protected]:drshika/CS410-Final-Project.git
- Install requirements with pip:
$ pip install -r requirements.txt
- Run the app on localhost:
$ python app.py
or
$ flask run
- Open in browser
* Serving Flask app 'app'
* Debug mode: on
WARNING: This is a development server. Do not use it in a production deployment. Use a production WSGI server instead.
* Running on http://127.0.0.1:5000
Paste http://127.0.0.1:5000 (or whatever port address it assigns) into your browser to view the webpage.
- Rank_bm25: this library has the Okapi BM25 ranking function to get the 10 most relevant document names.
- Flask: this is a framework used to make GET and POST requests for the user query and the search results from our backend.
- Numpy: this library was used to sort the outputted list from most to least relevant.
- BeautifulSoup: this library was used to parse the HTML web pages from the CS410 course website to get the documents that users can search for.
- PyPDF2: this library was used to scrape PDFs into text.
- Pytest this was used to test our program
├────── extraction_scripts/ # Scripts used to process raw HTML lecture transcripts and slides
│ ├── get_pdf.py # Scrapes lecture slide PDFs
│ └── get_transcripts.py # Scrapes lecture Transcripts
├───── html_files/ # Raw HTML files from CS410 Coursera Website
├───── lecture_slides/ # CS410 PDF lecture slides
├───── lecture_slides_extractions/ # Text from CS410 lecture slides
├───── lecture_transcripts/ # Transcripts from CS410 lecture videos
├───── static/images # Images used in this repository
├───── project_docs/ # Misc Project Docs
│ ├── Project Progress Report.md
│ ├── Project Proposal.md
│ └── sources.txt # Reference Code Documentation
├───── templates/ # HTML template pages
│ ├── answer.html # Query response page
│ └── home.html # Home page
├───── .gitignore
├───── app.py # Main Flask driver code
├───── ranking_function.py # Utility function to return ranked documents
├───── README.md
├───── test_pytest.py # Tests different queries against hand calulated rankings
└───── requirements.txt # Used to install required python libraries
- Proposal preparation and revision (5hrs) - all
- Project scoping and approach analysis (5hrs) -all
- Collect all documents in CS 410 Coursera (5 hrs) - Brooke, Drshika, Joanna
- Data clean up (tokenization, remove stop words, etc.) (20 hrs - verification by hand included) - Brooke, Rainy, Drshika
- Data processing, analysis, & storage (5 hrs) - Brooke, Rainy
- Website
- Frontend (5hrs) - Drshika
- Backend (5hrs) - Brooke, Rainy, Drshika
- Data routing (10hrs) - Brooke, Rainy, Drshika
- Search algorithms (5hrs) - Brooke, Rainy
- Summarization blurb for each query (5 hrs) - Brooke
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent