Light

nicholaskajoh / devsearch Goto Github PK

View Code? Open in Web Editor NEW

54.0 6.0 14.0 37 KB

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

Python 60.67% CSS 2.24% HTML 35.08% Dockerfile 1.21% Shell 0.80%

search-engine search python flask scrapy crawler spider pagerank tf-idf mongodb

devsearch's Introduction

devsearch

A web search engine built with Python which uses TF-IDF and PageRank to sort search results.

Stack

Flask (Python 3)
Scrapy
LXML
MongoEngine (MongoDB)
Bootstrap 4

Requirements

Docker
Docker Compose

Setup

Install Docker and Docker Compose.
Clone or download this repo.
Create a .env file from .env.example.
Run docker-compose up.

Crawling

Update the SPIDER_ALLOWED_DOMAINS variable in .env with domains you want the spider to crawl.
Add at least one url to the crawl_list collection (in MongoDB) for the spider to start with.
Run docker-compose run web flask crawl to crawl new web pages.
You can add the --recrawl option to update pages already crawled: docker-compose run web flask crawl --recrawl True.

Indexing

To index crawled pages, run docker-compose run web flask index.
To compute TFIDF, run the following one after the other:
- docker-compose run web flask idf
- docker-compose run web flask tfidf
To compute PageRank, run docker-compose run web flask rank.
To compute page-word score, run docker-compose run web flask score.

Deploy

Create a .env.secret file from .env.secret.example.
Run docker-compose -f docker-compose.prod.yml up --build -d.

devsearch's People

Contributors

Stargazers

Watchers

Forkers

viktortat guangyongxing amirunpri2018 nnorukaemeka dvoineu elhawlir zanachka yahyaghani jareducherek codingoscar man-made-code mlashcorp grayoj yikengboy

devsearch's Issues

SPIDER_ALLOWED_DOMAINS not auto append to crawl_list when first run crawler

def __init__(self, recrawl='no'):
    if app.config['SPIDER_ALLOWED_DOMAINS'] != None:
        self.allowed_domains = app.config['SPIDER_ALLOWED_DOMAINS']

    self.start_urls = []
    is_crawled = recrawl.lower() in ['y', 'yes', 't', 'true', '1']
    crawl_list = CrawlList \
        .objects(is_crawled=is_crawled) \
        .limit(app.config['CLOSESPIDER_PAGECOUNT']) \
        .order_by('updated_at')
    for link in crawl_list:
        self.start_urls.append(link.url)

first search error

File "/www/devsearch/views.py", line 19, in
@app.route('/search')
def search():
start_time = time.time()

# get autocomplete data
autocomplete_data = [qo.q for qo in Query.objects.limit(250).order_by('-frequency')]

q = request.args.get('q', default=None)
p = int(request.args.get('p', default=1))
pages = None
pages_count = 0

RuntimeError: generator raised StopIteration

Only support English Index， multiple language support

pip install --user flask python-dotenv mongoengine scrapy jieba

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.