- Scala
- Play Framework
- Sbt
- Kafka
- Flink
- PostgreSQL
In PostgreSQL create database and user:
postgres=# CREATE DATABASE crawler;
CREATE DATABASE
postgres=# CREATE USER crawler WITH PASSWORD '123';
CREATE ROLE
postgres=# ALTER USER crawler WITH SUPERUSER;
ALTER ROLE
postgres=# GRANT ALL PRIVILEGES ON DATABASE crawler TO crawler;
GRANT
Run Search API
sbt runSearchApiAfter successfully starting the service. Request the URL http://localhost:9000/ from your browser and click the "Apply this script now!"
./bin/zookeeper-server-start.sh config/zookeeper.properties./bin/kafka-server-start.sh ./config/server.properties./bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test-topic./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test-topicAnd send following messages to topic:
>Сэр
>Джонс
>Ваша
>карта
>бита
>привет
>как жизнь
>
This command starts the crawler service to read messages from the Kafka topic via Flink and writes the messages to PostgreSQL.
sbt "runCrawler --topic test-topic --group_id test --zookeeper_connect localhost:2181 --bootstrap_servers localhost:9092 --postgres_host localhost:5432 --postgres_database crawler --postgres_user crawler --postgres_password 123"This service returns the first 10 similar words stored by the crawler, in descending order. Query:
curl http://localhost:9000/?word=джакартаResponse:
карта
джонс
бита
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent