kims-developergroup / motorbike-crawler Goto Github PK

Spring rector로 구현 해봄 (related PR #30)

1. Document를 가져오는 걸 순차적으로 Blocking 방식으로 가져오던 걸 Non-blocking 방식으로 바꿔서 구현 해봄
2. Document를 크롤링 작업을 구독(Subscription)하고 있다가 작업이 완료되면 저장하는 방식으로 변경

Request - Response latency가 pageNumber 증가와 비례해서 증가함.
Request를 비동기로 전송하고 Response가 오면 처리하는 Listener를 구현하면 좋을 듯.

...(생략)&idx=682096&num=1

num (query parameter) 가 바이크 정보가 렌더링 되는 리스트의 위치를 표시.

파일당 필수 포함 단어(Primary Keyword) + 선택 단어 조합(Optional Keyword) 으로 조합 가능한 경우의 수만큼 쿼리문을 생성한다.

예)
Primay Keyword :서울 맛집
Optional Keyword : {한식, 중식, 양식}

조합 쿼리 : {서울 맛집, 서울 맛집+한식, 서울 맛집+중식 , 서울 맛집+양식, 서울 맛집+한식+중식, .....
서울 맛집+한식+중식+양식}

추가사항 : 단어 조합을 트리 구조로 구성하는것도 의미 있을 듯

2017-11-13 03:01:23.346 ERROR 6048 --- [ main] .d.e.r.s.AbstractElasticsearchRepository : failed to load elasticsearch nodes : org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{AGbEwZZ-R-umZFtFDFqhnA}{127.0.0.1}{127.0.0.1:9200}]
2017-11-13 03:01:23.397 ERROR 6048 --- [ main] .d.e.r.s.AbstractElasticsearchRepository : failed to load elasticsearch nodes : org.elasticsearch.client.transport.NoNodeAvailableException: None of the configured nodes are available: [{#transport#-1}{AGbEwZZ-R-umZFtFDFqhnA}{127.0.0.1}{127.0.0.1:9200}]
2017-11-13 03:01:23.551 WARN 6048 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'requestMappingHandlerAdapter' defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$EnableWebMvcConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Factory method 'requestMappingHandlerAdapter' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mvcContentNegotiationManager' defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$EnableWebMvcConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.accept.ContentNegotiationManager]: Factory method 'mvcContentNegotiationManager' threw exception; nested exception is java.lang.AbstractMethodError

• Save items from paxxo in repository with target url
• Index items by maker and model

Add PaxxoApiClient

• Crawl all of sale items registered in Paxxo (DONE)

Add NaverCafeApiClient

• Search Items with a given query input from Naver Cafe (DONE)

• Crawl Paxxo items with page link(url) (DONE)

Next step
Add PaxxoSalesItemRepository (Done)

• save meta data of items such as maker-id, model-id
• save sales item data with link
• Fix version issue in elasticsearch and spring-data #15

Add NaverCrawlingService

• Crawling Items from Naver Cafe and analyze items, which are type of sales items.

Find common type of sale items from Paxxo and Naver

elasticsearch 5.6 is supported by spring-data-elasticsearch 3.0.3
spring-data-elasticsearch 3.0.3 is compatible spring-boot 2.x.x
Check PR #27

(ref: https://github.com/spring-projects/spring-data-elasticsearch)

2017-11-19 18:48:03.781 WARN 4813 --- [ main] ationConfigEmbeddedWebApplicationContext : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'requestMappingHandlerAdapter' defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$EnableWebMvcConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter]: Factory method 'requestMappingHandlerAdapter' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'mvcContentNegotiationManager' defined in class path resource [org/springframework/boot/autoconfigure/web/WebMvcAutoConfiguration$EnableWebMvcConfiguration.class]: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.web.accept.ContentNegotiationManager]: Factory method 'mvcContentNegotiationManager' threw exception; nested exception is java.lang.AbstractMethodError