A web crawler to import today's news articles from NHK-Easy into LingQ.
Home Page: https://www.lingq.com/learn/ja/web/#/course/266730
A web crawler to fetch today's top news articles from the NHK easy page.
I built this so that I can easily import the articles into LingQ, including the image and audio.
Here are the courses containing them:
The program relies on the Selenium Chrome Driver which needs to be in the root project directory. Therefore, it cannot be run by a headless server. Remember to always use the latest version of the driver.
More to come ...
All browser activity is now headless.
Next step is to move everything to the cloud and then schedule it to run automatically from there.
Exception in thread "main" java.lang.NoClassDefFoundError: kotlin/jvm/internal/I ntrinsics
at me.avo.kumo.ApplicationKt.main(Application.kt)
Caused by: java.lang.ClassNotFoundException: kotlin.jvm.internal.Intrinsics
at java.net.URLClassLoader.findClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
at sun.misc.Launcher$AppClassLoader.loadClass(Unknown Source)
at java.lang.ClassLoader.loadClass(Unknown Source)
... 1 more
Retrieving articles can be done with headless chrome instead of selenium.
Example of writing the dom to file:
"C:/Program Files (x86)/Google/Chrome/Application/chrome.exe" --headless --disable-gpu --dump-dom https://www3.nhk.or.jp/news/easy/ > out.txt
19:52:24.012 [main] ERROR Kumo - error
java.lang.Exception: There was a problem with the MainPage Main_2018-05-30.html. Likely because the page didn't load correctly. Try reloading.
at me.avo.kumo.nhk.pages.MainPage.extractHeadline(MainPage.kt:31) ~[nhk-easy-crawler-1.2.jar:?]
at me.avo.kumo.nhk.pages.MainPage.getNewsList(MainPage.kt:20) ~[nhk-easy-crawler-1.2.jar:?]
at me.avo.kumo.nhk.pages.MainPage.get(MainPage.kt:16) ~[nhk-easy-crawler-1.2.jar:?]
at me.avo.kumo.nhk.Crawler.fetchArticles(Crawler.kt:41) ~[nhk-easy-crawler-1.2.jar:?]
at me.avo.kumo.nhk.Crawler.fetchAndImport(Crawler.kt:23) ~[nhk-easy-crawler-1.2.jar:?]
at me.avo.kumo.ApplicationKt.start(Application.kt:24) [nhk-easy-crawler-1.2.jar:?]
at me.avo.kumo.ApplicationKt.main(Application.kt:11) [nhk-easy-crawler-1.2.jar:?]
Caused by: kotlin.KotlinNullPointerException
at me.avo.kumo.util.ExtensionsKt.getFirstByTag(Extensions.kt:14) ~[nhk-easy-crawler-1.2.jar:?]
at me.avo.kumo.nhk.pages.MainPage.extractHeadline(MainPage.kt:23) ~[nhk-easy-crawler-1.2.jar:?]
... 6 more
The site has gotten a new layout, which breaks some of the logic.
Make a new branch for the old layout.
java.lang.IllegalStateException: body.getElementById("topnewslist") must not be null
Some articles cause problems because some of their data is missing.
For example 1 segment may be missing from the audio stream.
In such cases it should be possible to temporarily ignore an Article, so that the load can proceed with the others.
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.