intracer / scalawiki Goto Github PK

View Code? Open in Web Editor NEW

31.0 7.0 12.0 6.68 MB

scalawiki is a MediaWiki client in Scala

License: Apache License 2.0

Scala 98.96% HTML 0.40% Java 0.64%

scala mediawiki mediawiki-api mediawiki-client wikipedia-bot wikipedia-api

scalawiki's Introduction

scalawiki

scalawiki is an experimental MediaWiki client in Scala on early stages of development.

Why another client library for MediaWiki?

I didn't know any Java client that supported generators (fetching properties from articles listed by list query in a single request). JWBF [only recently] (eldur/jwbf#21) got the ability to query more than 1 page at a time.

When Wikipedia sites are real Big Data it is just a show stopper. Fetching information about Wiki Loves Monuments uploads in such ineffective way will take almost a day even for one country, when could be done in several minutes otherwise in batches.

This library uses Scala Futures for easy job parallelization.

Goals

Fully support MediaWiki API
Support different backends - MediaWiki API, xml dumps, MediWiki database. Support copying data between backends (importing and exporting xml dumps to database, storing data retrived by MediaWiki API to xml dumps or database).
Good test coverage

scalawiki's People

Contributors

Stargazers

Watchers

Forkers

whitecaribou bumbury roiviber tuxnani fmcejudo illko ik-wmua gaybro8777 scala-steward ulianahs madal3x saqib-trywe

scalawiki's Issues

Implement batch operations in database backend

Get username from first image or revision, not from page infobox

Write documentation

Detect images that where moved to local Wikipedia due to no FOP

implement proper continuation

mostly done, need tests for complex cases and limits quering

shortLinksBot hangs sometimes

depends on size of the input. does not hang in debug mode though

wlx: investigate and lower memory consumption

socket.io for recent changes

https://www.mediawiki.org/wiki/API:Recent_changes_stream
https://wikitech.wikimedia.org/wiki/RCStream
https://github.com/wandoulabs/spray-socketio

Write xml dumps

use traits for mocking

wlx: slow report with many images

Ideas about statistics and its visualization

Description of currently generated reports

Other WLM tools:

https://tools.wmflabs.org/wlm-stats/ - Uploads per day and per hour - Rankings for countries, uploaders, sizes and resolutions
http://stats.wikilovesmonuments.cl/germany - Country statistics: Top uploaders and Daily uploads, uploads per user and day, Year comparision for Users, Photos and Photos per user
http://tools.wmflabs.org/glamtools/glamorous.php - Number of images used by country
http://tools.wmflabs.org/ptwikis/WLM2015 - Graph of number of uploads by country, Number of Images, Images used in the wikis, Uploaders, Newly registered uploaders
http://tools.wmflabs.org/ptwikis/WLM2015:Ukraine - Images, Images used
in the wikis, Registration for each user in specific country

Other Wikimedia visualizations:

D3 Visualization Gallery

https://github.com/mbostock/d3/wiki/Gallery

Implement some heuristic for broken markup detection

Parse Update Write should not break markup even more if it's already broken (WLX ListFiller affected)

update to play-json 2.5

response caching

via some Kv-store
https://github.com/OpenHFT/Chronicle-Map
https://github.com/jankotek/mapdb/
benchmark: http://jetbrains.github.io/xodus/#features

Cleanup and commit wlx statistics

Support Erfgoed bot db backend

See

VCS: https://gerrit.wikimedia.org/r/#/admin/projects/labs/tools/heritage
Issue tracker: https://phabricator.wikimedia.org/tag/wiki-loves-monuments-database/
ToolLabs: https://tools.wmflabs.org/heritage/api/index.php
Documentation: https://commons.wikimedia.org/wiki/Commons:Monuments_database

Filter contest files by contest dates

Crossbuild for older scala & java

Wikimedia ToolLabs still does not support Java 8, Spark is packaged with Scala 2.10.

try https://github.com/lihaoyi/mainargs

Do not choke exceptions

rethink mwapi query DSL

Case classes for each parameter take to much space when compiled. Also not autocomleted or looked up in the code easily

add limit for number of queried elements

move files between sites

too frequent edits are not saved

Cleanup and commit article contest statistics

Switch from play-json to circe

https://github.com/circe/circe

Motivation:

does not depend on Play release cycle
does not require compatible Play version to be used
does not impose Play requirements such as only Java 8 since Play 2.4

update to Slick 3.0

API:Parameter information parser

https://www.mediawiki.org/wiki/API:Parameter_information

https://en.wikipedia.org/w/api.php?mainmodule=1&modules=query&format=json&action=paraminfo&helpformat=html

Reasons: To autogenerate query and response classes, documentation, check for changes

Create tools database

Reasons: analyze usage scenarios, provide own implementation, compare with alternatives.

WikiTech list thread https://lists.wikimedia.org/pipermail/wikitech-l/2015-October/083560.html

https://tools.wmflabs.org/
https://www.mediawiki.org/wiki/Manual:Extensions
https://www.wikidata.org/wiki/Q6584911
https://tools.wmflabs.org/hay/directory/#/
https://meta.wikimedia.org/wiki/Research:Index
http://seealso.org/
https://de.wikipedia.org/wiki/Benutzer:Atlasowa/edit_history_visualization
https://www.mediawiki.org/wiki/API:Client_code
https://outreach.wikimedia.org/wiki/GLAM/Resources/Tools
https://phabricator.wikimedia.org/T115650
https://phabricator.wikimedia.org/T1238

Mockup: https://meta.wikimedia.org/wiki/User:Ilya/Tool_repository

Put statistic pages and files in categories

Dependency injection

More Scala oriented http://scaldi.org/ or more adopted id Java world https://github.com/google/guice

bintray deprecated

Are there any executables can download directly?

Bintray is deprecated and I can not access it

see if xwiki-commons-blame dependency can be dropped

lz4 compressed dumps

Depends on lz4/lz4-java#21 for interoperability reasons

Configure sweble not to skip comments

Currently WLX ListFiller removes comments

optimize UserContribList

Use generic data aggregation for statistics

Each wlx statistics should support different output formats (wiki, html, csv, json)

handle errors

org.scalawiki.dto.MwException: MediaWiki Error: code: internal_api_error_DBQueryTimeoutError, info: [W10-QApAMEQAAIOVEKIAAABJ] Database query error., params: Map(gcmtitle -> Category:Images_from_Wiki_Loves_Earth_2017_in_Ukraine, format -> json, generator -> categorymembers, iiprop -> timestamp|user|size|metadata, gcmlimit -> 500, gcmnamespace -> 6, rvprop -> ids|content|timestamp|user|userid, prop -> info|revisions|imageinfo, gcmcontinue -> file|d094d095d09dd094d0a0d09ed09bd09ed093d086d0a7d09dd098d09920d09fd090d0a0d09a203133312e4a50470ad094d095d09dd094d0a0d09ed09bd09ed093d086d0a7d09dd098d09920d09fd090d0a0d09a203133312e4a5047|59413889, action -> query, continue -> gcmcontinue||)