molybdenum-99 / mediawiktory Goto Github PK

Current example:

response = client.
  query.
  generator(categorymembers: {title: 'Category:Countries_in_South_America', limit: 30}).
  prop(revisions: {prop: :content}).
  perform

Imaginary example:

response = client.
  query.
  generator(categorymembers: 'Category:Countries_in_South_America'). # singular values -- sets first key of hash
  limit(30). # limit is common and propagated to all sub-modules having limit parameters
  prop(revisions: :content). # again -- singular value by default is value for first key
  perform

This approach may one time become fragile, but worth trying? Or some another approach?

Older, and, unfortunately, still popular versions of MediaWiki had pretty different API description page, like http://harrypotter.wikia.com/api.php

Should also be parseable (though, probably, would not be that easy).

Docs: https://www.mediawiki.org/wiki/API:Upload
Approach to utilize: https://github.com/lostisland/faraday#advanced-middleware-usage

• Popular wikis shortcuts (copy from infoboxer):

MediaWiktory.wikipedia.query...
MediaWiktory.wikitravel('ru').query....
MediaWiktory.wikia('tardis', 'fr').query...

• Default path to /w/api.php (not always there, so, can't do it uncoditionally), like MediaWiktory::Client.new('https://en.wikipedia.org') — no need to path specification.

As per this docs

Most reasonable default behavior, I assume, is receive content (follow redirects), but there should be an opt-out (for writing bots and other study software, targeting EXACTLY redirects logic).

...for everything.

YARD allows "fake" docs for auto-generated methods, so, we just need to make API docs for all modules clearly but concisely resembling original.

For example: https://en.wikipedia.org/w/api.php?action=help&modules=query%2Bgeosearch
gscoord param is

Format: Latitude and longitude separated by pipe ("|").

Current usage:

api.query.geosearch('50.004444|36.231389')....

Should be:

api.query.geosearch(50.004444, 36.231389)....

Hi!

I want to extract a specific section of a page. How can I do that? Thx for your help!

It's not hard to generate some more code, and allow to have things like:

client.query.params
# => [:titles, :list, :meta, :generator....]
puts client.query.help
# Prints something like this page: https://en.wikipedia.org/w/api.php?action=help&modules=query

With multitude of actions and modules and modules and submodules... Can be REALLY helpful.

Like: https://en.wikipedia.org/w/api.php?action=query&generator=categorymembers&gcmtitle=Category:It%20is%20not%20a%20category&format=json

Currently fails trying to parse this empty response with undefined method 'pages' for nil:NilClass.

Should return response with empty pages list.

Something like

c = MediaWiktory::Wikipedia::Api.new do |b|
  # b is Faraday::Builder, allowing to change underlying web client, add middleware and so on
end

Cookies are important.
Need to find cookies details in MediaWiki docs, then implement! Use faraday-cookie_jar gem or something.

Many popular sites have MediaWiki version lower than latest. Some of them even as old as 1.19 (of 2012!)
Differences between versions can both in available modules and core logic (like token management)

Ideal support for multiple version should consist of:

• multiple generated modules lists, like lib/generated/<version>/*.rb (currently, code is generated from online help at mediawiki.org/w/api.php?action=help; for multiple versions it should be generated from MediaWiki sources, using tags?)
• multiple clients, like MediaWiki::Client_v1_19, loading different modules
• transparent API version discovery on client creation.

Sub-ideal solution should at least tag modules with minimal version they require, and warn user on attempt to use those modules on wikis with lower version.

NB: In older versions of MediaWiki (even in slightly older ones) query continuation logic is much more complex and inconsistent than now.

Only a draft thoughts currently, but I can imagine, for ex., some special QueryClient, which is more "semantic" than thin API wrapper with its query, revisions, prop and so on:

c = QueryClient.new('https://en.wikipedia.org')
c.page('Argentina').wikitext
c.pages('Argentina', 'Bolivia')
c.category('Soth America').count
c.category('Soth America').pages
c.search(prefix: 'List of').count
# ...and so on

Same way, specialized EditClient or AdminWatchClient could be implemented.

From this page many params, like "assert", "origin", "maxage", and so on, should be passed at client creation (and optionally can be rewritten on each request).

Like this:

client = MediaWiktory::Client.
  new('https://en.wikipedia.org', assert: 'user') # default value for each request

client.query.
  ...
  perform(assert: 'bot') # rewriting for concrete action

For example, wikipedia.query.list(prefixsearch: {search: 'List of cities in'}) -- will return key query.prefixsearch, and this key's content should be merged on subsequent calls

Hi,
Not an issue but a question. I'm doing my final project for a coding school and making an app with Rails backend/React frontend. I would like to allow a user to search a song title and return matches based on that search from Mediawiki API. I've been playing with some of the parameters but can't quite hone in on that specific search query. Would also be great if it returned an image with the link. Can anyone point me in the right direction? I've also been messing with the Mediawiki sandbox but also having trouble honing in with query search.
Thanks,
D

w = MediaWiktory::Client.new('https://en.wikipedia.org/w/api.php')
res = w.query.generator(prefixsearch: {search: 'Guardians', limit: 100}).prop(revisions: {prop: :content}, info: {prop: :url}).redirects(true).perform
res.continue!
# ArgumentError: Unknown key in continued page: revisions
#      from /media/storage/work/gems/mediawiktory/lib/mediawiktory/page.rb:32:in `block in merge!'

If you look at raw structure: page 1, page -- you'll notice some strange things, like one page is described as pageid+title, and its content is on continuation page. So, current merging algorithm is not sufficient.

Lots of clients does not require all 100+ actions, so generator for them could probably be "optimized" with command line argument which actions to choose. Less code, less classes required, less memory consumption and so on.

Follow-up of #8

• Not loose information from API:
  • API response: {"error"=>{"code"=>"gcmmissingparam", "info"=>"One of the parameters cmtitle, cmpageid is required", "*"=>"See https://en.wikipedia.org/w/api.php for API usage"}}
  • Response::Error currenty preserves only "info" field
• Maybe try to investigate/parse into errors; for ex., here we have gcm prefix in error code and cm-something params in message. In best case, MediaWiktory could say from it something like: params are missing for generator categorymembers, param list title, pageid and return this information structured. Or maybe it is completely superfluous :)

Currenlty, we have:

• in request: query.generator(:search).namespace(0)
• in response, page description: {..., namespace: 0, ...}

It is not that humane. Potentially, Api should introspect what namespaces current wiki have and provide some constants or shortcuts for more obvious work with them, not their ids.

Introspection could be done on code generation stage.

Mininum:

• gem version;
• Travis CI.

Better:

• code climate;
• gemnasium (dependencies actuality);
• test coverage;
• docs coverage (? with InchCI).

Of course, values in "better" section badges should became reasonable before publishing!

See also here: https://github.com/badges/shields for list of useful badges.

All data-modifying actions require some dance with "tokens". It should be done transparently for user.
See https://www.mediawiki.org/wiki/API:Tokens

Note, that token management have different behavior in:

• versions < 1.20
• version 1.20-1.23
• version 1.24+

See also #9 about version compatibility

NB: Current version of "manual" token management is:

token = api.query.meta(:tokens).response.dig('tokens', 'csrftoken')
response = api.edit.title('Wikipedia:Sandbox').text("Test '''me''', MediaWiktory!").token(token).response

It is, in fact, not that bad, and probably not worth over-automatization at all.

Action should have method for parameter validation, and call it automatically before perform. Possible validations:

• parameter types;
• parameter limitations (like "50 or less" for numerical params), including complex ones, like "50 for users, 500 for bots" (and MediaWiktory user can explicitly say "my code has bots right")
• forcing required params and param combinations, including complex ones (param A or B is required; param A and B are required; param A is required if param B is present, and prohibited otherwise; )

For now -- just a little better:

• actual information about basic usage, roadmap, contributing and so on;
• add License file (and maybe Contributing.md);
• add @smostovoy to list of authors.

No idea about correct API for now, just to remeber how it is done:

https://en.wikipedia.org/w/api.php?action=query&titles=Argentina&prop=pageimages&pilimit=50&format=json&pithumbsize=10000

For example, if you want page content, currently you'll do something like this: pages.first.revisions.first.content['*'], which is kinda awful.

Don't invented, still, what could be done here.
Things should became simpler, but magic should be avoided.

...otherwise just response string should be returned.

If response contains errors key, an error should be raised
If response contains warnings -- do nothing by default, but allow easy access to the warnings (and maybe have mode for printing the warnings?)

From here:

If you want to protect against edit conflicts (which is wise), you also need to get the timestamp of the last revision

MediaWiktory could automate it (check if you still editing last revision), with possible opt-out.

As per https://www.mediawiki.org/wiki/API:Main_page#Identifying_your_client

Comprehensive and "real" usage examples should be written for several use cases:

• useful bot (watching pages; fixing pages in batches);
• information extraction (gathering pages and information about them, or images, or...);
• study (gathering complicated statistical data like "how many edits had been done in last month by unregistered users with splitting by major categories, amount of edit and time of the day");
• alternative client ("casual" usage like "look for page, eventually do small edits")
  ...

Page is one of main concepts in MediaWiki content, and having it as a Hashie is a bit of under-implementation!

Follow-up of #8

• Response#warnings? (or Response#has_warnings?)
• Option to print warnings while performing actions: both for entire client and for concrete action

There should be pretty inspect both for Action and for Response: informative yet concise.

Examples of uncovered cases:

• To specify all values, use *.
• ...A "csrf" token retrieved from action=query&meta=tokens
  • should be probably converted to "Retrieved by api.query.meta(:tokens)"
• Flags: read/write rights
• This module is deprecated in favor of action=query&meta=tokens
• Sortkey to start listing from, as returned by cmprop=sortkey. Can only be used with cmsort=sortkey.

Also would nice to:

• have a links in each module to source docs
• enchanced docs (and probably even shorter apis) for parameter-less modules, there are few.

Currenlty MediaWiktory will fail on attempt to get something from http://en.wikipedia.org because of redirect to httpS. It should be handled transparently.

• Enum values are always converted to symbols
• Parameter expecting list of enums should accept one symbol (like prop: :categories, currently, developer is required to write prop: [:categories], which is dumb)