Background: the British Council is looking into distributing an offline app that could help it share resources currently available on its website.

The website is https://www.teachingenglish.org.uk/ It is based off Drupal.

Question: How long would it take to build a scraper and produce a quality zim file of it? Ultimately the content would not be integrated into the Kiwix library but only distributed via a custom app, so it would be rather important that the end product's navigation be similar to that experienced on a mobile device.

Bonus question (I): would it help in building the zim file if we had access to their backend?
Bonus question (II): can we estimate already the size of the corresponding zim file (ballpark figure)?

Having come across https://web.dev/web-bundles/ , Web Bundles look like an ideal technology for packaging ZIM files with a lightweight reader (such as Kiwix JS). Excerpt from the description:

Web Bundles have multiple unique features:

• Encapsulates multiple pages, enabling bundling of a complete website into a single file
• Enables executable JavaScript, unlike MHTML
• Uses HTTP Variants to do content negotiation, which enables internationalization with the Accept-Language header even if the bundle is used offline
• Loads in the context of its origin when cryptographically signed by its publisher
• Loads nearly instantly when served locally

These features open multiple scenarios. One common scenario is the ability to build a self-contained web app that's easy to share and usable without an internet connection. For example, say you're on an airplane from Tokyo to San Francisco with your friend. You don't like the in-flight entertainment...

The fix to dark mode has just turned most of the app into "white mode". Pages viewed, etc. are now black text on a white background. Settings menu dark but stored pages "white".

Release on 8 May 2020 on Apple AppStore. This is a NEW release (8/May 2020 TODAY) where the release notes claim iOS Release v 1.11.1 Dark Mode Fix #244 is fixed BUT IT IS NOT (so don't blindly close valid reports please!).

The AppStore version number of the fix release is 1.11.1 (can I help it if it's the same version number as the last one!).

If you don't want to sort bugs being reported then I'll give-up trying to help but I'm trying to help here being closed off without anything being checked!

Kiwix is compressing image files like wikipedia mobile website.

Yesterday I looked for this article (thoracodorsal nerve) in online wikipedia after I changed the setting to be desktop instead of mobile, and I discovered the difference.

The wikipedia desktop image is very much better than mobile wikipedia or kiwix files "zim"

Mobile version or kiwix picture example:

Wikipedia desktop picture (the same image but high quality):

Please provide us a medical wikipedia with full sized images 🙏

So it looks to me like WARC files implement most of what you want out of the ZIM file format, but there's a healthier third-party ecosystem.

Any thoughts on the possibility of kiwix using WARC files instead of ZIM files some day in the (distant) future?

English wikipedia releases seem to happen irregularly

https://dumps.wikimedia.org/other/kiwix/zim/wikipedia/

wikipedia_en_all_2016-05.zim                       17-May-2016 08:17         56911306910
wikipedia_en_all_2016-12.zim                       27-Dec-2016 18:11         62695819637
wikipedia_en_all_nopic_2017-01.zim                 07-Jan-2017 05:05         20180173311
wikipedia_en_all_nopic_2017-08.zim                 14-Sep-2017 10:12         21471834746
wikipedia_en_all_novid_2017-07.zim                 10-Aug-2017 21:33         64048847318
wikipedia_en_all_novid_2017-08.zim                 30-Aug-2017 18:40         64549076329

any chance we could get Kiwix dumps on a regular schedule?

(eg yearly in january, or quarterly Jan/Apr/Jul/Oct)

All previous reports have been marked as closed.

THIS BUG HAS NOT BEEN FIXED WHERE THE RELEASE NOTES CLAIM IT HAS BEEN FIXED.

I've never come across a "community" who has no interest in fixing reported bugs. The closed reports either need to be re-opened or you might as well give-up on this project as closing reports of bugs you claim are fixed BUT ARE NOT FIXED IN THE LATEST RELEASE CLAIMING IT IS FIXED.

IOf you are not interested in fixing bugs then just say you don't want bugs reported (or bugs that are not fixed identified as not fixed) and I'll stop bothering to try and help.

Hello.
Please make a completely portable Kiwix so that it can run on clean Windows without any add-ons installed in it (NET Framework, Java, etc.).
Thank you in advance.

The fix to dark mode has just turned most of the app into "white mode". Pages viewed, etc. are now black text on a white background. Settings menu dark but stored pages "white".

Release on 8 May 2020 on Apple AppStore.

Sometimes I open Russian Wikipedia or some other zim, and then cannot open a different zim: the zim currently open loads every time for many attempts. Kiwix 2.2 on Android. I usually have no problem with the middle of the screen, but have had this one with Kiwix since the beginning (one to one-and-half years).

I created user https://wiki.kiwix.org/wiki/User:Ynikitenko, but then I decided to change the name and created another user.

How the first user (with non-confirmed email) could be deleted? Could you please delete him?

I tried to ask for this in slack, but it wasn't done there. I don't know where else could I ask about that (maybe you should write about that in https://wiki.kiwix.org/wiki/Help? To say where to ask about wiki issues.) Thanks.

This rules should permit a unified workflow about all our repos.

It should normalize:

• The relationship between Tickets and PRs
• The relationship between PRs and git branches
• The process of review/merging
• The commit and merging approaches (re-evaluate commitstyle.md)

I can't open the wikipedia content download page. Its not working.

https://wiki.kiwix.org/wiki/Content_in_all_languages

Hi there,
I downloaded wikipedia_en_all_nopic_2020_04.zim (39.3G) and tried to put it into a SD card which does not hold files larger than 4G. So, according to my kiwix app help I splitted it with split --bytes and got the files .zimaa to .zimaj and copied them succesfully to the SD card (same folder as other zims)
But when I go to my kiwix app and update local zims on 'get content', I cannot see the new wiki. Mi app is running in Android (version 2.1 build 45)
I can format the SD card for a format that accepts >4G files, but I'd prefer not to.

Thanks a lot, great app!

http://reportabug.kiwix.org/ (that is mentioned in the wiki) is a redirect to this CONTRIBUTING.md.

The file looks like a boilerplate and together with the empty issue list makes the project appear unmaintained. It mentions the very relevant https://github.com/kiwix through "https://github.com/kiwix/<repository-name>/issues", but that part does not look read-worthy because it looks auto-generated.

Hello,

I'm wondering what happens to the work on incremental updates performed during GSoC 2013 summarised here. I do not find any follow up on the work performed at that time.

At the same time, it seems that the incremental updates are still not available (and it's an issue for me in France, downloading 40G every year is still difficult nowadays in many rural areas, so I could only imagine what is the situation in other countries).

I found that an alternative for implementing incremental updates seems to be situated at the protocol level with zsync but it still seems to be only an idea.

On my side, I have another idea I tried to PoC. It is based on the fact that a lot of entries are indeed not altered between two versions of the same archive (images, probably already detected by something like zsync) and for others only very small changes are done (and could hence be easily compressed with bsdiff, not by zsync due to the fact that entries are already compressed with LZMA with usually a different cluster location).

I began the implementation of this PoC and first figures are very promising (a 4x reduction at the very least, i.e. an incremental archive could be 4 times smaller than the full new version of the archive). But before spending more time on this PoC (which was fun to develop anyway) I would like to have more information from your side.

If I could help on this incremental updates topic, I would be more than happy.

There's no button for retry downloading so I have to uninstall/install Kiwix app.
Could you please add the feature for that?

Hi,

I'm a Linux user. I have downloaded the latest "wikipedia_en_all_novid_2018-10.zim" from the https://wiki.kiwix.org. But I got "Unable to load '/home/***/wikipedia_en_all_novid_2018-10.zim'. Are you sure this is a ZIM file?" when I want to open it with kiwix-0.9-linux-x86_64.

I check the md5 and it was correct. So I try other files, such as "wikipedia_zh_all_novid_2018-07"or"wikisource_en_all_novid_2018-10",and they all work good.

I try to use the newest version for kiwix. I tried kiwix-0.10-linux-x86_64 ,but it still not working.I also tryed kiwix-desktop_x86_64_2.0-beta2.appimage, but the terminal showed:

./kiwix-desktop_x86_64_2.0-beta2.appimage: /lib/x86_64-linux-gnu/libz.so.1: version `ZLIB_1.2.9' not found (required by /tmp/.mount_kiwix-VbywUs/usr/bin/../lib/libpng16.so.16)
./kiwix-desktop_x86_64_2.0-beta2.appimage: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.25' not found (required by /tmp/.mount_kiwix-VbywUs/usr/bin/../lib/libsystemd.so.0)"

I know a little about appimage. I used to create one on an Ubuntu18.04 and I find it could not run on Ubuntu16.04. But the one created on Ubuntu16.04 can run well in Ubuntu18.04. Maybe here is the same reason.

At last, I succeded to load this file with the chrome-extension of Kiwix. But it is not convenient to open an zim file in this way. I find some issues about this problem on Github and the fourm, but it seems that this problem is still not fixed, at least on Linux.

Something maybe useful:

OS:Ubuntu16.04 LTS x64
Tryed : kiwix-0.9-linux-x86_64,kiwix-0.10-x86_84,kiwix-desktop_x86_64_2.0-beta2.appimage,kiwix-chrome_2.5.0
Succeded : kiwix-chrome_2.5.0
Chrome version:Version 71.0.3578.98 (Official Build) (64-bit)

Sorry for trouble you.

Dear Kiwix Team,
I just downloaded the Version 2018-11 (German, novid) by Bitorrent.
Some Articles of this Version do not contain pictures.
Search e.g. for Myanmar, Kuba, Spanien, etc.
This does not happen in Version 2018-06.
Can you provide any idea or help?
Best regards,
Stephan

I'm running into an error with the following zim files:
wikipedia_en_all_novid_2018-10.zim
wikipedia_de_all_novid_2018-06.zim
wikipedia_de_all_novid_2018-11.zim

The error: error reading zim-file header

Aside: wikipedia_fr_all_novid_2018-10.zim works just fine.

I'm using the linux version of the reader https://download.kiwix.org/bin/kiwix-linux-x86_64.tar.bz2

The English all maxi is at this point one year old. What is the frequency of updates?

In updating these a manual process?

I downloaded a recent English wiktionary (created 2020-01-17).

I see the section Translations for many words, but I can't find translations themselves (even for such widespread words as 'cup'). This didn't work for two kiwix browsers I used, including the most recent one from flathub.

For older wiktionaries it worked, but I can't download and check them now (and not sure which ones I have to download).

Synology devices seem to be ARM devices, we should use our ARM compiled binary and then build the package (config? UI) around it.

PROBLEM
Many .zim files, eg for Wikipedia, are huge. If you wish to update your wikipedia zim, you must re-download everything. This takes time, uses lots of bandwidth, means that new torrents need to be started frequently, and fragements people seeding torrents across versions.

PROPOSAL
Wikipedia releases an annual zim, eg 2019. That is torrented. Then, there are say four updates per year, which arrive Spring Summmer, Autumn Winter. These updates would include the new edits and added material. Something like zsync would be used to update the downloaded torrent to the more recent version.

This would allow an accumulation throughtout the year of people seeding the same torrent, the 2019 one, and at the same time allow people to be up to date without having to re-download the entire wikipedia zim.

Following discussion on openzim/warc2zim#48, it would be interesting to create a kolibri2zim tool to convert content package made by Learning Equality's Kolibri into ZIM files.

Kolibri Library already has a lot of interesting content, including Aflatoun and Khan Academy for which our ZIMs are outdated and for which we have no scraper capable of updating.

The technical details for accessing content are detailed on that ticket. Main idea is to use their API to list the available channels and packages. Individual content sets (we will bundle multiple) are offline-friendly media files and custom HTML files.

We are working on warc2zim, a tool able to create ZIM file from a WARC file. These ZIM files integrates a JS player relying on service-worker. You can find an example of file here:
http://tmp.kiwix.org/warc2zim.zim

This ZIM file works fine with kiwix-serve but not on the following ports:

• Kiwix JS - kiwix/kiwix-js#644
• Kiwix Desktop (Linux/Windows) - kiwix/kiwix-desktop#487
• Kiwix Desktop macOS (iOS?) - kiwix/kiwix-apple#290
• Kiwix Android - kiwix/kiwix-android#2222

I see that this page is being used for data issues.

There is a complete set of English Gutenberg books on kiwix.org, which is great, but it is also prohibitively big (50+ GB).

But there has also been released a 'best of' Gutenberg CD. It would be a great option for smaller devices, and would certainly be appreciated by many people.

http://www.gutenberg.org/wiki/Gutenberg%3AThe_CD_and_DVD_Project

I would have made the ZIM file myself from the HTML/text files, but unfortunately I am not in the position to do so at the moment.

Thank you.

Sent from a disposable GitHub account.

Hi there, I've Downloaded the zim file from http://download.kiwix.org/zim/wikipedia/ but neither my PC nor my android phone will read it. Just getting the message "Unable to load 'E:\wikipedia_en_all_novid_2018-06.zim'. Are you sure this is a ZIM file?"
Any ideas why this would be?

Not sure if this is the right place to post this, but I noticed that the latest French Wiktionary ZIM (01-2018) has been posted to the Wikipedia directory on http://download.kiwix.org/zim/ instead of the Wiktionary directory.

Under within the Submitting Code and Documentation Changes section of contributing. "Project Guidelines" links to https://github.com/kiwix/overview/blob/master/guidelines/project_guidelines.md which does not exist.

We still do not have project guidelines for all of the projects hosted in our GitHub Organization, which new repositories should follow during their creation.

Considering the the context it may be purposeful but I would assume in that case that making the text bold would be sufficient.

I have windows 10 OS and downloaded the most recent versions of kiwix and also of the Wikipedia zim file that has pictures and no videos (around 70-80GB). I have tried opening the zim file but it just says that is can not open and asks whether I'm sure if it is a zim file. The Wikipedia zim was downloaded through utorrent, so I wouldn't think it would be corrupted. Get back to me if you need any more information.

Hi,

I notice that there is no RSS available (or at least not discoverable) for Kiwix Blog that would be great to have it and shouldn't require too much effort.

Thanks

Is it possible to use this on a USB stick or similar as a portable app, with its downloaded wikis and all? It would be really helpful for my emergency USB i keep with my supplies. Thank you!

Hi i am on a iPhone XS 13.1.1 i can't download the newest German Wiki 07.08.2019 (37,02GB) zim, all other are downloading normal. But by this .zim the status remains in "Queued"

The translatable message Android.ui.search history says "Search history" in English. It's not documented. In English, "Search history" can mean both "history of search queries" and "search in the history of previously visited pages". To translate correctly, this needs to be documented.

If you look at the list of current translations at https://translatewiki.net/w/i.php?title=Special:Translations&message=Kiwix%3AAndroid.ui.search+history , you'll see that translators to different languages understood it differently. For example, in French it's "Rechercher dans l’historique" and in Esperanto it's "Serĉi historion", which means "search in history", while in Spanish it's "Historial de búsquedas" and in Polish it's "Historia wyszukiwania", which is "history of searches".

Please document this message. (And all the rest, while you're at it, but this one is particularly ambiguous.)

You can do it most easily by editing the wiki page https://translatewiki.net/wiki/Kiwix:Android.ui.search_history/qqq .

Thanks!

There are two strings, in the "Online" tab of the Library, that keep remaining in my system's language (Italian), no matter what language I choose in Kiwix's settings: "Selected languages:" and "Other languages". It only happens on that page, in the one where I can select the languages they are correctly localized.

A app called Kiwix, created by “Kiwix Team” is available in F-Droid and I wonder if someone could get two other apps by “Kiwix Team” to the F-Droid repo?? Or release the source code, that F-Droid workers can use to include the apps in F-Droid?
The names of the apps are “Wikivoyage” and “Chemistry & Physics simulations”.

App-Link: https://play.google.com/store/apps/details?id=org.kiwix.kiwixcustomwikivoyage 2

App-Link:https://play.google.com/store/apps/details?id=org.kiwix.kiwixcustomphet

I have downloaded kiwix-desktop_windows_x64_2.0-rc4 for windows 10, and start kiwix-desktop.exe successfully. However, when I open file wikipedia_en_all_maxi_2018_10.zim, a popup window shows invalid magic number.

when I enter in the search field "ju 52"
or when I enter direct the url
http://localhost:8787/wikipedia_de_all_novid_2018-11/A/Junkers_Ju_52_3m.html

I get

Fehler: Umleitungsfehler "Die aufgerufene Website leitet die Anfrage so um, dass sie nie beendet werden kann."

translated
Error: Redirect Error "Die called website redirects requests in a never ending loop."

The following url works but is another article (1m instead of 3m)
http://localhost:8787/wikipedia_de_all_novid_2018-11/A/Junkers_Ju_52_1m.html

kiwix-serve of kiwixtools 1.2.1 has the bug with

wikipedia_de_all_novid_2018-11 - in new zim format
wikipedia_de_all_novid_2018-06 - in new zim format
and also in
wikipedia_de_all_novid_2017-08 - in OLD zim format

The bug does not occur in the current android kiwix app and also not
in the current kiwix mozilla plugin 2.5.0, and also not in kiwix-serve of kiwix-0.9,
which understood only the old zim format.

The Wiktionary (en) packages seems to be outdated
wiktionary (English) en 1.4G 2017-08 all nopic
wiktionary (English) en 4.2G 2018-06 all novid

But all the versions of the Simple packages are from 2019
wiktionary (English) en 45M 2019-09 simple all maxi
wiktionary (English) en 15M 2019-09 simple all nopic
wiktionary (English) en 46M 2019-04 simple all novid

https://wiki.kiwix.org/wiki/Content_in_all_languages

Please notice that for all those packages the language listed is: en
When for the last three it should be: simple
Is not the same English than Simple(en)

I have the feeling the recently ZIM files created got the Simple(en) version of the Wiktionary as the English version.

Could you please check your Wiktionary scraper?

Pues apenas soy un usuario principiante y viéndola por un tiempo, menor de un mes, me di cuenta que las imágenes eran necesarias, pero no vi una opción de descarga para estas :(
Y mi opinión es que sería muy util tener una opción en las páginas que diga "Descargar imágenes para esta página (si están disponibles)" pero no sé si haya algo similar o si lo estén pensando para proyectos futuros.

Hi all! Thanks so much for your incredible work making offline archival software and collecting content.

I recently set up my own Wikipedia.org mirror and created some instructions here, https://github.com/pirate/wikipedia-mirror, and now that it's all working smoothly I'd love to contribute back to Kiwix somehow. I made a one-time donation on https://www.kiwix.org/en/support-us/, but for recurring donations I prefer to manage them all via Patreon.

It would be awesome if ya'll had a Patreon page where users can donate monthly instead of having to use a one-off form on the website.

I've installed the hotspot using launcher.exe and not changed any details whatsoever. When I boot my Pi it asks for Kiwix login. I use the default details (admin, admin-password) I get the message Login Incorrect. Also tried no login or password, get the same message.

(as postet in sourceforge)

On an iPhone6sPlus 128GB with 81GB free space and iOS 11.2 (no beta), it is impossible to download any of the 15GB library-packages. They are greyed out leaving me with the message:
“You cannot start downloading, because your device does not have enough free space for this book.”
This is obviously wrong: on the iPhone under settings->info it is stated that there are 81,17GB of free space.

I've tried to import the wikipedia_en_all_novid_2018-10.zim file I download to my desktop into kiwix but each time I try the app crashes. Any work arounds?

Neither kiwix-desktop.exe nor kiwix-serve in kiwix-desktop_windows_x64_2.0-rc3 work with wikipedia_en_all_maxi_2020-08.zim. Kiwix-serve reports "Unable to add the ZIM file ... to the internal library".

kiwix-serve kiwix-tools_win-i686-3.1.2-3 works fine.

Isn't zim back compatible?

Context: contact is distributing e-books and examination papers to people in areas with low connectivity (South Africa / Uganda).

He's asked for a zim file (can be done via Nautilus) that can either be distributed or also queried via a whatsapp bot (they are using wati.io). The way it's been described the bot would offer a list of contents (files) as menu items, and based on user choice would send the corresponding file in return.

We've seen people use Kiwix-serve and feed it into a whatsapp bot already, though they were using PickyAssist for the whatsapp end of things. Here's their feedback :
_ "What we did was fetch content from kiwix server, then remove HTML since WhatsApp doesn't understand that. And then we added the little conveniences like allowing a user to read large pages in small chunks and that about it._

_Really the hard part is getting access to WhatsApp and then building a bot in general. After that, its as easy as adding any other functionality to your bot." _

Questions:

1. assuming we are given access to the wati backend, how much work would be needed to plug Kiwix-serve into their whatsapp bot?
2. Wouldn't this scenario only make sense if the bot is used to query more than one zim? (e.g. nautilus-based + wikipedia)
3. If we skip the Kiwix-serve part, can we still build it, and how long would it take?

On https://wiki.kiwix.org/wiki/Software many, if not most of the links are broken
e.g.

https://download.kiwix.org/bin/kiwix-linux-i686.tar.bz2
https://download.kiwix.org/bin/kiwix-linux-x86_64.tar.bz2
https://download.kiwix.org/bin/kiwix-server-arm.tar.bz2

Looking at those links - maybe that's the wrong directory for them ?

1) https://wiki.kiwix.org/wiki/Content_in_all_languages has no link to English 

2) https://wiki.kiwix.org/wiki/Content_in_all_languages lists many dump permutations but has no searchability, sortability, or way to browse in any organized way. It is a cluttered and awkward user experience.
3) The acronyms don't clearly convey content for new users. Short descriptions would help.
=====
Edit. I now see that the languages section at the top is for the page language, not the dump language. When new users come here for content, this may not be immediately apparent.  
====
https://wiki.kiwix.org/wiki/Content 
  is all english. 
How about creating systematically named and organized pages that are language specific? Or else just an interface to select dump by language, text|vid|pic, content, etc.
=====
The tables are arranged oddly so that the download options are listed above and below the main content row.