Code Monkey home page Code Monkey logo

kiwix-js's Introduction

Kiwix JS 4 0 Anniversary

Kiwix JS

Kiwix is your gateway for offline access to vast web resources like Wikipedia, Stackexchange, Project Gutenberg, TED Talks, and much more. These resources are packaged as highly compressed ZIM archives available for free from Kiwix. Learn more about the open-source Kiwix project on our main website.

Kiwix JS is our official HTML5/JavaScript version, primarily designed for browser extensions. Grab our free extension from the Mozilla, Chrome, and Edge extension stores. Check out the Store links for more details. We also offer an offline-first Progressive Web App (PWA) version at https://browser-extension.kiwix.org/current/. For a fully featured PWA based on Kiwix JS, visit https://pwa.kiwix.org.

To get started with Kiwix JS, download a free content archive (Usage), select it on your device, and start exploring articles. You can access the entire content of Wikipedia in your language, including images and audiovisual content, offline. Ideal for users with expensive, intermittent, slow, unreliable, or censored internet access. You can even load ZIM archives shared on a USB stick or external hard drive. No internet? No problem!

Kiwix supports all content in the OpenZIM format, including full support for Zimit content (based on the Web Archive format), both Zimit Classic and Zimit v2.0. Note: Some content requires your browser to support Service Workers.

Build Status: Continuous Integration Build Status: Release CodeFactor Licence: GPL v3

Kiwix for Firefox Kiwix for Chrome Kiwix for Edge

Usage

Install "Kiwix JS" from your browser's add-on store. This is the best way to get the extension, because it will be kept up to date automatically. If you would rather not use a store, you can get a file-based version of the extension from http://download.kiwix.org/release/browsers/ (and follow instructions below), but you will have to update this manually.

Alternatively, you can bookmark or install the PWA version from https://browser-extension.kiwix.org/current/ (it will auto-update), or try our dedicated PWA version at https://pwa.kiwix.org. To install the PWA (in Chromium browsers), go to Settings -> Apps -> Install this site as an app.

As mentioned above, the app requires at least one ZIM archive of offline content. You can download one from the in-app library, or else in any browser go to https://library.kiwix.org (this has a nice, graphical interface and a preview of each ZIM archive) or from https://download.kiwix.org/zim/ (a more basic list of archives). You have to download these separately, store them in your filesystem, and manually select them after starting the application (or you can drag-and-drop one into the app).

Zimit-based archives (available from the "zimit" directory on https://download.kiwix.org/zim/, or made yourself with youzim.it), are fully compatible with this reader. For the best experience, you will need a modern browser and to allow the app to run in ServiceWorker mode as an offline-first PWA (see below for an explanation of what this means). If the app is running in any other mode, then only static content will be viewable (if at all). Our sister app https://pwa.kiwix.org has some further support for Zimit archives in older browsers.

Security

Be sure to get your ZIM archives only from a secure source, such as the official Kiwix library. This is because ZIM archives can run dynamic code in your browser. While we do our best to sandbox the ZIM's content, a detemined malicious ZIM could remove the sandbox and redirect the iframe to, say, a phishing Web site. For this reason we now show a Security Warning when you open a ZIM with dynamic content in ServiceWorker mode for the first time. If you do not trust the source of the ZIM, and wish to browser static content safely, then open the ZIM first in Safe Mode before deciding whether to switch to ServiceWorker Mode.

Compatibility

Since the app is written in HTML/JavaScript, it should work in most recent browser engines and many older ones too, depending on the Content Injection mode supported by the specific browser engine. Archives containing dynamic content (most non-Wikimedia archives) work better in ServiceWorker mode (see below), but unfortunately this is not available in many older browsers. If you wish to read such archives, we would suggest that you upgrade to a browser that supports Service Workers (Chrome 58+, Firefox 61+ [not ESR versions], Edge 17+, Safari 11.3+).

Officially supported platforms

Deprecated platforms

These platforms/browsers are deprecated. We still partially test against them, and we'll try to keep compatibility as long as it's not too complicated:

  • Firefox OS >=1.2: needs to be installed manually on the device with WebIDE
  • Microsoft Edge Legacy >=17: no extension available, but bookmark https://browser-extension.kiwix.org or https://pwa.kiwix.org
  • Microsoft Edge Legacy 15-16: needs to run a bundled version of the source code in Safe mode only
  • Microsoft Internet Explorer 11: needs to run a bundled version of the source code in Safe mode only

You can build a bundled version by running npm install and npm run build in the root directory of this repo. Alternatively, a bundled version is served as a web app for testing from https://kiwix.github.io/kiwix-js/dist/ (also available on the gh-pages branch of this repo, under /dist).

Installing signed or unsigned extension files in Chromium

If you need to install Chromium (Chrome or Edge) extension from a file instead of from a Store, e.g. if your browser doesn't support Manifest V3, then you will need to download a signed or unsigned CRX or ZIP file from a relevant directory in https://download.kiwix.org/release/browsers/, or else a nightly version from https://download.kiwix.org/nightly/. Files with mv2 in their filename are in the legacy Manifest V2 format.

To install your CRX or ZIP, open the extension management page in your browser, e.g. chrome://extensions/ or edge://extensions/, and turn on Developer mode. Now, you should be able to drag and drop the ZIP file into this page. Verify the extension is showing in the management page.

Files that we deliver with a .crx file extension are files that have been validated by the Edge or Chrome Stores, and you should be able to install these as "first-class" apps. ZIP files provided in https://download.kiwix.org/release/browsers/, or the ones labelled signed in nightly, are actually signed CRX files that have been renamed with a .zip extension to facilitate downloading and installing them in Chromium browsers. Although signed, you cannot install them as CRX files, because they have not been validated by the Chrome or Edge Stores. For this reason, the browser will periodically ask you if you want to turn off developer-mode extensions. Just choose "ask again in two weeks".

If drag-and-drop is difficult, you can instead unzip the extension ZIP into a folder, and note the location. Then select "Load unpacked" and choose the folder that contains the unzipped extension. To unzip the MV2 files with a utility like 7Zip, you will need to change the extension name to .crx. On Linux, unzip can read them without changing the filename.

Some technical details

Technically, after reading an article from a ZIM file, it is necessary to "inject" the dependencies (images, css, etc). For compatibility reasons, there are two main ways of doing this:

  • "ServiceWorker" mode (the default) uses a Service Worker to catch any HTTP request the page may send and reply with content read from the ZIM file. It is a generic and clean way of serving content to the browser. It works in any recent browser, but not in older ones. Service Workers are currently disabled by Mozilla in Firefox extensions, and in Chromium extensions active content is severly restricted for security reasons. In both cases we offer a functional workaround (an offline-first PWA version) as a substitute within the extension;
  • "ServiceWorkerLocal" mode is a restricted ServiceWorker mode that is available only in Chromium extensions running fully locally. Chromium extensions running locally block (by design) a lot of dynamic content such as inline JavaScript and eval, which means this mode won't work with some modern dynamic content, and in particular, it won't work with Zimit-based archives (if you open one of these in this mode, you will be thrown back to Safe mode in order to view static content). However, this mode is useful if you cannot access the offline-first PWA, and should work with most official Kiwix ZIM archives;
  • "Safe" mode prevents running attached scripts in the iframe, and so is useful for checking the contents of a ZIM before deciding it is safe to run. This mode also works in browsers that do not support Service Workers. It parses the DOM to find the HTML tags of the dependencies and modifies them to point to content we extract from the ZIM. This mode is compatible with any browser, but becuase it cannot run JavaScript inside the ZIM file, does not work well (if at all) with ZIMs that depend on dynamic content. If you open a dynamic (including Zimit) archive in this mode (or if you are thrown into the mode due to another incompatibility), then we will do our best to display static content, but much functionality is likely to be broken. However, Mediawiki-based content (e.g. Wikipedia) works fine in this mode.

You can switch between these content injection modes in Configuration, but if your browser supports ServiceWorker mode as an offline-first PWA, you are strongly advised to remain in this mode (unless you wish to check the safety of a ZIM before running it -- the UI will ask you if you wish to do this when you first open a new ZIM).

File access and other limitations

You can only re-open an archive automatically if your browser supports the File System Access API and allows you to grant permanent access permission. In practice, this currently means Chromium browsers (Chrome, Edge, etc.) with a version number of 122 or higher. If that is the case, you will see a popup asking you whether you wish to grant access "on every visit" (this will appear only after the second or third time that you have picked an archive or folder). If you grant this permanent permission, then the browser will (optionally) re-open the last-visited archive when you open the app.

In other cases, your browser may fall back to using the webkitdirectory property of the File API, which allows you to re-open a folder or directory of ZIMs with a quick permission prompt. Another alternative is to drag-and-drop a ZIM file into the app.

There are versions of this app that have experimental support for the Origin Private File System, or that use frameworks like Electron, which do have the capability of remembering the chosen archive between app launches.

The app has fast title search, and slower full-text search for ZIM archives that have a full-text index, thanks to the openzim/javascript-libzim project. Currently, full-text searching only works in browsers that support Atomic Operations, which means recent desktop versions of Chromium and Firefox. There is also support in Safari on iOS 15+.

Licence

This application is released under the GPL v3 licence. See http://www.gnu.org/licenses/ or the included LICENSE-GPLv3.txt file The source code can be found at https://github.com/kiwix/kiwix-js.

Contributing

Kiwix JS is an open-source project. We encourage individuals with experience of HTML and JavaScript development to contribute to the documentation and code in this repository.

To report a bug, read our REPORT_BUG guide.

For code contributions, read our CONTRIBUTING guide.

To get to know the Kiwix project better, please familiarize yourself with the content on https://www.kiwix.org. There is also a Kiwix Slack group which you can join.

We also have a CODE_OF_CONDUCT: everybody is expected to follow it.

Public releases and nightly builds

The browser extensions are distributed through the stores of each vendor (see links above). But the packages are also saved in https://download.kiwix.org/release/browsers/ if necessary.

Some nightly builds are generated, and should only be used for testing purpose: https://download.kiwix.org/nightly/.

There is a test implementation of the latest code at https://kiwix.github.io/kiwix-js/ (unbundled: needs a modern browser that suppors native ES6 modules), and a bundled version for any HTML5 browser (>=IE11) at https://kiwix.github.io/kiwix-js/dist/, but these implementations are used for development, and may be buggy, experimental or unstable. A stable PWA version for use in the browser extensions is available from https://browser-extension.kiwix.org/current/.

Previous versions

The first versions of this application were originally part of the Evopedia project: http://www.evopedia.info (discontinued). There was an "articles nearby" feature, that was able to find articles around your location. It has been deleted from the source code with everything related to Evopedia (but still in git history in versions<=2.0.0).

These first versions were targeting Firefox OS (discontinued too: we're not lucky ;-) ).

See CHANGELOG.md for details of previous versions.

kiwix-js's People

Contributors

amitpanwar789 avatar ankur12-1610 avatar aritrocoder avatar ashiscs avatar cibersheep avatar d3v-d avatar dan-niles avatar dependabot[bot] avatar dheerajdlalwani avatar ernesst avatar greeshmanth1909 avatar hrishi-raj avatar ismaelvega avatar jaifroid avatar julianharty avatar kelson42 avatar krinkle avatar maneet-k avatar mossroy avatar peter-x avatar popolechien avatar ram690514 avatar rgaudin avatar rishabhg71 avatar sharun-s avatar soloturn avatar taniamalhotra avatar try-catch-stack avatar vasusharma7 avatar ykabusalah avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

kiwix-js's Issues

Switch back to Zepto.js, instead of jQuery?

jQuery adds the support for Internet Explorer (9+), including (I suppose) the one on Windows Mobile.

Zepto supports all the other major browsers/devices, and is a bit faster and lighter.
Zepto is less tested with other libraries like Bootstrap. It also needs to be compiled with modules selector and data, which are not in the default version of Zepto.

It seems possible to install both, and choose jQuery only when using IE. But it's not in a require.js way (see doc of Zepto and below)

<script>
document.write('<script src=' +
('__proto__' in {} ? 'zepto' : 'jquery') +
'.js><\/script>')
</script>

Make error handling code cleaner and safer

The error handling code is not very clean.
We sometimes test on the error message, which is very bad practice
We sometimes put the error message as the article content
We also call alert() from the algorithm, which should not have a dependency on the GUI.

I'm not proud of all this : it's not clean and not bullet-proof

Implement "articles nearby" feature (geolocation of articles)

I would geolocate the device, and display a list of articles nearby.

The algorithm to search articles of an archive based on their location seems a bit complicated to implement, but it works on the Qt version of Evopedia
On this Qt version, the articles are also displayed on an openstreetmap map. But the issue is how to have the tiles without Internet Access?
For now, I won't try to locate articles on a map, just display a list of articles nearby

Improve the look and behavior of the search results

The results should be better separated, and be highlighted when the cursor is above (on desktop)
On desktop, using the arrow keys should let the user move through the results, as a kind of "auto-suggest" feature

Add back/forward buttons

Ideally, it would be the standard buttons of the browser.
Like the ones that appear at the bottom of web pages in Firefox inside Firefox OS

If not possible, it's easy to add buttons with javascript that does that

Usage of the search field is not completely clear at first use

It seems not visible enough. Maybe simply outlining it?
If we simply put the focus on it, it will be naturally outlined byt bootstrap, and the keyboard would also appear

Or maybe center this search field in the page?
Some users were confused by the search icon on the right of this field, and thought they had to click on it before searching

Some links that use an anchor give encoding errors

Example : in the French dump, display the article "Lampassé". There is a link in the second paragraph with name "Léopard". The URL is "Lion_(h%C3%A9raldique)#L.C3.A9opard_et_L.C3.A9opard_lionn.C3.A9"
When you click on it, it gives an error "malformed URI sequence" in utf8.js

I suppose it comes from the strange encoding after the anchor
It works properly in the original application

Fix links between articles when special characters are used

The current implementation uses the same search algorithm as the prefix search. It simply takes the first result.
While it works in most cases, it can lead to a wrong article when there are similar titles that are the same when normalized (with different accents for example).
For example, make a search with "ascii" in the French dump, and display the first article "Ascii". At the end of paragraph "Table des 128 caractères ASCII", there are links to the article of each letter. If you click on the letter A, it should lead you to the article named "A_(lettre)". But it leads you to the article named "Å (lettre)", with an accent on A.

Use webworker for bzip2 decoding

As this step takes a long time (~5s on a device for an article), it should not block the browser.
The user feedback would be better (spinner icon turning instead of being stuck)

When using hyperlinks in an article, you sometimes end up on another one, with a slightly different name

It's caused by the current implementation of the readArticle function : it re-uses the prefix search, and takes the first result.
When the article can not be found, or if there are several with the same name (after noramlizing), you can end up on the wrong one.

The readArticle function should be re-implemented to lead only to the exact same name. And display an error if it is not found.

On the "small" dump, this behavior can be very easily seen, as most hyperlinks lead to articles that do not exist in the dump.

Optimize bzip2 decoding, if possible

Uncompressing an article is very slow on a device : usually around 5s on a Geeksphone Peak.

But I think this will be hard to improve.
Maybe another javascript implementation : https://github.com/kirilloid/bzip2-js or https://github.com/skeggse/node-bzip/blob/master/node-bzip/index.js
The bzip2 compression format seems to support multi-core decoding. On the Peak device, it would be useful (dual-core CPU), but I think that all these javascript implementations do not support multi-core decoding
A better way to optimize this would be to have a b2g API like https://wiki.mozilla.org/WebAPI/ArchiveAPI. But that does not seem to be planned by Mozilla for now...

Avoid scanning the sdcard at every startup

Currently, the sdcard is scanned at every startup of the application, and the user has to choose which archive he wants to use.
This should be done only once : the selected archive should be stored on the device, so that the sdcard is not scanned the next time.
The user should also have a way to change the archive stored in its user preferences, in the "Configure section"

Add a way to easily download Evopedia archives

It's a bit complicated for the end-user to download an archive separately (with a bittorrent client), then to put it on the sdcard of its phone. It might be a blocker for some.

An idea would be to put a bittorrent client inside Evopedia (as in the original application). The archive could be downloaded directly from it.
Maybe using http://btappjs.com/ or a similar library.

In any case, downloading all these GBytes will take numerous hours, and should be done through an unlimited internet connection (i.e. usually WiFi instead of 3G/4G)

The article "Mi" cannot be displayed on French dump

On the French dump (2013-02-16), when searching for the "Mi" article (with an "i" without accents and in small case), the article can not be displayed : nothing appears on the screen, and no error message

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.