czcorpus / kontext Goto Github PK

An advanced, extensible web front-end for the Manatee-open corpus search engine

License: GNU General Public License v2.0

Python 35.00% CSS 0.20% JavaScript 1.33% HTML 1.25% TypeScript 61.63% Makefile 0.01% Shell 0.24% PEG.js 0.24% Jinja 0.10%

corpora corpus-tools corpus-linguistics user-interface

kontext's Introduction

Introduction
Features
Installation
Customization and contribution
Notable users
How to cite

Introduction

KonText is an advanced corpus query interface and corpus data integration platform built around corpus search engine Manatee-open. It is written in Python 3 and TypeScript and it runs on any major Linux distribution. The development is maintained by the Institute of the Czech National Corpus.

Features

fully editable query chain
- any operation from a user defined sequence (e.g. query -> filter -> sample -> sorting) can be changed and the whole sequence is then re-executed.
multiple search modes:
- concordance,
- paradigmatic query,
- word list
- keyword analysis
simple and advanced query types
- advanced CQL editor with syntax highlighting and attribute recognition
- interactive PoS tag composing tool for positional and key-value tagsets
- customizable query suggestions and simple type query refinement (e.g. for homonym disambiguation)
support for spoken corpora
- defined text segments can be played back as audio
- KWIC detail with easily distinguishable speeches
rich concordance view options and tools
- any positional attribute can be set as primary
- multiple ways how to display other attributes
- user-defined line groups - filtering, reviewing groups ratios
- tokens and KWICs can be connected to external data services (e.g. dictionaries, encyclopedias)
rich subcorpus-related functionality
- any subcorpus is accesible by other users (in case they obtain a URL, otherwise the subcorpus is not discoverable by default)
  - once a public description is set, the subcorpus can be discovered on the "public subcorpora" page
- text types metadata can be gradually refined to a specific subcorpus ("which publishers are there in case only fiction is selected?")
- a custom text types ratio can be defined ("give me 20% fiction and 80% journalism")
- unused subcorpora can be archived (URLs with the subcorpus are still valid) or completely removed (URLs will become invalid)
- searching within a subcorpous can be further refined with ad-hoc text type selection
- a subcorpus can be created with respect to corpora aligned ("give me fiction in Czech but only if there is an English translation for it")
frequency distribution
- univariate
  - positional attributes (including tuples of multiple attributes per token)
  - structural attributes
- multivariate distribution (2 dimensions) for both positional and structural attributes
collocation analysis
persistent URLs - any result page can be easily shared even if the original query is megabytes long
access to previous queries, named queries
convenient corpus access
- finding corpus by a keyword (tag), size, description
- adding corpus to favorites (incl. subcorpora, aligned corpora)
saving result to Excel, CSV, XML, JSONL, TXT
HTTP API access

Internal features

modern client-side application (written in TypeScript, event stream architecture, React components, extensible)
server-side written using the Sanic framework with fully decoupled background concordance/frequency/collocation calculation (using an integrated Rq worker server)
modular code design with dynamically loadable plug-ins providing custom functionality implementation (e.g. custom database adapters, authentication method, corpus listing widgets, HTTP session management)
- integrability with existing information systems

Installation

Docker

Running KonText as a set of Docker containers is the most convenient and flexible way. Docker Compose V2 is required. To run a basic configuration instance (i.e. no MySQL/MariaDB server, no WebSocket server) use:

docker compose up

To run a production grade instance:

docker compose -f docker-compose.yml -f docker-compose.mysql.yml --env-file .env.mysql up

(the .env.mysql allows configuring custom MySQL/MariaDB credentials and KonText configuration file)

Manual installation

Key requirements

Python 3.6 (or newer)
Manatee corpus search engine - version 2.167.8 and onwards (for KonText v0.17, Manatee v2.2xx is recommended)
a key-value storage
- Redis (recommended), SQLite (supported), custom implementations possible
a task queue - Rq
HTTP proxy server
- Nginx (recommended), Apache,...

For Ubuntu OS users, it is recommended to use the install script which should perform most of the actions necessary to install and run KonText. For other Linux distributions we recommend running KonText within a container or a virtual machine. Please refer to the doc/INSTALL.md file for details.

Customization and contribution

Please refer to our Wiki.

Notable users

Institute of the Czech National Corpus
LINDAT/CLARIAH-CZ
CLARIN-PL
CLARIN-SI
Інститут української
Serbski Institut (API version of KonText)

How to cite KonText

Tomáš Machálek (2020) - KonText: Advanced and Flexible Corpus Query Interface

@inproceedings{machalek-2020-kontext,
    title = "{K}on{T}ext: Advanced and Flexible Corpus Query Interface",
    author = "Mach{\'a}lek, Tom{\'a}{\v{s}}",
    booktitle = "Proceedings of the 12th Language Resources and Evaluation Conference",
    month = may,
    year = "2020",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://www.aclweb.org/anthology/2020.lrec-1.865",
    pages = "7003--7008",
    language = "English",
    ISBN = "979-10-95546-34-4",
}

kontext's People

Contributors

Stargazers

Watchers

kontext's Issues

query_history page - long queries should be truncated

... and the full query should be visible on mouseover or something like that

CSV/Excel export of 3 or more aligned languages - incorrect language

how to replicate:

select a parallel corpus with 2 more languages
search something
export result to Excel or CSV
=> first language is OK, third too but second one is replaced by third one

implement KonText set-up validator

What should be checked:

XML validity
references (i.e. a string value referring to an element with specific attribute)
basic plug-ins configuration (especially whether there are required modules present)
paths - whether they exist and whether the web server has proper access rights

rename menu item identifiers according to the 0.7 state

e.g. there is no "subcorpus" menu item any more
in addition, it would be great to make this constant-based to make IDE do the job and prevent possible errors

Phrase query type - add option "case insensitive"

See the title.

make the values passed via _add_globals as lazy as possible :-)

... because not all of them are needed on all the pages

Czech language version mixed up with english tooltip text

You can find more in the screenshot attached.

Remove kontext/corpora/options_dir configuration key

... as it is used no more

sorting of shuffled concordances

Vasek has discovered a strange bug when Kontext behaves a bit differently from NoSkE. Search for "ano" (query type "basic") in ORAL2008, shuffle the concordandce (if not shuffled as default) and then sort according to sp.pohlavi. This will get you to the following screens:

NoSkE: https://korpus.cz/corpora/run.cgi/sortx?q=aword%2C[word%3D%22%28%3Fi%29ano%22]&q=f&corpname=oral2008&attrs=word&attr_allpos=kw&ctxattrs=word&structs=&refs=%3Ddoc.id%2C%3Dsp.pohlavi&iquery=ano&sattr=sp.pohlavi&skey=rc&spos=3

Kontext: https://korpus.cz/kontext/run.cgi/sortx?q=aword%2C[word%3D%22%28%3Fi%29ano%22]&q=f&corpname=oral2008&attrs=word&ctxattrs=word&structs=doc%2Csp&refs=%3Ddoc.id%2C%3Dsp.pohlavi&iquery=ano&sattr=sp.pohlavi&skey=rc&spos=3

Both screens look the same, the concordances are in the same order. But: when you use the "Collocation is sorted: Jump to:" feature and jump to "Z", NoSkE seems to work correctly as it takes you to the beginning of Z's, while kontext will show you a screen where one Z is in the middle of M's!

However, the difference seems to be caused by the way "Jump to" is implemented, because the ordering of concordances in both interfaces is the same (at least I didn't notice any difference) and by browsing through the pages you can see one more such "missorted" Z.

Can you please track down the problem and confirm whether this issue is caused by Manatee?

NB: when applying the same procedure on non-shuffled concordances, the output of sort is OK.

save form values/"within condition" user used to create a sub-corpus

The format must be able to allow reconstructing of a specific sub-corpus.

# kontext: brief descriptions of corpora in Czech in English interface

The corpus selection menu in the English interface is now correctly in English (thanks!), but the mouse hovering action brings out Czech descriptions.

Manatee cache size

Manatee cache takes up too much space now, a query that matches every corpus position (e.g. "[]") in SYN needs more than 50 GB that are stored there uless the cache is explicitly deleted.

It would be desirable to evaluate the usefulness of caching in general, and especially in cases when the concordance size is extremely large. It should be possible to come up with a better solution then, or we could also consider turning the caching off altogether. For the time being, more disk space will be reserved for the cache.

Implement searchable keywords for configured corpora

in config.xml, it should be possible to add 0..n keywords (or "tags") to any corpus
corpora archive should be able to filter corpora list using these keywords

This may replace tree-like structure in the corpus selection widget.

"View" section - some of the menu options should be disabled (concordance, filter,...)

see the title

query_history page - long queries should be truncated

... and the full query should be visible on mouseover or something like that

Background calculation error: Unknown action: ~

2015-02-22 18:37:31,990 [conclib] ERROR: Background calculation error: Unknown action: ~
2015-02-22 18:37:32,093 [controller] ERROR: Traceback (most recent call last):
  File "/opt/noske/share/kontext/public/../lib/controller.py", line 818, in process_method
    self._invoke_action(method, pos_args, named_args, tpl_data))
  File "/opt/noske/share/kontext/public/../lib/controller.py", line 490, in _invoke_action
    ans = apply(action, args[1:], na)
  File "/opt/noske/share/kontext/public/../lib/actions.py", line 184, in view
    conc = self.call_function(conclib.get_conc, (self._corp(),))
  File "/opt/noske/share/kontext/public/../lib/controller.py", line 513, in call_function
    return apply(func, args, na)
  File "/opt/noske/share/kontext/public/../lib/conclib.py", line 415, in get_conc
    samplesize=samplesize, fullsize=fullsize, minsize=minsize)
  File "/opt/noske/share/kontext/public/../lib/conclib.py", line 362, in _get_async_conc
    raise e
Exception: Unknown action: ~

view settings vs. attributes - possible setting overlapping

how to replicate:

set corpus syn2000, search something, select you want to see attribute "opus"
set corpus syn, search something => error is shown

Update copyright information

to match current bonito-open

make help menu items (URL + label) configurable via config.xml

see the title

Remove 'auth' attribute from settings

This means also getting rid of supports_password_change() method.

query_history page - long queries should be truncated

... and the full query should be visible on mouseover or something like that

multi-word kwic - make non-clickable part visually distinguishable from the active one

...e.g. by setting font to italic

empty result - no message

In the development version - in case no result is found no message is provided.
Maybe this is more general problem - it must be further investigated.

action function metadata - support for HTTP method assertion

Something like this:

@exposed(http_method='POST')
def foo(req):
  pass

Such method should be invoked iff the request is POST.

View->Alignment appears to do nothing

how to replicate:

use two or more aligned corpora
search something
click View->Alignment

Allow logging web server related information into the application log

The thing is we should be able to (optionally) obtain all the data we see in a typical Apache log (IP, user agent, url,...) from a single log - KonText's application log.

fix the logic which controls menu items visibility

E.g. - if you enter the "view" section without having a concordance active, concordance-related menu items are available which is wrong.

collocations - in some cases user sees one long page

... instead of paginated result

Conc. persistence - store persistence level flag

Currently, only initial TTL distincts between short and long persistence which may (and it already does) easily confuse archivation script. In other words - there is no clear sign of when a conc. persistence record should be archived.

Fatal error in case user tries to access non-existing sub-corpus

A normal KonText page should appear with some warning.

transform "concache" into a plug-in

... and implement a Redis-based solution. This should bring automatic record locking and simplify related code. Current solution should be still available (though as a plug-in).

ucnk_conc_persistence2 - incorrect TTL calculation

... for anonymous users

Implement searchable keywords for configured corpora

in config.xml, it should be possible to add 0..n keywords (or "tags") to any corpus
corpora archive should be able to filter corpora list using these keywords

This may replace tree-like structure in the corpus selection widget.

collocations and freqs pages - pagination problem

The original Bonito code does not count with two different paginators per page which is the case of the mentioned actions where "concordance" and "current page" paginations are always present.

Implement a page containing list of all corpora

query_history page - long queries should be truncated

... and the full query should be visible on mouseover or something like that

Implement sessions as a WSGI Middleware

see the title

collocations page - minimized concordance box does not contain paginator

plus it looks slightly different

Store sub-corpus parameters

... to be able to run the command again automatically

concordance persistence - implement an archiving script

There should be a 2-level solution:

1st level: fast access with short TTL (shorter for anonymous users)
2st level: slow(er) access with infinite TTL (probably sqlite3-based)

wordlist vs. quick save - missing attribute wlnums

see the title

problem with escaping of special chars in freq. list p/n links

how to replicate:

use e.g. a corpus of English language
search for "are"
select multi-level freq. distribution and "position = 1R"
display the result
find a line containing e.g. the apostrophe character and click either "p" or "n"

Kontext dont display last page in koncordance

In given query when you try to go for the page 12/12 with next page button Kontext reports error:
Failed to process your request. Please try again later or contact system support.

https://kontext.korpus.cz/view?q=~mKpaKzXj;fromp=11;corpname=intercorp_v7_cs&usesubcorp=Core+czech&maincorp=intercorp_v7_cs&viewmode=align&pagesize=100&align=intercorp_v7_be%2Cintercorp_v7_bg%2Cintercorp_v7_ca%2Cintercorp_v7_da&attrs=word&attr_allpos=kw&ctxattrs=word&structs=div%2Cs&refs=doc%2C%3Ddoc.id

implement a script for cleaning-up (some) caches and sessions

user sessions in db (incl. individual user's session)
concordance cache (also individual corpora)
tags cache
freqs cache

JavaScript seems to not work properly
there is a "closed concordance" box visible even if there cannot be any concordance available

Conc. description shows number of results as "null"

see the title

czcorpus / kontext Goto Github PK

kontext's Introduction

Contents

Introduction

Features

Internal features

Installation

Docker

Manual installation

Key requirements

Customization and contribution

Notable users

How to cite KonText

kontext's People

Contributors

Stargazers

Watchers

Forkers

kontext's Issues

Recommend Projects

Recommend Topics

Recommend Org