apertium / apertium-init Goto Github PK

View Code? Open in Web Editor NEW

9.0 13.0 14.0 957 KB

Scripts to bootstrap Apertium language modules and pairs

License: GNU General Public License v3.0

Makefile 38.99% Shell 1.64% Python 53.47% M4 5.90%

apertium-tools

apertium-init's Introduction

Apertium-Init (a.k.a. Apertium Bootstrap)

Bootstrap Apertium language modules and pairs using apertium-init.py.

Installation

There are 3 ways to obtain Apertium-Init:

Download the script from https://apertium.org/apertium-init to your local directory
Clone this repository and run make install
Install from PyPi with pip install apertium-init

If the script was downloaded directly, it can be run with python3 apertium-init.py. Otherwise it will be installed as apertium-init.

Usage

Usage depends on the desired module and is described below. Remember to search for TODO in the generated module to add example sentences, etc.

Monolingual Lttoolbox module

To bootstrap a monolingual language module apertium-foo using the lttoolbox formalism,

$ python3 apertium-init.py foo

To bootstrap and compile it at the same time,

$ python3 apertium-init.py foo && ( cd apertium-foo/ && make )

Monlingual HFST module

To bootstrap a monolingual language module apertium-foo using the HFST formalism,

$ python3 apertium-init.py foo --analyser=hfst

To bootstrap and compile it at the same time,

$ python3 apertium-init.py foo --analyser=hfst && ( cd apertium-foo/ && make )

To include a twoc file for handling prefixes,

$ python3 apertium-init.py foo --analyser=hfst --with-twoc

To include a spellrelax file for handling typographical variance,

$ python3 apertium-init.py foo --analyser=hfst --with-spellrelax

Bilingual Lttoolbox module

To bootstrap a bilingual language module apertium-foo-bar where the monolingual packages apertium-foo and apertium-bar both use the lttoolbox formalism,

$ python3 apertium-init.py foo-bar

To bootstrap and compile it at the same time,

$ python3 apertium-init.py foo-bar && ( cd apertium-foo-bar/ && ./autogen.sh && make test)

Bilingual HFST/Lttoolbox module

To bootstrap a bilingual language module apertium-foo-bar where the monolingual packages apertium-foo and apertium-bar use the HFST formalism and/or the lttoolbox formalism,

$ python3 apertium-init.py foo-bar --analysers=hfst # Both foo and bar use HFST
$ python3 apertium-init.py foo-bar --analyser1=hfst # Only foo (first language) uses HFST
$ python3 apertium-init.py foo-bar --analyser2=hfst # Only bar (second language) uses HFST

To bootstrap and compile it at the same time,

$ python3 apertium-init.py foo-bar --analysers=hfst && ( cd apertium-foo-bar/ && ./autogen.sh && make test) # Both foo and bar use HFST
$ python3 apertium-init.py foo-bar --analyser1=hfst && ( cd apertium-foo-bar/ && ./autogen.sh && make test) # Only foo (first language) uses HFST
$ python3 apertium-init.py foo-bar --analyser2=hfst && ( cd apertium-foo-bar/ && ./autogen.sh && make test) # Only bar (second language) uses HFST

Bilingual module for monolinguals that don't use apertium-tagger or CG

To bootstrap a bilingual module when one or both of the monolingual modules don't use apertium-tagger,

$ python3 apertium-init.py foo-bar --no-prob1            # Only foo doesn't have .prob
$ python3 apertium-init.py foo-bar --no-prob2            # Only bar doesn't have .prop
$ python3 apertium-init.py foo-bar --no-prob1 --no-prob2 # Neither foo nor bar have .prob

To bootstrap a bilingual module when one or both of the monolingual modules don't use CG,

$ python3 apertium-init.py foo-bar --no-rlx1           # Only foo doesn't have .rlx
$ python3 apertium-init.py foo-bar --no-rlx2           # Only bar doesn't have .rlx
$ python3 apertium-init.py foo-bar --no-rlx1 --no-rlx2 # Neither foo nor bar have .rlx

Bilingual module using recursive transfer

To bootstrap a bilingual module which uses apertium-recursive,

$ python3 apertium-init.py foo-bar --transfer=rtx

Bilingual module with anaphora resolution

To bootstrap a bilingual module which uses apertium-anaphora,

$ python3 apertium-init.py foo-bar --with-anaphora

Bilingual module with discontiguous multiwords

To bootstrap a bilingual module which uses apertium-separable,

$ python3 apertium-init.py foo-bar --with-separable

Adding features to an existing module

Apertium-init can reconfigure an existing module or pair. For example, to add apertium-separable to an existing pair:

$ python3 apertium-init.py foo-bar -r --with-separable

Note that all desired options must be specified. If the foo-bar pair used apertium-anaphora, the above command would remove it.

Pushing to Github

To bootstrap a module or pair and also add it to the apertium incubator,

$ python3 apertium-init.py foo -p     # Bootstrap module apertium-foo and push to Github
$ python3 apertium-init.py foo-bar -p # Bootstrap pair apertium-foo-bar and push to Github

To specify what username to push as (rather than relying on git config),

$ python3 apertium-init.py foo -p -u bar # Bootstrap module apertium-foo and push to Github under username bar

Development

After updating vanilla files, run make in the root of the repository to generate apertium-init.py.

You can also do sudo make install to install to /usr/local/bin/apertium-init or e.g. PREFIX=$HOME/local make install to install to $HOME/local/bin/apertium-init.

Use pipenv install --dev to install the requirements required for development, e.g. linters.

Releasing

After installing development resources following the instructions above, deploying to PyPi is relatively straightforward.

Use make dist to create a source distributable inside the dist directory that can be installed locally via pip.

Use make test-release and make release to deploy to the testing PyPi instance and the production PyPi instance respectively. Either step requires PyPi authentication credentials with access to the apertium-init package.

apertium-init's People

Contributors

Stargazers

Watchers

Forkers

unhammer sushain97 nuboro jonorthwash ethanachi hsumerf dolphingarlic taruen nstsj mr-martian khannatanmai nlhowell flammie joanluc

apertium-init's Issues

Support bootstrapping pairs where one side is from GT infra

Giellatekno pairs require slightly different makefile stuffs.

These pairs are both directions GT:

[email protected]:/apertium/apertium-sme-sma.git
[email protected]:/apertium/apertium-sme-smj.git
[email protected]:/apertium/apertium-sme-smn.git

These are one direction GT:

[email protected]:/apertium/apertium-sme-nob.git

make sure monolingual modules show only the right files in the README

Currently HFST modules show the "dix" file in the list of files in the README, and it's likely that lttoolbox modules show "hfst" files in their README. This needs to be checked and fixed.

should run ./autogen.sh in monolinguals

That way, you can do

apertium-init foo
apertium-init bar
apertium-init foo-bar
cd apertium-foo-bar
./autogen.sh --with-lang1=../apertium-foo --with-lang2=../apertium-bar
make langs

should warn if it couldn't find the English language name for an iso code

$ apertium-init foo                                                                                                                                                                 
Succesfully created /apertium-foo.
Unable to add /tmp/apertium-foo to SVN: svn: E155007: '/tmp' is not a working copy
$ head apertium-foo/README                                                                                                                                   
foo

                            apertium-foo
===============================================================================

This is an Apertium monolingual language package for foo. What
you can use this language package for:

* Morphological analysis of foo
* Morphological generation of foo
$

monolingual make rules and sample corpus for apertium-tagger .prob training

We should have some make rules for the unigram tagger, along with a tiny sample corpus of the right format (e.g. in the subdir corpus). Something like

LANG.prob: corpus/LANG.corpus
	apertium -d . tagger-training-pipeline <$< >$@

(Someone with an idea what the tagger training pipeline looks like will have to fill that out.)

default Makefiles and configs should work with make distcheck

I've had to comment out some stuff from the configure.ac and Makefile.am to get distcheck target working; distcheck is very useful in GNU autotools since it checks some expectations that are baked in to many package mangling systems (gentoo and some BSD's for example).

For example:

apertium/apertium-fin-deu@a8d3ad2
apertium/apertium-fin@0586173

add option to include spellrelax

Add an option to apertium-init to add for hfst modules a spellrelax file and compile it, along with relevant details in the README.

add option to include prefixes in monolingual HFST modules

Add an option to enable twoc-style prefixational morphology in HFST modules, along with a relevant example file and relevant details in the README.

Auto-set repository description/website on push

Website should be wiki page

trailing whitespace in makefile

after apertium-init foo-bar, the Makefile.am contains lines

    $(PREFIX1).automorf.bin \ 
    $(PREFIX1).autogen.bin \ 
    $(PREFIX2).automorf.bin \ 
    $(PREFIX2).autogen.bin \

with space after the \

Support different naming conventions.

E.g., instead of apertium-xyz, maybe ling073-xyz (like for the course I teach). Or even things like xyzmorph?

.prob is not installed (monolingual)

fixed by having

apertium_foo_DATA=$(TARGETS_COMMON) $(LANG1).prob

instead of

apertium_foo_DATA=$(TARGETS_COMMON)

Could this be changed to GPLv3+

Title says all.

define some LEXCFLAGS?

like in
apertium/apertium-nio@e1f4494

maybe we should also have some $(UPPERCASE) programs

autogen listed twice in monodix makefile.am

TARGETS_COMMON = $(LANG1).automorf.bin \
        $(LANG1).autogen.bin $(LANG1).autopgen.bin $(LANG1).autogen.bin \
        $(LANG1).automorf.att.gz $(LANG1).autogen.att.gz \
        $(LANG1).rlx.bin

$(LANG1).autogen.bin is listed twice. 9a77dcc

Add apertium-incubator topic on init

apertium-init.py instructions more clear

Is it possible to make the instructions of apertium-init.py more clear? Because it's not immediately obvious (to beginners like me) that you need to create your own repo when bootstrapping a new language pair and push to it.
Also could it be added to the wiki that when compiling the language pair you need to enter the apertium-XXX-YYY directory first? Otherwise you get the error that auto-gen.sh doesn't exist...

¿<?xml for xml files?

Could all xml files start with:

<?xml version='1.0' encoding='utf-8'?>

It seems to make vim happy to select xml mode in my system, currently it's a bit random which files have this.

Update README

The readme should be updated to remove references to SVN. There should be a symlink to README.md. The README should be made markdowny.

Add CI configs

"apertium-init -a hfst nio" doesn't give language code is configure.ac

 -AC_INIT([Apertium {{languageName1}}-{{languageName2}}], [0.1.0], [[email protected]], [apertium-{{languageCode1}}-{{languageCode2}}], [http://wiki.apertium.org/wiki/Apertium-{{languageCode1}}-{{languageCode2}}])
+AC_INIT([apertium-nio], [0.0.0], [[email protected]], [apertium-nio], [http://wiki.apertium.org/wiki/Apertium-nio])

--Werror to hfst-lexc

The instructions are wrong

I tried apertium-init out:

$ apertium-init/apertium-init.py tlh-swe --analyser1=hfst
Successfully created /home/tpirinen/github/apertium/apertium-tlh-swe.
Initialized git repository apertium-tlh-swe.
Successfully added and committed files to git repository apertium-tlh-swe.
To push your new local repository to incubator in the Apertium organisation on GitHub:
	apertium-init.py -pe /home/tpirinen/github/apertium/apertium-tlh-swe
$ apertium-init/apertium-init.py -pe apertium-tlh-swe/
usage: apertium-init.py [-h] [-d DESTINATION] [-p]
                        [-pe PUSH_EXISTING_TO_GITHUB] [-u USERNAME]
                        [-a {lt,lttoolbox,hfst}] [-a1 {lt,lttoolbox,hfst}]
                        [-a2 {lt,lttoolbox,hfst}] [--no-rlx1] [--no-rlx2]
                        [--no-prob1] [--no-prob2] [--no-pgen1] [--no-pgen2]
                        name
apertium-init.py: error: the following arguments are required: name

I also cannot execute git push -u origin master or so in the dir:

$ git push -u origin master
fatal: 'origin' does not appear to be a git repository
fatal: Konnte nicht vom Remote-Repository lesen.

Bitte stellen Sie sicher, dass die korrekten Zugriffsberechtigungen bestehen
und das Repository existiert.

add option to include apertium-separable

Add an option to apertium-init to include appropriate apertium-separable stages in the pipeline for translation modes, along with a relevant example file and relevant details in the README.

add goal for untrimmed debug modes?

See https://sourceforge.net/p/apertium/tickets/88/#b3d3

For all pairs, we should add a debug target that includes the untrimmed analyser (easy), and debug modes that are like the regular ones, but use the untrimmed analyser (boring).

pip issues

apertium-init does not seem to install correctly using pip.

jwashin1@pearson105:~/Documents$ sudo pip3 install apertium-init
Collecting apertium-init
  Downloading apertium_init-2.1.0-py3-none-any.whl (75kB)
    100% |████████████████████████████████| 81kB 4.2MB/s 
Installing collected packages: apertium-init
Successfully installed apertium-init-2.1.0
jwashin1@pearson105:~/Documents$ apertium-init -h
Traceback (most recent call last):
  File "/usr/local/bin/apertium-init", line 7, in <module>
    from apertium_init import main
  File "/usr/local/lib/python3.6/dist-packages/apertium_init.py", line 13, in <module>
    module = __import__('apertium-init')
ModuleNotFoundError: No module named 'apertium-init'

add option to include prefix features

Add an option to apertium-init to add for hfst modules the ability to write twol-type prefix constraints. This will require adjusting Makefile.am, adding a twoc file, and providing relevant details in the README.

DTRT when user types "apertium-lol" instead of just "lol"

Should be safe to create apertium-lol instead of apertium-apertium-lol if the user gave "apertium-lol"

Set repository metadata on create (apertium-incubator)

topic
website
description

change lrx-proc to be lrx-proc -m by default

use AS_IF in configure.ac checks for programs

The current rules for checking whether some program exists don't work very well, e.g. they set the CGCOMP variable to "no", which might make you try to compile with the program "no" (or check for the existence of the program "no", leading to No rule to make target 'no').

According to @flammie, this is a better check:

AC_PATH_PROG([CGCOMP], [cg-comp], [false], [$PATH$PATH_SEPARATOR$with_cg_comp/bin])
AS_IF([test x$CGCOMP = xfalse], [AC_MSG_ERROR([You don't have cg-comp installed])])

It's also much shorter than

AC_PATH_PROG(CGCOMP, cg-comp, no)
if test x$ac_cv_path_CGCOMP = x
then
  AC_MSG_ERROR([You don't have cg-comp installed.])
fi
if test x$ac_cv_path_CGCOMP = xno
then
  AC_MSG_ERROR([You don't have cg-comp installed.])
fi

:-)

Increase test coverage to 95%

Compress encoded files

Could easily be a GCI task

Switch to pipenv

See apertium/streamparser#33 for an approach.

Correctly set repository description

missing \ after .prob in TARGETS_COMMON when --no-rlx1

did

$ apertium-init -a2 hfst --no-rlx1 --no-pgen2 lol-wat

got

TARGETS_COMMON = \
    $(PREFIX1).automorf.bin \ 
    $(PREFIX1).prob 
    $(PREFIX1).rlx.bin \
    $(PREFIX1).autobil.bin \

and

config.status: creating Makefile
Makefile:656: *** missing separator.  Stop.

Package for pypi and expose apertium-init script.

apertium-init hangs waiting for input

fran@matxine:/tmp$ python3 ~/scripts/apertium-init.py --help
^CTraceback (most recent call last):
  File "/home/fran/scripts/apertium-init.py", line 18, in <module>
    import subprocess
  File "/usr/lib/python3.6/subprocess.py", line 50, in <module>
    import signal
  File "/home/fran/scripts/signal.py", line 4, in <module>
    doc = sys.stdin.read()
KeyboardInterrupt

What is the XML that it expects to read from stdin ?

$ echo "" | python3 ~/scripts/apertium-init.py --help
Traceback (most recent call last):
  File "/home/fran/scripts/apertium-init.py", line 18, in <module>
    import subprocess
  File "/usr/lib/python3.6/subprocess.py", line 50, in <module>
    import signal
  File "/home/fran/scripts/signal.py", line 6, in <module>
    root = ET.fromstring(doc)
  File "/usr/lib/python3.6/xml/etree/ElementTree.py", line 1315, in XML
    return parser.close()
xml.etree.ElementTree.ParseError: no element found: line 2, column 0

Ignore .prob on bilingual modules

Prob files have to exist, but they are on the monolingual modules. They shouldn't be versioned on the bilingual ones.

Example:

apertium-cat -> apertium-cat.cat.prob added to GIT
apertium-spa -> apertium-cat.spa.prob added to GIT
apertium-spa-cat -> cat-spa.prob and spa-cat.prob ignored in GIT

Add tests

add to Travis

Auto warn when TODOs not replaced on push

cc @ftyers @jonorthwash

feature: support for giella-pairs

readme says rlx instead of lrx

e.g.

apertium-nor-eng.nor-eng.rlx - Lexical selection rules for translating into Norwegian

should be .lrx

Add description on how to run it to the wiki page

This page: http://wiki.apertium.org/wiki/Apertium-init

Should specify how to run the thing.

fix issues with bidix

There are some issues with the included bidix: the alphabet should be empty (or non-existant?) and the "sg" tags shouldn't be in the example entries. It also might be good to have entries in two different languages, especially ones with incompatible POS sub-categories (e.g. casa<n><f>).

apertium-tagger needs -x sometimes

apertium-init needs to check whether the .prob files are made with the perceptron algorithm, and add -x to apertium-tagger if so. Otherwise, bootstrapped pairs will gobble up all memory and segfault as tagger tries to load the .prob as HMM.

E.g. apertium-eng's eng.prob is perceptron.

Use base 64 instead of base 81 for < Python 3.4

more docs for other options

in the README, we have no docs/examples for:

--no-prob
--no-rlx
--with-twoc
github stuff

support for more features in hfst modules

Add optional support to hfst modules for enabling spelling modules, an extra twoc module for morphotactic constraints, and spellrelax. Perhaps these should all be options at configure/autogen time, so that they can be enabled as needed after a module is started, but skeletons of the files they require should probably also be provided.

apertium-init doesn't make corpus dir

fran@matxine:/tmp/apertium-pus$ ./autogen.sh 
make
configure.ac:4: installing './install-sh'
configure.ac:4: installing './missing'
Makefile.am: installing './INSTALL'
checking for a BSD-compatible install... /usr/bin/install -c
checking whether build environment is sane... yes
checking for a thread-safe mkdir -p... /bin/mkdir -p
checking for gawk... gawk
checking whether make sets $(MAKE)... yes
checking whether make supports nested variables... yes
checking whether ln -s works... yes
checking for gawk... (cached) gawk
checking for pkg-config... /usr/bin/pkg-config
checking pkg-config is at least version 0.9.0... yes
checking for APERTIUM... yes
checking for LTTOOLBOX... yes
checking for cg-comp... /usr/bin/cg-comp
checking that generated files are newer than configure... done
configure: creating ./config.status
config.status: creating Makefile
config.status: creating apertium-pus.pc
fran@matxine:/tmp/apertium-pus$ make
apertium-validate-dictionary apertium-pus.pus.dix
lt-comp lr apertium-pus.pus.dix pus.automorf.bin apertium-pus.pus.acx
final@inconditional 4 11
main@standard 16 22
lt-print pus.automorf.bin | gzip -9 -c -n > pus.automorf.att.gz
test -d .deps || mkdir .deps
touch .deps/.d
apertium-validate-dictionary apertium-pus.pus.dix
lt-comp rl apertium-pus.pus.dix pus.autogen.bin apertium-pus.pus.acx
final@inconditional 4 11
main@standard 16 22
lt-print pus.autogen.bin | gzip -9 -c -n > pus.autogen.att.gz
lt-comp lr apertium-pus.post-pus.dix pus.autopgen.bin
main@standard 5 8
make: *** No rule to make target 'corpus/pus.tagged', needed by 'pus.prob'.  Stop.

apertium / apertium-init Goto Github PK

apertium-init's Introduction

Apertium-Init (a.k.a. Apertium Bootstrap)

Installation

Usage

Monolingual Lttoolbox module

Monlingual HFST module

Bilingual Lttoolbox module

Bilingual HFST/Lttoolbox module

Bilingual module for monolinguals that don't use apertium-tagger or CG

Bilingual module using recursive transfer

Bilingual module with anaphora resolution

Bilingual module with discontiguous multiwords

Adding features to an existing module

Pushing to Github

Development

Releasing

apertium-init's People

Contributors

Stargazers

Watchers

Forkers

apertium-init's Issues

Recommend Projects

Recommend Topics

Recommend Org