giellalt / lang-fkv Goto Github PK

Finite state and Constraint Grammar based analysers and proofing tools, and language resources for the Kven Finnish language

Home Page: https://giellalt.uit.no

License: GNU Lesser General Public License v3.0

Makefile 1.40% Shell 2.35% M4 1.81% Python 0.01% Perl 0.04% Regular Expression 0.63% XML 0.14% YAML 55.22% Text 38.41%

finite-state-transducers constraint-grammar minority-language nlp language-resources proofing-tools giellalt-langs geo-nordic langfam-uralic kven-finnish-language

lang-fkv's Introduction

The Kven Finnish morphology and tools

Download nightly / CI/CD installation packages for testing (contains the core zhfst file(s)):

NB!! Note that the nightly / CI/CD installation packages are not tested for language quality, and might contain regressions and errors.

This repository contains finite state source files for the Kven Finnish language, for building morphological analysers, proofing tools and dictionaries. The data and implementation are licenced under LICENSE licence, also detailed in the LICENSE. The authors named in the AUTHORS file are available to grant other licencing choices.

Install proofing tools and keyboards for the Kven Finnish language by using the Divvun Installer (some languages are only available via the nightly channel).

Download and test speller files

The speller files downloadable at the top of this page (the *.bhfst files) can be used with divvunspell, to test their performance. These files are the exact same ones as installed on users' computers and mobile phones. Desktop and mobile speller files differ from each other in the error model and should be tested separately — thus also two different downloads.

Documentation

Documentation can be found at:

Core dependencies

In order to compile and use Kven Finnish language morphology and dictionaries, you need:

an FST compiler: HFST, Foma or Xerox Xfst
VislCG3 Constraint Grammar tools

To install VislCG3 and HFST, just copy/paste this into your Terminal on macOS:

curl https://apertium.projectjj.com/osx/install-nightly.sh | sudo bash

or terminal on Ubuntu, Debian or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/apt/install-nightly.sh -O - | sudo bash
sudo apt-get install cg3 hfst

or terminal on RedHat, Fedora, CentOS or Windows Subsystem for Linux:

wget https://apertium.projectjj.com/rpm/install-nightly.sh -O - | sudo bash
sudo dnf install cg3 hfst

Alternatively, the Apertium wiki has good instructions on how to install the dependencies for Mac OS X and how to install the dependencies on linux

Further details and dependencies are described on the GiellaLT Getting Started pages.

Downloading

Using Git:

git clone https://github.com/giellalt/lang-fkv

Using Subversion:

svn checkout https://github.com/giellalt/lang-fkv.git/trunk lang-fkv

Building and installation

INSTALL describes the GNU build system in detail, but for most users it is the usual:

./autogen.sh # This will automatically clone or check out other GiellaLT dependencies
./configure
make
(as root) make install

Citing

If you use language data from more than one GiellaLT language, consider citing our LREC 2022 article on whole infra:

Linda Wiechetek, Katri Hiovain-Asikainen, Inga Lill Sigga Mikkelsen, Sjur Moshagen, Flammie Pirinen, Trond Trosterud, and Børre Gaup. 2022. Unmasking the Myth of Effortless Big Data - Making an Open Source Multi-lingual Infrastructure and Building Language Resources from Scratch. In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 1167–1177, Marseille, France. European Language Resources Association.

If you use bibtex, following is as it is on ACL anthology:

@inproceedings{wiechetek-etal-2022-unmasking,
    title = "Unmasking the Myth of Effortless Big Data - Making an Open Source
    Multi-lingual Infrastructure and Building Language Resources from Scratch",
    author = "Wiechetek, Linda  and
      Hiovain-Asikainen, Katri  and
      Mikkelsen, Inga Lill Sigga  and
      Moshagen, Sjur  and
      Pirinen, Flammie  and
      Trosterud, Trond  and
      Gaup, B{\o}rre",
    booktitle = "Proceedings of the Thirteenth Language Resources and Evaluation
    Conference",
    month = jun,
    year = "2022",
    address = "Marseille, France",
    publisher = "European Language Resources Association",
    url = "https://aclanthology.org/2022.lrec-1.125",
    pages = "1167--1177"
}

lang-fkv's People

Contributors

Stargazers

Watchers

lang-fkv's Issues

After infra reorg this spring, editdistance info comes up as 0

After make clean in lang-fkv/tools/spellcheckers/, I get the following for make V=1

uit-mac-443 lang-fkv (main)$ make V=1

*** Compiling fkv - Kven Finnish. ***

CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh '/Users/ttr000/git/giellalt/lang-fkv/build-aux/missing' autoconf
Making all in .

*** Compiling fkv - Kven Finnish. ***

CDPATH="${ZSH_VERSION+.}:" && cd . && /bin/sh '/Users/ttr000/git/giellalt/lang-fkv/build-aux/missing' autoconf
make[1]: Nothing to be done for `all-am'.
Making all in src
Making all in fst
Making all in morphology
make[3]: Nothing to be done for `all'.
Making all in filters
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt area-tags.txt.tmp Area/
if test -f area-tags.txt ; then \
		if ! diff -q area-tags.txt area-tags.txt.tmp ; then \
			mv -f area-tags.txt.tmp area-tags.txt ; \
		else \
			rm -f area-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f area-tags.txt.tmp area-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt derivation-tags.txt.tmp Der/
if test -f derivation-tags.txt ; then \
		if ! diff -q derivation-tags.txt derivation-tags.txt.tmp ; then \
			mv -f derivation-tags.txt.tmp derivation-tags.txt ; \
		else \
			rm -f derivation-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f derivation-tags.txt.tmp derivation-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt dialect-tags.txt.tmp Dial/
if test -f dialect-tags.txt ; then \
		if ! diff -q dialect-tags.txt dialect-tags.txt.tmp ; then \
			mv -f dialect-tags.txt.tmp dialect-tags.txt ; \
		else \
			rm -f dialect-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f dialect-tags.txt.tmp dialect-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt error-tags.txt.tmp Err/
if test -f error-tags.txt ; then \
		if ! diff -q error-tags.txt error-tags.txt.tmp ; then \
			mv -f error-tags.txt.tmp error-tags.txt ; \
		else \
			rm -f error-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f error-tags.txt.tmp error-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt grammatical-tags.txt.tmp Gram/
if test -f grammatical-tags.txt ; then \
		if ! diff -q grammatical-tags.txt grammatical-tags.txt.tmp ; then \
			mv -f grammatical-tags.txt.tmp grammatical-tags.txt ; \
		else \
			rm -f grammatical-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f grammatical-tags.txt.tmp grammatical-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt orig_lang-tags.txt.tmp OLang/
if test -f orig_lang-tags.txt ; then \
		if ! diff -q orig_lang-tags.txt orig_lang-tags.txt.tmp ; then \
			mv -f orig_lang-tags.txt.tmp orig_lang-tags.txt ; \
		else \
			rm -f orig_lang-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f orig_lang-tags.txt.tmp orig_lang-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt semantic-tags.txt.tmp Sem/
if test -f semantic-tags.txt ; then \
		if ! diff -q semantic-tags.txt semantic-tags.txt.tmp ; then \
			mv -f semantic-tags.txt.tmp semantic-tags.txt ; \
		else \
			rm -f semantic-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f semantic-tags.txt.tmp semantic-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt usage-tags.txt.tmp Use/
if test -f usage-tags.txt ; then \
		if ! diff -q usage-tags.txt usage-tags.txt.tmp ; then \
			mv -f usage-tags.txt.tmp usage-tags.txt ; \
		else \
			rm -f usage-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f usage-tags.txt.tmp usage-tags.txt ; \
	fi
/Library/Developer/CommandLineTools/usr/bin/make  all-am
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt orig_lang-tags.txt.tmp OLang/
if test -f orig_lang-tags.txt ; then \
		if ! diff -q orig_lang-tags.txt orig_lang-tags.txt.tmp ; then \
			mv -f orig_lang-tags.txt.tmp orig_lang-tags.txt ; \
		else \
			rm -f orig_lang-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f orig_lang-tags.txt.tmp orig_lang-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt area-tags.txt.tmp Area/
if test -f area-tags.txt ; then \
		if ! diff -q area-tags.txt area-tags.txt.tmp ; then \
			mv -f area-tags.txt.tmp area-tags.txt ; \
		else \
			rm -f area-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f area-tags.txt.tmp area-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt derivation-tags.txt.tmp Der/
if test -f derivation-tags.txt ; then \
		if ! diff -q derivation-tags.txt derivation-tags.txt.tmp ; then \
			mv -f derivation-tags.txt.tmp derivation-tags.txt ; \
		else \
			rm -f derivation-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f derivation-tags.txt.tmp derivation-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt dialect-tags.txt.tmp Dial/
if test -f dialect-tags.txt ; then \
		if ! diff -q dialect-tags.txt dialect-tags.txt.tmp ; then \
			mv -f dialect-tags.txt.tmp dialect-tags.txt ; \
		else \
			rm -f dialect-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f dialect-tags.txt.tmp dialect-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt error-tags.txt.tmp Err/
if test -f error-tags.txt ; then \
		if ! diff -q error-tags.txt error-tags.txt.tmp ; then \
			mv -f error-tags.txt.tmp error-tags.txt ; \
		else \
			rm -f error-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f error-tags.txt.tmp error-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt grammatical-tags.txt.tmp Gram/
if test -f grammatical-tags.txt ; then \
		if ! diff -q grammatical-tags.txt grammatical-tags.txt.tmp ; then \
			mv -f grammatical-tags.txt.tmp grammatical-tags.txt ; \
		else \
			rm -f grammatical-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f grammatical-tags.txt.tmp grammatical-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt semantic-tags.txt.tmp Sem/
if test -f semantic-tags.txt ; then \
		if ! diff -q semantic-tags.txt semantic-tags.txt.tmp ; then \
			mv -f semantic-tags.txt.tmp semantic-tags.txt ; \
		else \
			rm -f semantic-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f semantic-tags.txt.tmp semantic-tags.txt ; \
	fi
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/extract-tags.sh lexicon-tags.txt usage-tags.txt.tmp Use/
if test -f usage-tags.txt ; then \
		if ! diff -q usage-tags.txt usage-tags.txt.tmp ; then \
			mv -f usage-tags.txt.tmp usage-tags.txt ; \
		else \
			rm -f usage-tags.txt.tmp ; \
		fi ;\
	else \
		mv -f usage-tags.txt.tmp usage-tags.txt ; \
	fi
Making all in phonetics
Making all in .
make[4]: Nothing to be done for `all-am'.
Making all in tests
make[4]: Nothing to be done for `all'.
Making all in syllabification
make[3]: Nothing to be done for `all'.
Making all in orthography
make[3]: Nothing to be done for `all'.
Making all in transcriptions
make[3]: Nothing to be done for `all'.
Making all in tagsets
make[3]: Nothing to be done for `all'.
Making all in .
make[3]: Nothing to be done for `all-am'.
Making all in cg3
make[2]: Nothing to be done for `all'.
make[2]: Nothing to be done for `all-am'.
Making all in tools
Making all in tokenisers
Making all in filters
make[3]: Nothing to be done for `all'.
Making all in .
make[3]: Nothing to be done for `all-am'.
Making all in tests
make[3]: Nothing to be done for `all'.
Making all in analysers
Making all in .
make[3]: Nothing to be done for `all-am'.
Making all in shellscripts
make[2]: Nothing to be done for `all'.
Making all in spellcheckers
Making all in filters
cp -f /Users/ttr000/git/giellalt/lang-fkv/./../giella-core/fst-filters/remove-word-boundary.regex remove-word-boundary.regex
"/usr/local/bin/hfst-regexp2fst"  \
	   --format=openfst-tropical \
		"--xerox-composition=ON" -v -S -i remove-word-boundary.regex -o remove-word-boundary.hfst
Reading from remove-word-boundary.regex, writing to remove-word-boundary.hfst
Compiling expression #1
Compiling expression #2
cp -f /Users/ttr000/git/giellalt/lang-fkv/src/fst/filters/remove-usage-tags.regex remove-usage-tags.regex
"/usr/local/bin/hfst-regexp2fst"  \
	   --format=openfst-tropical \
		"--xerox-composition=ON" -v -S -i remove-usage-tags.regex -o remove-usage-tags.hfst
Reading from remove-usage-tags.regex, writing to remove-usage-tags.hfst
Compiling expression #1
Compiling expression #2
cp -f /Users/ttr000/git/giellalt/lang-fkv/src/fst/filters/remove-error-strings.regex remove-error-strings.regex
"/usr/local/bin/hfst-regexp2fst"  \
	   --format=openfst-tropical \
		"--xerox-composition=ON" -v -S -i remove-error-strings.regex -o remove-error-strings.hfst
Reading from remove-error-strings.regex, writing to remove-error-strings.hfst
Compiling expression #1
Compiling expression #2
Making all in weights
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/clean_weight_corpus.sh spellercorpus.raw.txt spellercorpus.clean.txt
Making all in neural
make[3]: Nothing to be done for `all'.
Making all in .
/usr/local/bin/gmkdir -p `dirname .generated/.stamp`
touch .generated/.stamp
/usr/bin/printf "read regex \
		    @\"../../src/fst/filters/remove-usage_except_speller-tags.hfst\" \
		.o. @\"../../src/fst/filters/remove-orig_lang-tags.hfst\"        \
		.o. @\"../../src/fst/filters/remove-CLB-strings.hfst\"           \
		.o. @\"../../src/fst/filters/remove-error-strings.hfst\"         \
		.o. @\"../../src/fst/filters/remove-MinusSpell-strings.hfst\"    \
		.o. @\"../../src/fst/filters/remove-PUNCT-strings.hfst\"         \
		.o. @\"../../src/fst/filters/remove-use_marg-strings.hfst\"      \
		.o. @\"../../src/fst/filters/remove-Use_minus_PMatch-tags.hfst\" \
		.o. @\"../../src/fst/filters/remove-Use_minusGC-tags.hfst\"      \
		.o. @\"../../src/fst/filters/remove-Use_GC-strings.hfst\"        \
		.o. @\"../../src/fst/filters/remove-Use_PMatch-strings.hfst\"    \
		.o. @\"../../src/fst/filters/remove-mwe-tags.hfst\"              \
		.o. @\"../../src/fst/filters/remove-use_mt-strings.hfst\"        \
		.o. @\"../../src/fst/analyser-raw-gt-desc.hfst\" \
		.o. @\"../../src/fst/orthography/downcase-derived_proper-strings.compose.hfst\" \
		;\n\
		 save stack .generated/generator-speller-gt-norm.tmp.hfst\n\
		 quit\n" | /usr/local/bin/hfst-xfst -p -v --format=foma
Using foma as output handler
Reading from standard input...
warning: both composition arguments contain flag diacritics that are not harmonized
? bytes. 136976 states, 335416 arcs, ? paths
hfst[1]: hfst[1]: hfst[1]: .
cp -f .generated/generator-speller-gt-norm.tmp.hfst .generated/generator-speller-gt-norm.hfst
cp .generated/generator-speller-gt-norm.hfst generator-speller-gt-norm.hfst
/usr/bin/printf "read regex                                        \
			                                      \
			@\"generator-speller-gt-norm.hfst\"                                                           \
		.o. @\"../../src/fst/filters/remove-hyphenation-marks.hfst\"    \
		.o. @\"../../src/fst/filters/remove-infl_deriv-borders.hfst\"   \
		; \n\
		 twosided flag-diacritics\n\
		 save stack .generated/generator-fstspeller-gt-norm.tmp.hfst\n\
		 quit\n" | /usr/local/bin/hfst-xfst -p -v --format=foma
Using foma as output handler
Reading from standard input...
? bytes. 135461 states, 335095 arcs, ? paths
hfst[1]: hfst[1]: hfst[1]: hfst[1]: .
cp -f .generated/generator-fstspeller-gt-norm.tmp.hfst .generated/generator-fstspeller-gt-norm.hfst
/usr/local/bin/hfst-fst2fst  -f openfst-tropical .generated/generator-fstspeller-gt-norm.hfst -o .generated/generator-mobilespeller-gt-norm-base.hfst
/usr/local/bin/hfst-reweight -v  \
			-T ./weights/tags.reweight --arcs-only -i .generated/generator-mobilespeller-gt-norm-base.hfst \
		-o .generated/generator-mobilespeller-gt-norm-tag_weighted.hfst
Reading from .generated/generator-mobilespeller-gt-norm-base.hfst, writing to .generated/generator-mobilespeller-gt-norm-tag_weighted.hfst
Modifying weights 0.000000 < w < 340282346638528859811704183484516925440.000000 as 1.000000 * id(w) + 0.000000
only on arc weights, no end states
Reweighting convert(UNNAMED)...
Reading reweights from ./weights/tags.reweight
Modifying weights 0.000000 < w < 340282346638528859811704183484516925440.000000 as 1.000000 * id(w) + 2.000000 for symbol +Cmp
Modifying weights 0.000000 < w < 340282346638528859811704183484516925440.000000 as 1.000000 * id(w) + 1.000000 for symbol +Der/minen
Modifying weights 0.000000 < w < 340282346638528859811704183484516925440.000000 as 1.000000 * id(w) + 50000.000000 for symbol +Use/SpellNoSugg
/usr/bin/printf "\
	set encode-weights ON \n\
	read regex \
	    @\"filters/remove-usage-tags.hfst\" \
	.o. @\".generated/generator-mobilespeller-gt-norm-tag_weighted.hfst\" \
	.o. @\"filters/remove-word-boundary.hfst\"  \
	; \n\
	save stack .generated/generator-mobilespeller-gt-norm.tmp.hfst\n\
	quit\n" | /usr/local/bin/hfst-xfst -p -v
Using default output format OpenFst with tropical weight class
Using OpenFst's tropical weights as output
Reading from standard input...
variable encode-weights = ON
hfst[0]: ? bytes. 131662 states, 334472 arcs, ? paths
hfst[1]: hfst[1]: hfst[1]: .
cp -f .generated/generator-mobilespeller-gt-norm.tmp.hfst .generated/generator-mobilespeller-gt-norm.hfst
cp .generated/generator-mobilespeller-gt-norm.hfst generator-mobilespeller-gt-norm.hfst
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/make-hfstspeller-version-easter-egg.sh \
		fkv \
		../.. \
		0.1.1 \
		"mobile" \
		default \
		> easteregg.default.mobile.txt
echo "?*" | /usr/local/bin/hfst-regexp2fst -o .generated/anystar.hfst
cp .generated/anystar.hfst anystar.hfst
sed -e 's/^/nuvviDspeller:/' < easteregg.default.mobile.txt \
		| sed = \
		| sed 'N;s/\n/	/' \
		| perl -pe 's/(.)\t(.+)/\2\t\1/' \
		> easteregg.default.mobile.suggtxt
grep -h -v '^#' words.default.txt easteregg.default.mobile.suggtxt | grep -v '^$'   \
		| /usr/local/bin/hfst-strings2fst  -j \
		-o words.default.mobile.hfst
grep -v '^#' strings.default.txt | grep -v '^$' | cut -f1-2 \
		| /usr/local/bin/hfst-strings2fst  -j \
		| /usr/local/bin/hfst-concatenate   anystar.hfst - \
		| /usr/local/bin/hfst-concatenate - anystar.hfst   \
		-o strings.txt.default.mobile.hfst
cp -f strings.txt.default.mobile.hfst strings.all.default.mobile.hfst
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/editdist.py \
		--verbose \
		 \
		--epsilon='@0@' \
		--default-weight=10 \
		--regex \
		--input=editdist.default.txt \
		--output-file=editdist.default.mobile.regex \
		
/usr/local/bin/hfst-regexp2fst -S  -i editdist.default.mobile.regex\
	   --format=openfst-tropical \
		-o .generated/editdist.default.mobile.hfst
cp .generated/editdist.default.mobile.hfst editdist.default.mobile.hfst
/usr/local/bin/hfst-disjunct strings.all.default.mobile.hfst editdist.default.mobile.hfst \
		| /usr/local/bin/hfst-repeat -f 1 -t 2 \
		-o editdist.all.default.mobile.hfst
grep -v '^#' final_strings.default.txt | grep -v '^$' | cut -f1-2 \
		| /usr/local/bin/hfst-strings2fst  -j \
	   --format=openfst-tropical \
		-o final_strings.txt.default.mobile.hfst
cp -f final_strings.txt.default.mobile.hfst final_strings.all.default.mobile.hfst
printf "\
		[ @\"words.default.mobile.hfst\"    \
		| \
		  [ \
		       \
		      @\"editdist.all.default.mobile.hfst\"  \
		      ( @\"final_strings.all.default.mobile.hfst\" ) \
		  ] \
		];" \
		| /usr/local/bin/hfst-regexp2fst -S -E  \
		| /usr/local/bin/hfst-push-weights --push=initial    \
		| /usr/local/bin/hfst-fst2fst  -f olw   \
		  -o errmodel.default.mobile.hfst
/usr/local/bin/hfst-strings2fst  -j < easteregg.default.mobile.txt \
		| /usr/local/bin/hfst-project  --project=lower > easteregg.default.mobile.hfst
/usr/local/bin/hfst-project  \
			-v --project=lower < generator-mobilespeller-gt-norm.hfst \
		| /usr/local/bin/hfst-remove-epsilons  -v                       \
		| /usr/local/bin/hfst-disjunct -v - easteregg.default.mobile.hfst          \
		| /usr/local/bin/hfst-push-weights -v --push=initial           \
		| /usr/local/bin/hfst-fst2fst -v  -f olw          \
		-o acceptor.default.mobile.hfst
Reading from <stdin>, writing to <stdout>
Reading from <stdin>, writing to <stdout>
Reading from - and easteregg.default.mobile.hfst, writing to <stdout>
Reading from <stdin>, writing to <stdout>
Reading from <stdin>, writing to acceptor.default.mobile.hfst
Writing Hfst's lookup optimized, weighted format transducers with HFST3 headers
Projecting second <stdin>...
Removing epsilons project-2nd(UNNAMED)...
Disjuncting remove-epsilons(project-2nd(UNNAMED)) and project-2nd(strings(?))...
Pushing towards start union(remove-epsilons(project-2nd(UNNAMED)), project-2nd(strings(?)))...
Converting push-weights-i(union(remove-epsilons(project-2nd(UNNAMED)), project-2nd(strings(?))))...
rm -f fkv-mobile.zhfst
/usr/local/bin/gmkdir -p build/fkv-mobile.zhfst
rm -f build/fkv-mobile.zhfst/*
cp index.mobile.xml build/fkv-mobile.zhfst/index.xml
cp acceptor.default.mobile.hfst build/fkv-mobile.zhfst/acceptor.default.hfst
cp errmodel.default.mobile.hfst build/fkv-mobile.zhfst/errmodel.default.hfst
cd build/fkv-mobile.zhfst/ && /usr/bin/zip -0   ../../fkv-mobile.zhfst *
  adding: acceptor.default.hfst (stored 0%)
  adding: errmodel.default.hfst (stored 0%)
  adding: index.xml (stored 0%)
/usr/local/bin/gmkdir -p 3
cp -f fkv-mobile.zhfst 3/
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/predict.py \
		--verbose \
		--epsilon='@0@' \
		--default-weight=10 \
		--regex \
		--input=editdist.default.txt \
		--output=predict.default.regex
being verbose...
reading from editdist.default.txt...
writing to   predict.default.regex...
"/usr/local/bin/hfst-regexp2fst"  --format=foma \
		 -v -S predict.default.regex -o predict.default.hfst
Reading from predict.default.regex, writing to predict.default.hfst
Compiling expression #1
Compiling expression #2
hfst-fst2fst -f olw -v -i predict.default.hfst -o errmodel.default.predict.hfst
Reading from predict.default.hfst, writing to errmodel.default.predict.hfst
Writing Hfst's lookup optimized, weighted format transducers with HFST3 headers
Converting xre(?)...
/usr/local/bin/hfst-fst2fst -f olw -i acceptor.default.mobile.hfst -o acceptor.default.predict.hfst
rm -f fkv-x-predict-mobile.zhfst
/usr/local/bin/gmkdir -p build/fkv-x-predict-mobile.zhfst
rm -f build/fkv-x-predict-mobile.zhfst/*
cp index.predict.xml build/fkv-x-predict-mobile.zhfst/index.xml
cp acceptor.default.predict.hfst build/fkv-x-predict-mobile.zhfst/acceptor.default.hfst
cp errmodel.default.predict.hfst build/fkv-x-predict-mobile.zhfst/errmodel.default.hfst
cd build/fkv-x-predict-mobile.zhfst/ && /usr/bin/zip -0   ../../fkv-x-predict-mobile.zhfst *
  adding: acceptor.default.hfst (stored 0%)
  adding: errmodel.default.hfst (stored 0%)
  adding: index.xml (stored 0%)
/usr/local/bin/gmkdir -p 3
cp -f fkv-x-predict-mobile.zhfst 3/
/usr/local/bin/hfst-invert -v  -i .generated/generator-speller-gt-norm.hfst -o .generated/analyser-speller-gt-norm.hfst
Reading from .generated/generator-speller-gt-norm.hfst, writing to .generated/analyser-speller-gt-norm.hfst
Inverting .generated/generator-speller-gt-norm.hfst...
cp .generated/analyser-speller-gt-norm.hfst analyser-speller-gt-norm.hfst
cp .generated/generator-fstspeller-gt-norm.hfst generator-fstspeller-gt-norm.hfst
/usr/local/bin/hfst-invert -v  -i .generated/generator-fstspeller-gt-norm.hfst -o .generated/analyser-fstspeller-gt-norm.hfst
Reading from .generated/generator-fstspeller-gt-norm.hfst, writing to .generated/analyser-fstspeller-gt-norm.hfst
Inverting .generated/generator-fstspeller-gt-norm.hfst...
cp .generated/analyser-fstspeller-gt-norm.hfst analyser-fstspeller-gt-norm.hfst
/usr/local/bin/hfst-fst2fst --format=openfst-tropical -i .generated/generator-fstspeller-gt-norm.hfst -o .generated/generator-desktopspeller-gt-norm-base.hfst
/usr/local/bin/hfst-reweight -v  \
			-T ./weights/tags.reweight --arcs-only -i .generated/generator-desktopspeller-gt-norm-base.hfst \
		-o .generated/generator-desktopspeller-gt-norm-tag_weighted.hfst
Reading from .generated/generator-desktopspeller-gt-norm-base.hfst, writing to .generated/generator-desktopspeller-gt-norm-tag_weighted.hfst
Modifying weights 0.000000 < w < 340282346638528859811704183484516925440.000000 as 1.000000 * id(w) + 0.000000
only on arc weights, no end states
Reweighting convert(UNNAMED)...
Reading reweights from ./weights/tags.reweight
Modifying weights 0.000000 < w < 340282346638528859811704183484516925440.000000 as 1.000000 * id(w) + 2.000000 for symbol +Cmp
Modifying weights 0.000000 < w < 340282346638528859811704183484516925440.000000 as 1.000000 * id(w) + 1.000000 for symbol +Der/minen
Modifying weights 0.000000 < w < 340282346638528859811704183484516925440.000000 as 1.000000 * id(w) + 50000.000000 for symbol +Use/SpellNoSugg
/usr/local/bin/hfst-strings2fst  -j < easteregg.default.desktop.txt \
		> .generated/easteregg.default.desktop.hfst
/usr/bin/printf "\
	set encode-weights ON \n\
	read regex [ \
	    @\"filters/remove-usage-tags.hfst\" \
	.o. @\".generated/generator-desktopspeller-gt-norm-tag_weighted.hfst\" \
	.o. @\"filters/remove-word-boundary.hfst\" ] \
	| @\".generated/easteregg.default.desktop.hfst\" \
	; \n\
	save stack .generated/generator-desktopspeller-gt-norm.tmp.hfst\n\
	quit\n" | /usr/local/bin/hfst-xfst -p -v
Using default output format OpenFst with tropical weight class
Using OpenFst's tropical weights as output
Reading from standard input...
variable encode-weights = ON
hfst[0]: ? bytes. 131751 states, 334564 arcs, ? paths
hfst[1]: hfst[1]: hfst[1]: .
cp -f .generated/generator-desktopspeller-gt-norm.tmp.hfst .generated/generator-desktopspeller-gt-norm.hfst
cp .generated/generator-desktopspeller-gt-norm.hfst generator-desktopspeller-gt-norm.hfst
/usr/local/bin/hfst-invert -v  -i .generated/generator-desktopspeller-gt-norm.hfst \
	| /usr/local/bin/hfst-prune-alphabet -v \
	| /usr/local/bin/hfst-remove-epsilons -v -o .generated/analyser-desktopspeller-gt-norm.hfst
Reading from .generated/generator-desktopspeller-gt-norm.hfst, writing to <stdout>
Reading from <stdin>, writing to <stdout>
Reading from <stdin>, writing to .generated/analyser-desktopspeller-gt-norm.hfst
Inverting .generated/generator-desktopspeller-gt-norm.hfst...
Pruning invert(UNNAMED)...
Removing epsilons prune-alphabet(invert(UNNAMED))...
cp .generated/analyser-desktopspeller-gt-norm.hfst analyser-desktopspeller-gt-norm.hfst
/Users/ttr000/git/giellalt/lang-fkv/./../giella-core/scripts/editdist.py \
		--verbose \
		 \
		--epsilon='@0@' \
		--default-weight=10 \
		--regex \
		--input=editdist.default.txt \
		--output-file=editdist.default.regex \
		
"/usr/local/bin/hfst-regexp2fst"  --format=foma \
		 -v -S editdist.default.regex -o editdist.default.hfst
Reading from editdist.default.regex, writing to editdist.default.hfst
Compiling expression #1
warning: '<CORR> ' is an ordinary symbol in hfst
warning: '<CORR> ' is an ordinary symbol in hfst

[ then the previous line was repeated 4000 times] <== comment, TT

warning: '<CORR> ' is an ordinary symbol in hfst
warning: '<CORR> ' is an ordinary symbol in hfst
warning: '@_UNKNOWN_SYMBOL_@' is not an ordinary symbol in hfst
warning: '<CORR> ' is an ordinary symbol in hfst
warning: '<CORR> ' is an ordinary symbol in hfst

[ then the previous line was repeated 40 times ] <== comment, TT

warning: '<CORR> ' is an ordinary symbol in hfst
warning: '<CORR> ' is an ordinary symbol in hfst
warning: '<CORR> ' is an ordinary symbol in hfst
Compiling expression #2
grep -v '^#' strings.default.txt | grep -v '^$' | cut -f1-2 \
		| /usr/local/bin/hfst-strings2fst  -j \
		| /usr/local/bin/hfst-concatenate   .generated/anystar.hfst - \
		| /usr/local/bin/hfst-concatenate - .generated/anystar.hfst   \
		-o .generated/strings.txt.default.hfst
cp .generated/strings.txt.default.hfst strings.txt.default.hfst
cp -f strings.txt.default.hfst strings.all.default.hfst
/usr/local/bin/hfst-disjunct strings.all.default.hfst editdist.default.hfst \
		| /usr/local/bin/hfst-repeat -f 1 -t 2 \
		-o editdist.all.default.hfst
/usr/local/bin/hfst-disjunct: warning: Transducer type mismatch in strings.all.default.hfst and editdist.default.hfst; using former type as output
grep -v '^#' final_strings.default.txt | grep -v '^$' | cut -f1-2 \
		| /usr/local/bin/hfst-strings2fst  -j \
	   --format=openfst-tropical \
		-o .generated/final_strings.txt.default.hfst
cp .generated/final_strings.txt.default.hfst final_strings.txt.default.hfst
cp -f final_strings.txt.default.hfst final_strings.all.default.hfst
printf "\
		[ @\"words.default.hfst\"    \
		| \
		  [ \
		       \
		      @\"editdist.all.default.hfst\"  \
		      ( @\"final_strings.all.default.hfst\" ) \
		  ] \
		];" \
		| /usr/local/bin/hfst-regexp2fst -S -E  \
		| /usr/local/bin/hfst-push-weights --push=initial    \
		| /usr/local/bin/hfst-fst2fst  -f olw   \
		  -o errmodel.default.hfst
/usr/local/bin/hfst-invert -v  -i .generated/generator-mobilespeller-gt-norm.hfst -o .generated/analyser-mobilespeller-gt-norm.hfst
Reading from .generated/generator-mobilespeller-gt-norm.hfst, writing to .generated/analyser-mobilespeller-gt-norm.hfst
Inverting .generated/generator-mobilespeller-gt-norm.hfst...
cp .generated/analyser-mobilespeller-gt-norm.hfst analyser-mobilespeller-gt-norm.hfst
/usr/local/bin/hfst-compose -1 filters/remove-error-strings.hfst -2 generator-desktopspeller-gt-norm.hfst -F \
		| /usr/local/bin/hfst-project  \
			-v --project=lower \
		| /usr/local/bin/hfst-remove-epsilons  -v                       \
		| /usr/local/bin/hfst-disjunct -v - .generated/easteregg.default.desktop.hfst  \
		| /usr/local/bin/hfst-push-weights -v --push=initial           \
		| /usr/local/bin/hfst-fst2fst -v  -f olw          \
		-o acceptor.default.hfst
Reading from <stdin>, writing to <stdout>
Reading from <stdin>, writing to <stdout>
Reading from - and .generated/easteregg.default.desktop.hfst, writing to <stdout>
Reading from <stdin>, writing to <stdout>
Reading from <stdin>, writing to acceptor.default.hfst
Writing Hfst's lookup optimized, weighted format transducers with HFST3 headers
Projecting second compose(UNNAMED, UNNAMED)...
Removing epsilons project-2nd(compose(UNNAMED, UNNAMED))...
Disjuncting remove-epsilons(project-2nd(compose(UNNAMED, UNNAMED))) and strings(?)...
Pushing towards start union(remove-epsilons(project-2nd(compose(UNNAMED, UNNAMED))), strings(?))...
Converting push-weights-i(union(remove-epsilons(project-2nd(compose(UNNAMED, UNNAMED))), strings(?)))...
rm -f fkv.zhfst
/usr/local/bin/gmkdir -p build/fkv.zhfst
rm -f build/fkv.zhfst/*
cp index.xml build/fkv.zhfst/index.xml
cp acceptor.default.hfst build/fkv.zhfst/acceptor.default.hfst
cp errmodel.default.hfst build/fkv.zhfst/errmodel.default.hfst
cd build/fkv.zhfst/ && /usr/bin/zip -9   ../../fkv.zhfst *
  adding: acceptor.default.hfst (deflated 81%)
  adding: errmodel.default.hfst (deflated 80%)
  adding: index.xml (deflated 55%)
/usr/local/bin/gmkdir -p 3
cp -f fkv.zhfst 3/
ln -f fkv.zhfst fkv-desktop.zhfst
rm editdist.all.default.mobile.hfst .generated/analyser-speller-gt-norm.hfst final_strings.txt.default.hfst strings.txt.default.hfst editdist.default.mobile.hfst strings.txt.default.mobile.hfst predict.default.regex .generated/strings.txt.default.hfst final_strings.all.default.mobile.hfst .generated/editdist.default.mobile.hfst editdist.default.mobile.regex .generated/analyser-fstspeller-gt-norm.hfst words.default.mobile.hfst final_strings.all.default.hfst strings.all.default.mobile.hfst predict.default.hfst easteregg.default.mobile.suggtxt .generated/final_strings.txt.default.hfst strings.all.default.hfst final_strings.txt.default.mobile.hfst .generated/generator-speller-gt-norm.hfst
Making all in hyphenators
Making all in filters
make[3]: Nothing to be done for `all'.
Making all in .
make[3]: Nothing to be done for `all-am'.
Making all in grammarcheckers
Making all in filters
make[3]: Nothing to be done for `all'.
Making all in .
make[3]: Nothing to be done for `all-am'.
Making all in tests
make[3]: Nothing to be done for `all'.
Making all in mt
Making all in filters
make[3]: Nothing to be done for `all'.
Making all in .
make[3]: Nothing to be done for `all-am'.
Making all in apertium
Making all in filters
make[4]: Nothing to be done for `all'.
Making all in tagsets
make[4]: Nothing to be done for `all'.
Making all in .
make[4]: Nothing to be done for `all-am'.
Making all in cgbased
Making all in .
make[4]: Nothing to be done for `all-am'.
Making all in tts
Making all in .
make[3]: Nothing to be done for `all-am'.
Making all in test
make[3]: Nothing to be done for `all'.
make[2]: Nothing to be done for `all-am'.
Making all in docs
make[1]: Nothing to be done for `all'.
Making all in test
Making all in src
Making all in phonology
make[3]: Nothing to be done for `all'.
Making all in orthography
make[3]: Nothing to be done for `all'.
Making all in morphology
make[3]: Nothing to be done for `all'.
Making all in syntax
make[3]: Nothing to be done for `all'.
make[3]: Nothing to be done for `all-am'.
Making all in tools
Making all in spellcheckers
Making all in fstbased
Making all in desktop
Making all in hfst
make[6]: Nothing to be done for `all'.
make[6]: Nothing to be done for `all-am'.
Making all in mobile
make[6]: Nothing to be done for `all-am'.
make[5]: Nothing to be done for `all-am'.
make[4]: Nothing to be done for `all-am'.
Making all in mt
Making all in apertium
make[4]: Nothing to be done for `all'.
make[4]: Nothing to be done for `all-am'.
Making all in hyphenators
Making all in fstbased
make[4]: Nothing to be done for `all'.
Making all in patternbased
make[4]: Nothing to be done for `all'.
make[4]: Nothing to be done for `all-am'.
make[3]: Nothing to be done for `all-am'.
make[2]: Nothing to be done for `all-am'.

No rule to make target 'analyser-oahpa-gt-norm-dial_Jok.hfstol' or 'generator-gramcheck-gt-norm-dial_Jok.hfstol'

If --enable-oahpa then:

make[4]: Entering directory '/build/giella-fkv-0.2.0+g8333~b00aacd1-1~sid1/src/fst'
make[4]: *** No rule to make target 'analyser-oahpa-gt-norm-dial_Jok.hfstol', needed by 'all-am'.  Stop.

And if --enable-grammarchecker then make[4]: *** No rule to make target 'generator-gramcheck-gt-norm-dial_Jok.hfstol', needed by 'all-am'. Stop.

These used to work.

vislcg3 fails on korp.cg3

"/usr/bin/cg-comp" korp.cg3 korp.bin
korp.cg3: Error: Cannot stat functions.cg3 due to error -1 - bailing out!
Makefile:851: recipe for target 'korp.bin' failed

This breaks the build.

Error when running make

I get the following error when running make:

make[2]: *** No rule to make target `generator-dict-gt-norm-dial_Jok.hfstol', needed by `all-am'.  Stop.
rm generator-gt-norm.tmp.hfst analyser-disamb-gt-desc.tmp.xfst generator-raw-gt-desc.tmp.hfst analyser-dict-gt-desc.tmp.xfst analyser-gt-desc.tmp.xfst analyser-gt-norm-dial_Var.tmp.xfst analyser-gt-norm-dial_Jok.tmp.hfst analyser-gt-norm-dial_Jok.hfst analyser-gt-norm-dial_Por.hfst analyser-dict-gt-desc-mobile.tmp.hfst analyser-disamb-gt-desc.tmp1.xfst analyser-gt-norm-dial_Por.tmp.xfst analyser-gt-desc.tmp.hfst generator-dict-gt-norm.tmp.hfst analyser-dict-gt-desc.tmp.hfst generator-gt-desc.tmp.hfst analyser-gt-norm-dial_Var.hfst analyser-gt-norm-dial_Jok.tmp.xfst analyser-disamb-gt-desc.tmp.hfst analyser-gt-norm-dial_Var.tmp.hfst generator-dict-gt-norm.tmp.xfst analyser-gt-norm.tmp.hfst analyser-gt-norm-dial_Por.tmp.hfst generator-gt-desc.tmp.xfst analyser-disamb-gt-desc.tmp1.hfst analyser-dict-gt-desc-mobile.tmp.xfst analyser-raw-gt-desc.tmp.xfst
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

I have the following configuration:

./configure --enable-tokenisers --with-xfst --enable-dialects --enable-dicts

gut template apply does nothing

Expected outcome: updates to .gitignore.

Cf also giellalt/template-lang-und#4.

Error on make

Get the following:

make[2]: *** No rule to make target `generator-dict-gt-norm-dial_Jok.xfst', needed by `all-am'.  Stop.
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

Grammarchecker not tuned to dialect fsts

A Kven grammarchecker reáuires control over the dialect involved. But compiling grammarcheckers with --enable-dialecton is blocked:

 ./configure --enable-grammarchecker --enable-spellers --enable-dialects && make
(...)
make[2]: *** No rule to make target `generator-gramcheck-gt-norm-dial_Jok.hfstol', needed by `all-am'.  Stop.
make[2]: *** Waiting for unfinished jobs....
  HINTRSCT generator-raw-gt-desc.tmp1.hfst
make[1]: *** [all-recursive] Error 1
make: *** [all-recursive] Error 1

Compiling basic fsts with dialect option on works fine.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.