fgnt / meeteval Goto Github PK
View Code? Open in Web Editor NEWMeetEval - A meeting transcription evaluation toolkit
License: MIT License
MeetEval - A meeting transcription evaluation toolkit
License: MIT License
Python 3.9.17 (main, Jul 28 2023, 05:54:52)
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from meeteval.wer.matching import orc_matching, mimo_matching
...: ref=[['a']]
...: hyps=['aaabb']
...: mimo_matching.mimo_matching_v4(ref, hyps)
...:
Out[1]: (2, [(0, 0)])
Python 3.11.4 (main, Jul 5 2023, 13:45:01) [GCC 11.2.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.14.0 -- An enhanced Interactive Python. Type '?' for help.
In [1]: from meeteval.wer.matching import orc_mat
ching, mimo_matching
...: ref=[['a']]
...: hyps=['aaabb']
...: mimo_matching.mimo_matching_v4(ref, hyps)
...:
Out[1]: (4, [(0, 0)])
See GitHub Action log: logs_144.zip
This issue is for documentation: See conda/conda#10757
ImportError: .../meeteval/wer/matching/cy_levenshtein.cpython-38-x86_[64-linux-gnu.so](http://64-linux-gnu.so/): undefined symbol: _ZSt28__throw_bad_array_new_lengthv
According to conda/conda#10757 this error can happen, when you use different gcc versions, e.g. conda and system gcc, for details, see that issue.
If you try to use CTM files with comments in them (as defined in https://github.com/usnistgov/SCTK/blob/f48376a203ab17f0d479995d87275db6772dcb4a/doc/infmts.htm#L285 ) meeteval throws a error.
Reason: When reading lines from CTM files, meeteval ignores empty lines, but lines with comments (starting with ";;") also need to be ignored.
Hi folks,
Thanks for your work here - much appreciated!
Would you please consider adding a LICENSE
text file to the repository?
Thanks in advance!
I am thinking of a metric for long-form ASR and segmentation. Consider the following scenario:
If reference is STM and hypothesis is CTM, this may correspond to computing the asclite aWER metric, but we also want to support (i) other kinds of systems that may not provide word-level timestamps, and (ii) tighter penalty on segmentation by providing reference CTM.
Additionally, we also want to be able to include multiple possible references (e.g., references may be orthographic or normalized in some way), although I understand that this may be beyond the scope of this toolkit.
I am looking for suggestions about what would be a good metric (if one exists) for this scenario.
(cc @MartinKocour since we were having related discussions.)
Hi,
Thank you for open sourcing these multispeaker ASR metrics :).
Just reporting some issue I encountered:
Installing as follows works fine
pip install cython
git clone [email protected]:fgnt/meeteval.git
pip install -e ./meeteval[cli]
However installing like:
pip install cython
pip install https://github.com/fgnt/meeteval/archive/refs/heads/main.zip
Causes circular import issue:
import meeteval
Traceback (most recent call last):
File "", line 1, in
File "/opt/conda/lib/python3.7/site-packages/meeteval/init.py", line 1, in
from . import io
ImportError: cannot import name 'io' from 'meeteval' (/opt/conda/lib/python3.7/site-packages/meeteval/init.py)
Very strange, wondering if you have any ideas for workarounds for this? The build software we are using doesn't support installation via the first method.
Thank you very much for sharing this repository. It is very useful to have a single repo with many audio metrics :)
Tools like pyannote allows us to choose the collar and if we want to compute DER on overlapped speech regions or not.
With mdeval
, we can specify the collar but it seems like there is no option for including overlapped speech in the metric or not.
Does that mean that by default it computes over overlapped regions? Or are they excluded for the calculations?
Thank you for your answer !
Is there a simple way to get the WER break-down into ins/del/sub when computing the ORC-WER?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.