The orbis_eval from orbis-eval

Support for retrieving different corpus versions from the web

idea: create an evaluation.yaml that obtains corpora from the web.

evaluation_name: reuters128-flavours
versions:
  https://github.com/orbis/reuters128-v1.zip
  https://github.com/orbis/reuters128-v2.zip
  https://github.com/orbis/reuters128-dbpedia3-v0.zip

cheers, albert :)

Feature request: Search for terms/phrases that have been misclassified

Workflow:

the user opens an evaluation and sees that a certain mention (e.g., "Apple") has been misclassified.
allow the user to trigger a search for this phrase (e.g., via a search field or by clicking on it).
all corpus passages that contain the phrase (e.g., the sentence + a context window of 10 tokens) are presented in the Orbis evaluation window (left: gold standard, right: system results).

e.g.,

doc1:
... in August 2021 Apple Inc. announced ... 
... Apple has become ...

doc2
... Apple is considered an ambiguous term...

@adib2011 - what do you think of this idea?

orbis_plugin_aggregation_local_cache missing in clone_plugins.sh

I tried to follow the instructions for "Local Development (Pycharm)". When trying to run orbis_eval/main.py -t for the first time to create the folder the process stopped with the following error message:

/Users/reslr/git/Orbis/orbis_eval/venv/bin/python /Users/reslr/git/Orbis/orbis_eval/orbis_eval/main.py -t
User folder location not found. Creating new...

Where would you like to install the Orbis user directory?
> (/Users/reslr/orbis-eval):

Creating: /Users/reslr/orbis-eval
Copying: /Users/reslr/git/Orbis/orbis_eval/orbis_eval/data/tests/corpora -> /Users/reslr/orbis-eval/data/corpora
Copying: /Users/reslr/git/Orbis/orbis_eval/orbis_eval/data/tests/queue -> /Users/reslr/orbis-eval/queue/tests
Copying: /Users/reslr/git/Orbis/orbis_eval/orbis_eval/data/tests/../orbis_config -> /Users/reslr/orbis-eval/config
User settings location not found. Creating new...

Copying: /Users/reslr/git/Orbis/orbis_eval/orbis_eval/data/tests/../orbis_config -> /Users/reslr/orbis-eval/config
2022-03-25 08:40:50 alita.local __main__[13779] INFO Welcome to Orbis!
2022-03-25 08:40:50 alita.local orbis_eval.libs.config[13779] INFO Adding harvest_weblyzard_harvest_local.yaml to queue.
2022-03-25 08:40:50 alita.local orbis_eval.libs.config[13779] INFO Adding rss1_aida_local.yaml to queue.
2022-03-25 08:40:50 alita.local orbis_eval.libs.config[13779] INFO Adding rss1_aida_web.yaml to queue.
2022-03-25 08:40:50 alita.local orbis_eval.libs.config[13779] INFO Adding rss1_babelfly_local.yaml to queue.
2022-03-25 08:40:50 alita.local orbis_eval.libs.config[13779] INFO Adding rss1_babelfly_web.yaml to queue.
2022-03-25 08:40:50 alita.local orbis_eval.libs.config[13779] INFO Adding rss1_spotlight_local.yaml to queue.
2022-03-25 08:40:50 alita.local orbis_eval.libs.config[13779] INFO Adding rss1_spotlight_web.yaml to queue.
2022-03-25 08:40:50 alita.local orbis_eval.plugins.aggregation.monocle.main[13779] INFO Checking for resources
2022-03-25 08:40:50 alita.local orbis_eval.libs.cli[13779] INFO Starting: Lense
2022-03-25 08:40:50 alita.local orbis_eval.libs.cli[13779] INFO Runtime: (00:00:00.00) (lense)
2022-03-25 08:40:50 alita.local orbis_eval.libs.cli[13779] INFO Starting: Mapping
2022-03-25 08:40:50 alita.local orbis_eval.libs.cli[13779] INFO Runtime: (00:00:00.00) (mapping)
2022-03-25 08:40:50 alita.local orbis_eval.libs.cli[13779] INFO Starting: Filter
2022-03-25 08:40:50 alita.local orbis_eval.libs.cli[13779] INFO Runtime: (00:00:00.00) (filter)
2022-03-25 08:40:50 alita.local orbis_eval.core.pipeline[13779] INFO Running: harvest_weblyzard_harvest_local.yaml
2022-03-25 08:40:50 alita.local orbis_eval.core.pipeline[13779] INFO Starting aggregation for harvest_weblyzard_harvest_local.yaml
Traceback (most recent call last):
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/main.py", line 88, in <module>
    run()
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/main.py", line 84, in run
    run_orbis(args.config or None, args)
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/main.py", line 62, in run_orbis
    start_runner(config)
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/main.py", line 25, in start_runner
    p.run()
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/core/pipeline.py", line 53, in run
    self.rucksack = Aggregation(self.rucksack).run()
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/core/pipeline.py", line 107, in run
    self.rucksack.pack_computed(self.run_plugin(self.pipeline_stage_name, self.aggregator_service,
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/core/pipeline.py", line 44, in run_plugin
    plugin = cls.get_plugin(cls, pipeline_stage_name, plugin_name)
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/core/pipeline.py", line 37, in get_plugin
    imported_module = load_plugin(pipeline_stage_name, plugin_name)
  File "/Users/reslr/git/Orbis/orbis_eval/orbis_eval/libs/plugins.py", line 22, in load_plugin
    return orbis_plugins[module_name]
KeyError: 'orbis_plugin_aggregation_local_cache'

Process finished with exit code 1

The reason seems to be the missing plugin orbis_plugin_aggregation_local_cache. This wasn't installed unlike the other plugins as it was missing from clone_plugins.sh. after cloning that repo manually main.py -t ran without errors.

orbis-eval / orbis_eval Goto Github PK

orbis_eval's People

Stargazers

Watchers

Forkers

orbis_eval's Issues

Support for retrieving different corpus versions from the web

Feature request: Search for terms/phrases that have been misclassified

orbis_plugin_aggregation_local_cache missing in clone_plugins.sh

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent