bigscience-workshop / evaluation Goto Github PK

Talk to Stas in engineering if need help with this

(per question raised about slide 6 at the evaluation meeting on 9/1).

As the repository gets larger, we will eventually need to decide on a code convention. Though preferences may vary, it's probably safe to stick to the setup in transformers, which is black + isort.

We could create a simplified Makefile and do something like

.PHONY: style

style:
	black .
	isort . --profile=black .

Alternatively, create a .pre-commit-config.yaml.

repos:
  - repo: https://github.com/psf/black
    rev: 21.7b0
    hooks:
      - id: black
  - repo: https://github.com/pycqa/isort
    rev: 5.9.3
    hooks:
      - id: isort
        args: ["--profile", "black"]

Use to specify the API that all others will follow

This might sounds like a bit of re-structuring but for the sake of future compatibility, I propose the following,

1. Move to huggingface trainer: This will help the repo to automatically adapt to deepspeed and all the exclusive features of transformers library.
2. We don't have to re-invent the wheel. Given that we are using huggingface trainer, we only need to implement the following functions for a trainer for different tasks.
   -- data_loader
   -- DataCollator
   -- compute_metrics
   -- predictions (if needed)
3. In case if we want to finetune our full model, we don't have to change a lot in the surface level.

I would love to take some responsibility if needed. Let me know. @jaketae @tianjianjiang @wilsonyhlee

Response generation in Schema-Guided Dialog (including shuffle challenge set)

#56 set up a basic unit test, but we have to consider what kind of tests we want to run. This is especially important given that GitHub workflows does not have any GPU support, and will thus take a non-trivial amount of time to complete even a basic simple benchmark run. The proposal is to ideate some ways in which we could make tests modular and reasonably fast.

use to test generalization to unseen domain; maybe use FLEX?

all 18 languages

Per request by the modeling WG, we'd like to create a simple evaluation benchmark.

This issue tackles adding the TyDi QA dataset to the benchmark.

E2E NLG (+shuffle challenge set)

use to test generalization to unseen task; maybe use FLEX?

Per request by the modeling WG, we'd like to create a simple evaluation benchmark.

This issue tackles adding the WMT dataset to the benchmark.

use to test generalization to unseen labels; maybe use FLEX?

use to test generalization to unseen task; maybe use FLEX?

including COVID+bfp02 challenge sets

Russian/English (+shuffle/numbers challenge sets)

use to test generalization to unseen domain; maybe use FLEX?

Spanish/German (including COVID challenge sets)

coordinate with whoever is working on SuperGLUE, we only need to include MNLI once. But NLI will be held-out from model training (whereas the other SuperGLUE tasks will not) so interpreting MNLI results is different from other superglue tasks.

use to test generalization to unseen task; maybe use FLEX?

use to test generalization to unseen language; maybe use FLEX?

with TURK/ASSET test sets (including bfp02+backtranslation challenge sets)

Coordinate with Meg Mitchell about this