stanford-crfm / helm Goto Github PK

Currently, we are using CamelCase in the backend to make life easier on Javascript, but this makes casing inconsistent on the backend. We should just do the conversion.

The API should be in snake case. Most of this is in settings, so Javascript can just pass everything through.

Do some general code cleanup.

Paper: https://arxiv.org/pdf/2009.11462.pdf

For now, just run the unit tests.

So users can just do pip install crfm-benchmarking and run benchmarking without downloading the code.

Right now, queries are deterministic because of caching.

Add a random: <string> field in request, so that we can get the best of both worlds. This string would have to be propagated down into the cache, which currently is on the raw request, which we can't change.

Also, we would ideally support this feature in a backward compatible way so we don't blow away the existing cache. Not sure what the best way to do this is.

Currently, the cache for the LM uses jsonl. Switch it over to using sqlitedict.

Compute the difference between average over instances with some tag. See metric.py

We need to know the number of tokens in Service::make_request to keep track of the quotas accurately. Need to figure out how this is actually computed since each service presumably uses a different tokenizer (Jurassic has larger tokens for example).

When benchmarking, print out

• the Instances of a Scenario
• the RequestStates that are produced by the Adapter
• the Metrics and the resulting Stats.

Output:

• Summary statistics to stdout using htrack and hlog (this is basically fine)
• Human readable to a text file.
• Serializable information to a json file.

Format for text file:

Scenario ${scenario.name}
${scenario.description}
Tags: ${scenario.tags}
${n} instances

------- Instance ${i}/${total}: ${instance.tags}
Input: <input>
Reference(tags): ${reference.text}
...
Reference(tags): ${reference.text}

Where to store these files:

• Each RunSpec should be given a path (e.g., reasoning/mmlu/chemistry) which a new field of RunSpec.
• Output to scenario.{txt,json}, scenario_state.{txt,json}, metrics.{txt,json} in that path.

Ask in the OpenAI discussion forum.

More context here: #29

When user types in API key into the web interface, authenticate it right away and show an error message rather than waiting to see the error later.

Worst case over a group of instances with some tag - see metric.py

Run an nginx that routes to the server on port 1959 and use Let's Encrypt.

Create a thread pool with some parallelism in Executor, so that we can process the examples in parallel.

        # TODO: make a thread pool to process all of these in parallel up to a certain number
        def render_instance(instance: Instance, pred_output: Optional[str]) -> str:
            tags_str = ",".join(instance.tags)
            gold_output: Optional[str] = None
            if instance.first_correct_reference is not None:
                gold_output = instance.first_correct_reference.output

            if request_state.output_mapping is not None and pred_output is not None:
                pred_output = request_state.output_mapping.get(pred_output.strip())
            correct_str = "CORRECT" if gold_output == pred_output else "WRONG"
            return (
                f'[{tags_str}] "{instance.input[:100]}" => "{gold_output}", predicted "{pred_output}" [{correct_str}]'
            )

It's technically implemented, but we need to make sure that we're reproducing the numbers from the paper. Make sure all the decoding parameters are the same and that the eval set is the same. There is also some stuff involving truncation which we should take a look at.

def get_requirements(path: str):
    # TODO: don't include all the dev packages
    requirements = []
    for line in open(path):
        if not line.startswith('-r'):
            requirements.append(line.strip())
    return requirements

Verify package builds correctly.

In basic_metrics.py:

        # TODO: pass in arguments to `BasicMetric`
        return compute_metrics("exact_match", exact_match)

Create a RemoteService class that has the same signature as Service (for the functions used byServer). This can be used client-side. Then demo.py should be:

auth = Authentication(api_key="crfm")
service = RemoteService("http://crfm-models.stanford.edu")

# Make a request
request = Request(prompt="Life is like a box of")
print(service.make_request(auth, request))

# Modify account
account = service.get_account()
account.description = "Updated"
service.update_account(auth, account)

Behind the hood, it will make all the necessary REST calls.

See general.py: ensure_file_download.

wget:
-c,  --continue                  resume getting a partially-downloaded file

but, wget -c doesn't work for some URLs

Unicode contained in the completions - for example, see:
http://crfm-models.stanford.edu/static/index.html?prompt=A%20%24%7Boccupation%7D%20is%20someone%20who&settings=temperature%3A%200.5%0Amax_tokens%3A%20100%0Amodel%3A%20%24%7Bmodel%7D&environments=occupation%3A%20%5Bmathematician%2C%20lawyer%2C%20doctor%2C%20programmer%2C%20president%5D%0Amodel%3A%20%5Bai21%2Fj1-jumbo%2C%20openai%2Fdavinci%5D

Create the following functions (with corresponding REST endpoints) for managing accounts:

• Service.create_account(auth): creates a new account with a random API key and returns that Account
• Service.get_accounts(auth): returns a list of Account
• Service.update_account(auth, account): updates the account given by account.api_key with the fields in account.

Note that these methods should look at auth and the corresponding is_admin and make sure that the following permissions are enforced:

• If is_admin = True, then anything is possible EXCEPT the used field in any Usage object (because this is changing, we don't want to override it).
• Otherwise, one can only view their account and edit their account (EXCEPT api_key, usages).
  Any disallowed changes are just discarded silently. This allows the client to have a simple uniform interface for updating an Account.

We want to be able to run queries against GPT-2 and GPT-J.

Two options:

1. Write a separate server that allows us to spin up Hugging Face servers locally.
2. Try to use Hugging Face servers directly (https://api-inference.huggingface.co/docs/python/html/quicktour.html#api-options-and-parameters)

Let us start with 2 since it's simpler (we might want to do 1 later). Either way, need to write a subclass of Client to support the separate server with the HF model.

Note: the existing implementation HuggingFaceClient is broken and tries to load the model directly, which is not a good idea.

Add statistics with the individual instances too and serialize them out. See metric.py.

How to: https://www.py4u.net/discuss/204728

# TODO: put a flag on this so that it's easy to use pytest to still run these slow tests
@pytest.mark.skip(reason="Requires production")
def test_prod_continue():
    # Test that we're continuing
    prompt = "Paris is the capital of"
    for model in prod_models:
        request = Request(prompt=prompt, model=model, max_tokens=1, num_completions=1, temperature=0)
        helper_prod_test_service(request, " France")


@pytest.mark.skip(reason="Requires production")
def test_prod_echo():
    # If we're echoing the prompt, make sure we're getting the same thing back
    prompt = "I like pickles."
    for model in prod_models:
        request = Request(prompt=prompt, model=model, max_tokens=0, num_completions=1, echo_prompt=True)
        helper_prod_test_service(request, prompt)

The current API key we are using in production was created with the hai-gcp-models account and allows 200 queries per second:

"Thanks for sharing how you're using Perspective and contacting us about a quota increase. We reviewed your request and are happy to grant your project 200 queries per second until 7/30/2022."

When renewing, fill out the form with your Stanford email address. For GCP ID, put down hai-gcp-models.

See server_service.py:

        """Actually make a request to an API."""
        # TODO: try to invoke the API even if we're not authenticated, and if
        #       it turns out the results are cached, then we can just hand back the results.

Executing benchmark-run on the cluster machines should write all the scenarios, scenario states, and metrics to JSON files for each run specification. Display results as a text file for now.

In the web interface, try the third example query (or anything with codex). The last token of the prompt seems to be repeated in the first token of the completion.

The first log probability and top choices are None, so we skip. See openai_client.py

            for text, logprob, top_logprobs in zip(
                raw_data["tokens"], raw_data["token_logprobs"], raw_data["top_logprobs"]
            ):
                tokens.append(Token(text=text, logprob=logprob or 0, top_logprobs=dict(top_logprobs or {})))

In basic_metrics.py:

        # TODO: Sort the predictions, or take them from the top tokens of the first completion
        preds = [completion.text.strip() for completion in request_state.result.completions]

Make the RequestResult contain log probabilities.

See test_adapter.py

Dump the scenario states, but skip execution. Estimate the number of tokens in metrics.

When querying GPT-3, stop sequences doesn't seem to be respected.

http://crfm-models.stanford.edu/static/index.html?prompt=Answer%20the%20following%20question%20about%20geography.%0A%0AQuestion%3A%20What%20is%20the%20longest%20river%3F%0AAnswer%3A%20Nile%20%23%23%0A%0AQuestion%3A%20What%20is%20the%20tallest%20mountain%3F%0AAnswer%3A&settings=temperature%3A%200%0Astop_sequences%3A%20%5B%22%23%23%22%5D%0Atop_k_per_token%3A%205&environments=

In basic_metrics.py:

def compute_metrics(name: str, score_func: Callable[[str, str], float]) -> List[Stat]:

Used in the Gopher paper for measuring perplexity.

January 25 update: The LM pipeline has been implemented in #79. TODO: Implement the data filtering process that extracts tweets of different demographics from the raw data.

Design doc: https://docs.google.com/document/d/1RHIraoaL6NGj8RhDFv__-6PYqZxEWTRIfIpfGIeo2yI/edit#

Add ThePileScenario which allows you to sub select different eval sets. Do what Gopher did. Make sure we're not downloading more than needed.

Make Authentication api_key instead of username and password. The API keys will be randomly generated and assigned to people. The User isn't so much tied to an individual person, but rather a use case. Add a description field, and maybe make emails a list of emails. Also add group and is_admin.

Need to update the web interface too.

This is called hendryksTest in EleutherAI's LM Evaluation Harness.

Use black, mypy, flake8 (can copy what we have in CodaLab).

Move this to a configuration file (default_quotas.conf alongside accounts.jsonl):

DEFAULT_QUOTAS = {
    # model group -> {granularity -> quota}
    "gpt3": {"daily": 10000},
    "codex": {"daily": 10000},
    "jurassic": {"daily": 10000},
}

Currently, everything is in src.

• Move the clients, users, the server code to a separate directory service for managing access to the APIs.
• Split out models.py into different files. Put common files in the src directory.
• Create a separate directory benchmark for scenarios, metrics, etc..

Test RemoteService.

From https://community.openai.com/t/how-do-i-calculate-the-pricing-for-generation-of-text/11662/5 , "we’re planning to implement a tokenizer into the client to make it easier. And adding an easy cost estimator into the api would be a great addition too."