Related: the second issue in #87.

Replace print with logging.info in

and

Cloud build request failed due to invalid build tag name. This is likely due to UNIQUE_POD_NAME not expanded properly.
Here is an example.

For example:
output-jsonnet-jsonnet::internal:: in 2024-02-11-64-dg-comparison

The name is incomplete because of having a ( in the namespace, and our parser mistook that as the ( for params:

jsonnet::internal::(anonymous namespace)::Interpreter::builtinExtVar(jsonnet::internal::LocationRange const&, std::__1::vector<jsonnet::internal::(anonymous namespace)::Value, std::__1::allocator<jsonnet::internal::(anonymous namespace)::Value> > const&)

Currently, both the function signature and function name store the function signature, which will be used as the unique function identifier.

We can simplify the code by only keeping the function signature.

Related:
#64 (comment)

The error is caused by 'Text too long'. Here is the message:
https://pantheon.corp.google.com/logs/query;aroundTime=2024-02-13T12:58:00.000Z;cursorTimestamp=2024-02-13T13:00:23.372992698Z;duration=PT15M;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22oss-fuzz%22%0Aresource.labels.location%3D%22us-central1%22%0Aresource.labels.cluster_name%3D%22llm-experiment%22%0Aresource.labels.namespace_name%3D%22default%22%0Alabels.k8s-pod%2Fbatch_kubernetes_io%2Fcontroller-uid%3D%228f6b14ca-f7a2-4653-85f1-90e542b95e79%22%20severity%3E%3DWARNING%0Atimestamp%3D%222024-02-13T13:00:23.372992698Z%22%0AinsertId%3D%225lnkmvgn576p21g1%22?project=oss-fuzz

The related logs before/after the message:
https://pantheon.corp.google.com/logs/query;query=resource.type%3D%22k8s_container%22%0Aresource.labels.project_id%3D%22oss-fuzz%22%0Aresource.labels.location%3D%22us-central1%22%0Aresource.labels.cluster_name%3D%22llm-experiment%22%0Aresource.labels.namespace_name%3D%22default%22%0Alabels.k8s-pod%2Fbatch_kubernetes_io%2Fcontroller-uid%3D%228f6b14ca-f7a2-4653-85f1-90e542b95e79%22%20severity%3E%3DWARNING;cursorTimestamp=2024-02-13T13:00:23.372992698Z;aroundTime=2024-02-13T12:58:00.000Z;duration=PT15M?e=-13802955&mods=logs_tg_prod&project=oss-fuzz

Handling this requires two tasks:

1. Investigate its root cause. This appears to happen in the code fixing step. We need to understand what the prompt was, and why it is overlong (e.g., Did the error parser parse too much text?)
2. Capture this error and log it so that the experiment won't break because of it.

We need unambiguous names:
Each benchmark should be a function, and a project may have multiple benchmarks/functions.

For example, our current benchmark.yaml should be project.yaml.
This will involve other renaming/refactoring modifications.

e.g. if the top of the crash stacktrace points to the target source file.

This requires replacing gsutil download with code similar to this example solution.

Again, this requires a temporary fix before #10 is ready so that all people can access ASTs. We will switch to something different once #10 is on.

Thanks, @trashvisor : )

Error examples.

This needs a temporary fix before #10 removes all AST functionalities, and similar error handling code after #10.

@trashvisor, thanks!

e.g. to increase coverage of a generated target.

Integrate this into production OSS-Fuzz projects so all projects benefit from coverage increases and new bugs.

Our current code automatically fails an LLM-generated fuzz target if it does not contain the function under test.
However, the current pattern-matching is native and has false negatives (i.e., function used correctly but not recognized by us), and rejects valid fuzz targets.

This relates to our function name parsing regex:

A quick temporary solution is checking the function names without special chars.
Later we can make better use of data from FI

• Automatically push docker images from the PR to gcloud
• Provide an easy trigger for experiments runs using the image from the PR

Related: #54

Similar to #22.

Analyzing the current set of compilation errors, and seeing what fixes can be made via jcc, prompt engineering or other to improve the results.

Context: #11

Capturing this error, logging it as a warning, and re-sendind the request is possibly the best solution for now before knowing the exact cause.

Some initial cloud build requests failed due to glcoud authentication errors (example1, example2). Some related observations and guesses:

1. This has recurred on multiple GKE experiments, yet I failed to reproduce them in local experiments. Maybe this is because we have to authenticate gcloud manually before local experiments?
2. The number of initial experiments affected by this seems random: sometimes only the first one, sometimes multiple. This could be due to parallelism in experiments.
3. This error disappears after a while in each experiment.

It's likely that we aren't handling template instantiations properly in our textcov diffing, leading to some inflated coverage numbers for certain C++ projects that use templates extensively.

Related: #54

Currently our prompt generation is tied to our template format here: https://github.com/google/oss-fuzz-gen/tree/main/prompts/template_xml

We should make it easier and more flexible for others to test different prompt generation strategies, by allowing these custom prompts to be python modules instead that look something like the following:

def generate(benchmark:  Benchmark) -> str:
  ...

i.e. the module would be expected to define a generate function which produces a full prompt to pass to the LLM.

Similarly, we should also make our generation/evaluation loop more configurable, e.g. extract the logic here:

into a driver.py that can be similarly replaced:

def evaluate(model: models.LLM,  benchmark: Benchmark, prompt_generator: Module):
  prompt = generator.generate(benchmark)
  targets = generate_targets(model, prompt)
  results = evaluate(targets) 
  ...

And tying this all together, the resulting invocations would look something like:

./run_all_experiments --driver /path/to/driver.py --prompt_generator prompts/custom_generator.py

Many of our benchmarks have incorrect target_name set, leading to bad runtime results.

This requires looking into the failures and selecting benchmarks from recent results based on the following guidelines:

1. < 30 benchmarks in total. This ensures we can finish the experiment quickly. Particularly for PR experiments.
2. <=2 benchmarks from one project to include more projects.
3. <= 1 benchmark with a 100% build rate or > 10% coverage increase from one project, with ~10 benchmarks in total. Well-performing benchmarks are for regression checks; No point in having too many.
4. <=1 benchmark failed with the same error from the same project to track improvements.
5. Avoid the same general error from different projects (e.g., size_t or others from standard lib undefined)
6. Interesting projects to include:

• tinyxml2
• icu
• avahi
• Project with complex function signature (e.g., #89, guetzli, abseil-cpp, cppitertools's operator*, etc.)

7. Some interesting failures to include in comparison:

• Incorrect path in #include <...> / Undefined function.
• Missing build failure.
• Function not called.
• Incorrect function usage.
• Need the defs of data types used in the function under test.
• Incorrect usage of the function under test.

8. <=1 relatively good performing benchmark from a project (e.g., 60% build rate, 1% coverage increase).

Given there are many benchmarks to select, please document all selected benchmarks with justification here for future references.
Feel free to document some interesting benchmarks that are not selected or not sure if they should be selected.

If more benchmarks are needed, use https://llm-exp.oss-fuzz.com/Result-reports/ochang-2024-01-25/sort.html.
However, their names are not as convenient as the one above. Some of them might not be available either.

Thanks, @jonathanmetzman : )

One-off experiments (with volumeMounts) failed due exceeding the 10Gi limit.
These two experiments (1, 2) failed because of this, but the corresponding jobs and PoEs were not preserved.

This can be reproduced by running an one-off experiment will all benchmark.

Before #10 is ready, the old public AST bucket should be public.

• Existing fuzz targets (path, binary name)
• Function implementations
• Data structure definitions
• Usages of functions and data structures.

For example,
from:

functions:
  - int main2(int argc, char **argv)
  - int scanopt_usage(scanopt_t *scanner, FILE *fp, const char *usage)
project: flex
target_path: /src/fuzz-main.c

to:

"functions":
- "name": "main2"
  "param_names":
  - "argc"
  - "argv"
  "param_types":
  - "int"
  - "char **"
  "return_type": "int"
  "signature": "int main2(int argc, char ** argv)"
- "name": "scanopt_usage"
  "param_names":
  - "scanner"
  - "fp"
  - "usage"
  "param_types":
  - "char **"
  - "struct._IO_FILE *"
  - "char *"
  "return_type": "int"
  "signature": "int scanopt_usage(char ** scanner, struct _IO_FILE * fp, char * usage)"

project: flex
target_path: /src/fuzz-main.c

I had some success manually including this in prompts for certain benchmarks to improve the quality of the fuzzing:

IMPORTANT: If the solution needs to load from a file,  you need to create a temporary file called "/tmp/input", write data to it from FuzzedDataProvider, and use that as the file input. Example:
<code>
FILE *handle = fopen("/tmp/input", "w");
std::vector<uint8_t> contents = stream.ConsumeRemainingBytes(); // or stream.ConsumeBytes
fwrite(contents.data(), contents.size(), 1, handle);
fclose(handle);

int result = ParseFile("/tmp/input");
</code>

Need to evaluate this against more targets.

This will likely be heavily dependent on #10, since we won't have existing targets to provide as context.

Allow comparing the results (e.g., build rate, crash rate, bug discovery) against past data.

Some of the currently generated targets incorrectly assume that the data parameter is null terminated, leading to false positive overflows when they're passed to functions expecting null terminated strings.

We need to better detect these, and experiment with including instructions in our prompts to avoid this.

1. Add a logging module that stores more data in each log's jsonPayload (e.g., project, function/benchmark, LLM response id (1-10), code fixing id(0-5), etc.). FuzzBench reference 1 , reference 2.
2. Replace current logging and print with it.

Example view:

This is useful to help us identify error messages to reproduce/fix them.

Some places to change:

1. No need to COPY
2. Need to pull from this repo in https://github.com/google/oss-fuzz-gen/blob/main/report/docker_run.sh

1. Test if this works on a few past targets (some successful cases and some failed ones).
2. Ask LLM to use it in the final problem of code generation prompt.
3. Add this to the example questions and solutions in code generation prompt in the same format.
4. Ask LLM to use it in the code fixing prompt if the error is about undeclared function.

Having this context will likely significantly improve the quality of targets generated.

The prompts should include:

• function source code
• function xrefs (sample callsites and usages)
• data structure definitions, including which header path they're defined in.

The header file where a function prototype is declared is hard to extract accurately, but to work around this we may be able to just include the function prototype in the target itself without knowing where the header is. The header files that contain the relevant type definitions that are part of the prototype will still need to be included.

This is likely another cause of missing error in code-fixing prompt.

We need to:

1. Capture this error.
2. Log the corresponding instance (i.e., project, benchmark, sample id).
3. Re-submit the experiment.