Current Differential Testing Logic during Fuzzing

tvm-optimized v.s. tvm-debug;

ort-optimized v.s. ort-debug;

This is not quite effective as we are going to compile the same model twice. The compilation is kinda expensive so...

Expectation

As we use PyTorch to generate models, we can simply use PyTorch as the oracle.

So basically when doing input generation, we want to verify that outputs do not have (or have a lot of) NaN values. The <input, output> pair can be used as data samples to test the target framework. We can simply use IPC to transfer the data in the form of Tuple[Dict[str, np.array], Dict[str, np.array]] or simply Tuple[List[np.array], List[np.array]].

Would like any volunteers to handle this as I am working on other features. Suggestions are welcomed.

Since both latest PyTorch and TensorFlow are deprecating Python 3.7. Maybe we should also embrace Python 3.8+ to make use of more features.

There are a list of changes where we can make it mandatory:

• Use f-string with "equals" operator to simplify "terms = {terms}". @Co1lin
• Tweak setup.cfg to constraint python_version>=3.8

cc: @Co1lin

• Sanitize and push TF coverage evaluation scripts.
• Add tutorial in experiments/README.md.

The thing is that the coverage tool is extended to be used by multiple libraries.

When fuzzing TVM, we should avoid importing onnxruntime as it will mess with the coverage...

However, the NaN checker relies on onnxruntime which makes things much harder...

So @lazycal do you think there is other options to replace onnxruntime for NaN checking? Anyhow, I would say not using tvm and onnxruntime as "utility" because it will mess with the coverage. Maybe we can consider PyTorch (JIT)?

Description

I used nnsmith to generate a set of models and extracted features from them. In a model, I found a tensor with a very strange shape, as follows:

I don't quite understand what this li64, 40i64 is.

Installation

pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade

Reproduction

# Paste the commands or python script for reproducing the issue.

Screenshots

![DESCRIPTION](LINK.png)

Logs

No response

Front-end framework

onnx

Version of the front-end framework

No response

Back-end engine

tvm

Version of the front-end engine

No response

Other context

No response

Description

I wanted to excute nnsmith.model_gen mgen.max_nodes=5 mgen.dtype_choices="[f32]" model.type=onnx debug.viz=false mgen.save=$path. But it reurned :

Could not override 'mgen.dtype_choices'.
To append to your config use +mgen.dtype_choices=[f32]
Key 'dtype_choices' is not in struct
    full_key: mgen.dtype_choices
    object_type=dict

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

When I excuted without mgen.dtype_choices="[f32]", it can work correctly. Could anyone tell me what should I do?

Installation

pip install "nnsmith[torch,onnx]" --upgrade

Reproduction

# Paste the commands or python script for reproducing the issue.

Screenshots

![DESCRIPTION](LINK.png)

Logs

No response

Front-end framework

No response

Version of the front-end framework

No response

Back-end engine

No response

Version of the front-end engine

No response

Other context

No response

Excuse me. I encountered a series of errors while converting the onnx model generated by nnsmith to tvm.relay. I would like to ask if this is a bug caused by an internal error in TVM or an error in the model generated by NNSmith.



The command I am using is "tvm.relay.frontend.onxx.from_onnx(onnx_model)".

Motivation: Test a few more operators.

We should focus on operators which are often common and optimizable (of course supported by PyTorch and can be directly mapped to ONNX). For example, we don't want to add more trivial element-wise layers (it's hard to implement it wrong, right?).

Easy

• torch.tril
• torch.triu
• torch.nn.ConvTranspose1d
• torch.nn.ConvTranspose2d
• torch.nn.ConvTranspose3d
• torch.nn.Conv1d
• torch.nn.Conv3d
• torch.cumsum
• torch.nn.functional.one_hot

Medium

• torch.scatter
• torch.gather
• torch.topk

Hard

• torch.einsum 🤔

Motivation

Hi there, when I invoke nnsmith.model_gen ... mgen.save=gen/, it warns me the report folder already exists and let me choose, even if such directory doesn't exist in $pwd.

Basic Example

If my guess is right, now it will warns me the reports folder exists if nnsmith has seen this path before.

Drawbacks

It will facilitate the integration of nnsmith for automation, like, I was not warned to choose Y/N every single run. Is there a way to disable this reminder? :)

How do I know if this model has caused an error after executing the "nnsmith.model_exec model.type=onnx backend.type=onnxruntime model.path=nnsmith_output/model.onnx", " nnsmith.model_exec model.type=onnx
backend.type=onnxruntime
model.path=nnsmith_output/model.onnx
cmp.with='{type:tvm, optmax:true, target:cpu}'"?

Those logs are so frequent and will mess up with rich console printer.

Submitted an issue to PyTorch and let's track it here.

pytorch/pytorch#77494

It is not a good workaround to use check_call here which:

• requires everything to be re-initialized.
• makes it extremely harder to perform stateful fuzzing (I find it hard to merge my new implementation here).

We can discuss better solutions in this thread.

Previously my implementation makes the generation phase in-process which works well. It seems there are some issues regarding timeout and generation failure.

• If timeout is a bug of z3 then we should avoid any usage of it;
• In terms of generation failure, we should be able to detect it by catching exceptions;
• At least we should use sub-process instead of check_call to allow stateful fuzzing;

We can discuss more in this thread.

Hi, I'm seeking a way to generate runable python code from bug report directory, so that I can reproduce bugs without NNSmith's code. I found that there is a class named Render that may help me to do this. However, It does not work well when I want to emit a TorchModel. It seems that fx.symbolic_trace cannot generate the model declaration in TorchMdel.emit_def(). Is the Render supposed to work but I did not use it right? Or they are just some deprecated code?

There has been a rich set of improvements since v0.0.1 (the preview version). a8307f2...8cee6f6 (until 03/08/2022). Here, some noticeable changes are listed, as part of the release for v0.1.0. Additional non-documentation improvements will also be listed and targeted for v0.1.0.

Following standard open-source convention, a release candidate will be pre-released and later "sealed" until no issues are found in two weeks.

Main improvements

• Adding more ops for TF (#55)
• Add UB and vulnerable ops filter (#56)
• Resuming coverage instrumentation (#60)
• GraphIR for systematic model manipulation (#61, #64, #66)
• Adding versioning information for nnsmith/model/backend (#63)
• Concolic generation (#66)
• Sharpening and testing user-defined constraints (#72)
• Include and exclude ops by name and data types (#72)
• Adding more complete data types (#74, #76)
• Deprecate support for iree (#74)

Other improvements

• Fixing or improving operator rules: Pool2d, InterpBase, ReduceBase (#66), Reshape (11a29ef), broadcast, Abs, ReflectPad, ArgMin/Max (#74).
• Adapting TF's naming convention in GraphIR's var names (#66)
• Harnessing hydra dependencies (#66, #68)
• Documentation (#57, #58, #60, bf1f592, dce784a, ba486ca)

TBD

• Upstream backend of torchjit. @Co1lin

How is the shape of the output tensor of each operator obtained in the image output by pygraphviz?

Motivation

Hi, I'm studying the implementation of NNSmith. Is there any tutorial telling developers how to add a new operator for GIR? I want to add a Split operator to the GIR, but still try to find a way. I think a tutorial would be very helpful.

As far as I understand, one should add a new class for the operator in nnsmith/abstract/op.py, and add some functions in nnsmith/backends/*. Do I understand it correctly?

Basic Example

None.

Drawbacks

None.

Reference Issues

No response

Other context

No response

Contribution

I can partially help.

Hi. I am currently learning to use nnsmith. I would like to know if I can customize the generated graph as a graph with only a single input variable and a single output variable.

Is there any way to quickly implement this?

A lot of inconsistency bugs show something similar below:

/home/ganler/miniconda3/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:55: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
  warnings.warn("Specified provider '{}' is not in available provider names."
Process Process-12:
Traceback (most recent call last):
  File "/home/ganler/miniconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/ganler/miniconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "nnsmith/fuzz.py", line 310, in difftest
    assert_allclose(
  File "/home/ganler/Documents/nnsmith/nnsmith/difftest.py", line 22, in assert_allclose
    raise IncorrectResult(
nnsmith.error.IncorrectResult: torch v.s. ort-opt have different output tensor names: {'o1', 'o2', 'o3', 'o0'} ~ {'73', 'o2', 'o3', 'o0'}

'73' is not an expected output name.

Before open-sourcing the nnsmith project, I want to simplify and standardize a bit the development and user accessibility. Below is a tracking list of TODOs for @ganler to make the repository a better open-source one in the next few months.

• Setup pre-commit;
• Setup black formatting (within pre-commit);
• Improve code quality and make NNSmith extensible.
  • Sync with paper naming.
  • Modularize data in abstract domain (nnsmith/abstract);
  • Make model generation dependency-minimal and split it against model materialization;
  • Make operator constraints extensible for other model formats;
  • Refine backend runners to allow different model formats;
  • Operator dialect;
  • Add unit tests for serialization, op spec validity, etc.
  • Hydra configuration;
  • Fix dtype_test, fuzz and backend execution [X];
  • Allow single-step fuzzing (currently we do multi-step fuzzing for the sake of coverage evaluation);
• Setup Github CI; Thanks to @Co1lin!
• Setup pip package;

I've install nnsmith by using pip3.8 install "nnsmith[torch,onnx]" --upgrade and install successfully. But when I try to excute

nnsmith.fuzz fuzz.time=86400s model.type=onnx backend.type=tvm fuzz.root=fuzz_report_1 debug.viz=true mgen.max_nodes=100

I met fuzz error. The part of the fuzz.log is as follow:

[2023-02-14 10:14:45,357][dtest][INFO] - Loading topset from /root/.cache/nnsmith/ONNXModel_tvm_llvm -keys=cpu .yaml.
[2023-02-14 10:14:45,357][dtest][INFO] - To regenerate the topset, delete the cache file and restart.
[2023-02-14 10:14:45,358][dtest][INFO] - rm /root/.cache/nnsmith/ONNXModel_tvm_llvm -keys=cpu .yaml
[2023-02-14 10:14:45,359][fuzz][INFO] - Test success info supressed -- only showing logs for failed tests
[2023-02-14 10:14:45,360][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,360][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,361][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3943882080 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,361][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,361][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,361][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=627982728 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,362][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,362][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,362][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=2470370689 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,362][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,362][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,363][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=2671980766 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,363][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,363][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,363][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3215383915 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,364][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,364][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,364][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=1079210586 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,364][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,364][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,365][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=1807477749 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,365][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,365][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,365][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3642741048 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,366][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,366][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,366][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3396546204 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,366][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,367][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,367][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=395719398 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,367][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,367][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,367][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3371485272 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,368][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,368][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,368][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3200508025 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,368][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,369][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

[2023-02-14 10:14:45,369][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=759925757 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,369][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,369][fuzz][ERROR] - Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
    testcase = self.make_testcase(seed)
  File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
    gen = random_model_gen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
    gen = PureSymbolGen(
  File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

Could anyone tell me what should I do next?

Seems to me the multi-input head and the constant nodes get unconnected quite often (almost always).

On the one hand, it helps us get new bugs on frameworks that poorly support this unreal models;

However, I feel that it produces a specific kind of bug in the very beginning all the time that might mask other bugs...

For example, 80% of TVM bugs look like this which shuts down the compilation in the very beginning.

That said, our current implementation of multi-input & constant node insertion might bias the model generation because we did not apply any strategies to consume these unconnected nodes. esp. for constant nodes.

I would recommend either reducing the probability of such kinds of nodes since they seldom connect prior nodes; or generating constant nodes on non-unary operators. For example, Add(Var, Cons) etc.

Hi. Thanks for developing this amazing project!

Just curious if nnsmith supports generating tests with dtype float16? Directly adding float16 to abstract/dtype.py seems not working.

Hi
I ran into this error, trying to run basic instruction written in Quick Start section
Any idea what might cause this error?

Traceback (most recent call last):
  File "/venv/test/bin/nnsmith.model_gen", line 5, in <module>
    from nnsmith.cli.model_gen import main
  File "/venv/test/lib/python3.8/site-packages/nnsmith/cli/model_gen.py", line 16, in <module>
    @hydra.main(version_base=None, config_path="../config", config_name="main")
TypeError: main() got an unexpected keyword argument 'version_base'

The command I used:
nnsmith.model_gen model.type=onnx debug.viz=true

Here is my env/lib settings/version:

python: 3.8.7
nnsmith: 0.0.1
z3-solver: 4.11.2
hydra-core: 1.1.2
hydra_colorlog: 1.0.0
multipledispatch: 0.6.0
appdirs: 1.4.4

Description

Hi!

When trying to run NNSmith on TorchJIT, I encountered errors like

ERROR   fuzz   - `make_testcase` failed with seed 1412986392. It can be NNSmith or Generator (torch) bug.
ERROR   fuzz   - Traceback (most recent call last):
  File "nnsmith/cli/fuzz.py", line 215, in run
    testcase = self.make_testcase(seed)
  File "nnsmith/cli/fuzz.py", line 187, in make_testcase
    model.refine_weights()  # either random generated or gradient-based.
  File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/__init__.py", line 40, in refine_weights
    _, inputs = searcher.search(
  File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/input_gen.py", line 56, in search
    res = self.search_one(cur_input, max_time_ms)
  File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/input_gen.py", line 110, in search_one
    return SamplingSearch.search_one(self, start_inp, timeout_ms)
  File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/input_gen.py", line 70, in search_one
    _ = self.net(**start_inp)
  File "/home/su/accdiff/thirdparty/pytorch/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/symbolnet.py", line 341, in forward
    output_tensors = inst(*input_tensors)
  File "/home/su/accdiff/thirdparty/pytorch/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/proxy_grad.py", line 108, in forward
    return PGReLUFunc.apply(x)
  File "/home/su/accdiff/thirdparty/pytorch/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/proxy_grad.py", line 40, in forward
    return torch.relu(input)
RuntimeError: "clamp_min_scalar_cpu" not implemented for 'Half'

Installation

Source file without edits

Reproduction

yes | python nnsmith/cli/fuzz.py --config-dir nnsmith/config/ model.type=torch backend.type=torchjit fuzz.root=fuzz_report debug.viz=true backend.target=cuda

Screenshots

![DESCRIPTION](LINK.png)

Logs

No response

Front-end framework

No response

Version of the front-end framework

torch-2.0.0a0+git062380d

Back-end engine

torchjit

Version of the front-end engine

No response

Other context

No response

1. Switch to tf branch.
2. Install tensorflow following this.
3. pip installl icecream dill
4. export TF_FORCE_GPU_ALLOW_GROWTH=true
5. python nnsmith/graph_gen.py --max_nodes 5 --viz --framework tensorflow --seed 384525877

Then we can see this error:

2022-08-25 13:42:21.370475: E tensorflow/stream_executor/cuda/cuda_blas.cc:197] failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
2022-08-25 13:42:21.370486: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:438 : INTERNAL: Failed initializing math mode
Traceback (most recent call last):
  File "/home/colin/code/nnsmith/nnsmith/graph_gen.py", line 1326, in <module>
    out_eager = model.run_eagerly(inputs)
  File "/home/colin/code/nnsmith/nnsmith/materialize/tensorflow/__init__.py", line 133, in run_eagerly
    return self.net(**inputs)
  File "/home/colin/miniconda3/envs/py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/colin/code/nnsmith/nnsmith/materialize/tensorflow/tfnet.py", line 91, in __call__
    out_tensors = instr.fwd_fn(
  File "/home/colin/miniconda3/envs/py39/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
tensorflow.python.framework.errors_impl.InternalError: Exception encountered when calling layer "dense" (type Dense).

Failed initializing math mode [Op:MatMul]

Call arguments received by layer "dense" (type Dense):
  • inputs=tf.Tensor(shape=(1, 1, 1, 1), dtype=float32)

Discussions about similar issues on the Internet:

1. tensorflow/tensorflow#9489 (comment)
2. Many people think this is caused by an OOM error actually, because by default TF will use nearly all of the memory, which can lead to cublas initialization error. Way to avoid using most of the memory The most simple way to disable this behavior of TF is export TF_FORCE_GPU_ALLOW_GROWTH=1 . But this will not resolve this issue.

Using the seed above 384525877 , we can get this graph:

Manually building this graph using the program below will not lead to the same error.

from typing import List, Tuple, Dict, Callable
import os
import tensorflow as tf
from tensorflow import keras
# import tensorflow.keras as keras
from keras import layers
from tqdm import tqdm
from icecream import ic

def randn_from_specs(specs: Dict[str, tf.TensorSpec]) -> Dict[str, tf.Tensor]:
    return {
        name: tf.cast(tf.random.normal(shape=spec.shape), dtype=spec.dtype)
        for name, spec in specs.items()
    }

def get_dense(units: int) -> layers.Dense:
    return layers.Dense(units=units, dtype=tf.float32, autocast=False)

class MyModule(tf.Module):
    def __init__(self) -> None:
        self.mlist: List[Callable] = []
        # self._7_linear = layers.Dense(units=2, dtype=tf.float32, autocast=False)
        # self._3_linear = layers.Dense(units=1, dtype=tf.float32, autocast=False)
        # self._5_linear = layers.Dense(units=1, dtype=tf.float32, autocast=False)
        print('init 0...')
        self.mlist.append(get_dense(2))
        print('init 1...')
        self.mlist.append(get_dense(1))
        print('init 2...')
        self.mlist.append(get_dense(1))

    @tf.function # (jit_compile=False)
    def __call__(self, *args, **kwargs) -> Dict[str, tf.Tensor]:
        key2tensor: Dict[int, tf.Tensor] = {}
        key2tensor[0] = kwargs['i0']
        key2tensor[1] = self.mlist[0](key2tensor[0])
        key2tensor[2] = self.mlist[1](key2tensor[1])
        key2tensor[3] = tf.add(key2tensor[1], key2tensor[2])
        key2tensor[4] = self.mlist[2](key2tensor[3])
        key2tensor[5] = tf.add(key2tensor[2], key2tensor[2])
        return {
            "o0": key2tensor[5],
            "o1": key2tensor[4],
        }

if __name__ == '__main__':
    net = MyModule()
    input_dict = randn_from_specs({
        "i0": tf.TensorSpec(shape=[1, 1, 1, 1], dtype=tf.float32),
    })
    ic(input_dict)
    ic(net(**input_dict))
    

I am migrating my coverage guidance update into the latest branch. Since @lazycal largely changed my graph generation implementation to support multi-input graphs, I am opening a new thread to ask a few questions about the change as a "post-review" section, though some of them might be not quite related to my merged update.

I would also recommend having a PR in the future if one made a significant change (say 500+ line new feature) to a codebase previously implemented by the other which is helpful to avoid inconsistency and bugs in our implementation. Just like what Fabian did. :-)

Hello, sorry for disturbing you.

I am currently testing a machine learning compiler that only accepts sequential models. I would like to know if there is a straightforward method to configure NNSmith in a way that it exclusively generates sequential models. (The sequntial model here means the model can be implemented by only using one torch.nn.Sequential())

Looking foward to your response!
Thanks a lot!

Description

Hi, authors.

When I tried to reproduce the experiment, I had a problem in this step: Step 2: Run NNSmith over instrumented SUT

When I wanted to run the fuzz.py below:

python nnsmith/cli/fuzz.py  fuzz.time=4h fuzz.root=${PATH_TO_REPORT} \
                            model.type=torch backend.type=pt2        \
                            filter.type="[nan,inf,dup]"              \
                            fuzz.save_test=${PATH_TO_SAVE_TESTS}

A relative import error occurred.

To be honest, I'm not quite sure if there's something wrong with the structure of project directory or my operation.

Here is the error log:

Traceback (most recent call last):
  File "nnsmith/cli/fuzz.py", line 13, in <module>
    from nnsmith.abstract.extension import activate_ext
ModuleNotFoundError: No module named 'nnsmith.abstract.extension'

Looking forward to your reply.
Thx in advance :)

Installation

Source file without edits

Reproduction

python nnsmith/cli/fuzz.py  fuzz.time=4h fuzz.root=${PATH_TO_REPORT} \
                            model.type=torch backend.type=pt2        \
                            filter.type="[nan,inf,dup]"              \
                            fuzz.save_test=${PATH_TO_SAVE_TESTS}

### Screenshots

_No response_

### Logs

_No response_

### Front-end framework

_No response_

### Version of the front-end framework

_No response_

### Back-end engine

_No response_

### Version of the front-end engine

_No response_

### Other context

_No response_

Description

I encountered the following error when using tvm.relay.frontend.onnx to load an onnx model generated by nnsmith.

Installation

pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade

Reproduction

# Paste the commands or python script for reproducing the issue.

Screenshots

![DESCRIPTION](LINK.png)

Logs

No response

Front-end framework

onnx

Version of the front-end framework

No response

Back-end engine

tvm

Version of the front-end engine

onnx-1.14.0

Other context

No response

I am getting a bit upset after using hydra for a long time. I gradually realized that it actually has more cons than its pros...

Here are the main concerns in my mind that make it hard to use for both active developers and new users:

• the argument specifying style of hydra is not compatible with that of other more popular CLI tools such as argparse -- while click allows us to make things argparse-like
• no built-in completion: writing things in a=b style just cannot let you do auto path completion...
• extended traceback... will extend the error stack substantially... making it harder to locate the real error.
• inactive -- there has been a number of valuable feature requests for hydra but it is somehow unimplemented for years...
• cannot add --help information for non-hydra arguments

Of course hydra has its own edge:

1. Keep a record of what you typed -- however I never used that... as I am just inclined to check history
2. Composible(?) -- well it is actually not that composable as I only find it usable by pumping things into a single main.yaml file -- this is also main reason why I found click attractive

Proposing click:

1. Really composable as we can separate required commands and inject that into different applications
2. Everything in Python... We don't need another .yaml which requires lots of extra configurations in package building
3. Actively maintained -- can see the commit history
4. It allows a lot more command-line features such as prompting.

Also cc'ing active contributors for any comments: @Co1lin @Kristoff-starling

I am planning to take this into the v0.2.0 release when we fully merged NeuRI into our main repo.

Hi, apologies for this issue, I am working with NNSmith (and Neuri) and have several generated bug reports from fuzzing. However, when looking at bug reports developed as a result of NNSmith like this one here there's a minimal code sample that can be used to trigger and reproduce the bug encountered. However, I can't seem to figure out how to generate such examples utilizing NNSmith. Is there an additional utility, or a technique to do so? I can see the model.pth and pkl files, but would like a nudge or some guidance in how to utilize them to create a reproducible code sample if that makes sense?

Sorry again for the call for help.

Description

ONNXModelCPU_tvm_0.9.0_cpu.yaml file was not generated properly, resulting in the following error message
opset: {}

�[31mERROR  �[0m �[35mfuzz  �[0m - Traceback (most recent call last):
  File "/home/vincy/nnsmith/nnsmith/cli/fuzz.py", line 224, in run
    testcase = self.make_testcase(seed)
  File "/home/vincy/nnsmith/nnsmith/cli/fuzz.py", line 179, in make_testcase
    gen = model_gen(
  File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 796, in model_gen
    gen = SymbolicGen(opset, seed, symbolic_init=symbolic_init, **kwargs)
  File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 467, in __init__
    super().__init__(opset, seed, **kwargs)
  File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 42, in __init__
    assert len(opset) > 0, "opset must not be empty"

tvm version: 0.9.0
nnsmith: 0.2.0.dev16

Installation

pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade

Reproduction

# Paste the commands or python script for reproducing the issue.
command: `python nnsmith/cli/fuzz.py  fuzz.time=1s fuzz.root=${PATH_TO_REPORT}    model.type=onnx backend.type=tvm      filter.type="[nan,inf,dup]"     fuzz.save_test=${PATH_TO_SAVE_TESTS} `

or 

command: `pytest -s tests/tvm/ `

Screenshots

![DESCRIPTION](LINK.png)

Logs

�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((float16, float16)) => (float16,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((float32, float32)) => (float32,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((float64, float64)) => (float64,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((int8, int8)) => (int8,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((int16, int16)) => (int16,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((int32, int32)) => (int32,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((int64, int64)) => (int64,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================

�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((uint8, uint8)) => (uint8,)�[0m
�[32mINFO   �[0m �[35mfuzz  �[0m - Test success info supressed -- only showing logs for failed tests�[0m
�[32mINFO   �[0m �[35mfuzz  �[0m - Saving all intermediate testcases to PATH_TO_SAVE_TESTS�[0m
�[33mWARNING�[0m �[35mcore  �[0m - Report folder already exists. Press [Y/N] to continue or exit...�[0m
�[31mERROR  �[0m �[35mfuzz  �[0m - `make_testcase` failed with seed 2091741040. It can be NNSmith or Generator (onnx) bug.�[0m
�[31mERROR  �[0m �[35mfuzz  �[0m - Traceback (most recent call last):
  File "/home/vincy/nnsmith/nnsmith/cli/fuzz.py", line 224, in run
    testcase = self.make_testcase(seed)
  File "/home/vincy/nnsmith/nnsmith/cli/fuzz.py", line 179, in make_testcase
    gen = model_gen(
  File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 796, in model_gen
    gen = SymbolicGen(opset, seed, symbolic_init=symbolic_init, **kwargs)
  File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 467, in __init__
    super().__init__(opset, seed, **kwargs)
  File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 42, in __init__
    assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty

Front-end framework

onnx

Version of the front-end framework

onnx-1.14.0 onnxruntime-1.15.1

Back-end engine

tvm

Version of the front-end engine

tvm-0.9.0

Other context

The onnxruntime back-end engine is working fine

Motivation

• Sanitize and push TF coverage evaluation scripts.
• Add tutorial in experiments/README.md.

Basic Example

None

Drawbacks

None

Reference Issues

No response

Other context

No response

Contribution

yes

Recently I am trying to make our generator stable and bug-free. The only left sanity error that comes to my eye is:

This is obviously caused by the z3 timeout... Since we use sub-processes to control the timeout, why do we still need:

https://github.com/ise-uiuc/nnsmith/blob/main/nnsmith/graph_gen.py#L257

I have been knowing that z3.set_params can be buggy from some z3 issues. @lazycal

I created an nnsmith environment using Conda and installed the required libraries according to the installation tutorial "python3- m pip install" nnsmith [torch, onnx, tvm, onnxruntime] "-- upgrade" in cli.md. I can generate the onnx model normally, but I encountered an issue when trying to debug this model locally.
When I used the command "nnsmith.model_exec model.type=onnx backend.type=onnxruntime model.path=nnsmith_output/model.onnx", an AttributeError has occurred
module 'onnx' has no attribute '_serialize'

This seems to be an issue with the onnx version, so I checked the onnx version and found that the current onnx version is 1.15.0, then I reinstalled version 1.14.0 of onnx, then I run the same command again, the onnx model will compile and run smoothly.

Hi @lazycal , could you kindly improve our reporter by also logging inputs if there is an inconsistency error? I found it hard to debug w/o this info. Our current impl only record <model, err message>.

Currently assertions are all caught and silently ignored. I think it might be better to have a specific new type of exceptions for this purpose just like RequiredDimNotFound.

Is there any approach/setting to control the search space for value of some specific OP attributes?
For example, any approach to control kernel_size of Conv2d to be in range [1, 15]?

I tried to generate a model with conv2d and relu only, with max_nodes set to 100.
If the timeout_ms was set to 10000 (relativly small number) ms, the solver seems to find nothing, then model_gen failed with following errors.
If timeout_ms set to 3000000 (a large number) ms, the entire model_gen process takes a very long time to generate a single model

  File "/venv/test/bin/nnsmith.model_gen", line 8, in <module>
    sys.exit(main())
  File "/venv/test/lib/python3.8/site-packages/hydra/main.py", line 90, in decorated_main
    _run_hydra(
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
    _run_app(
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
    run_and_report(
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 222, in run_and_report
    raise ex
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 219, in run_and_report
    return func()
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
    lambda: hydra.run(
  File "/venv/test/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
    _ = ret.return_value
  File "/venv/test/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
    raise self._return_value
  File "/venv/test/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
    ret.return_value = task_function(task_cfg)
  File "/venv/test/lib/python3.8/site-packages/nnsmith/cli/model_gen.py", line 57, in main
    f"{len(gen.last_solution)} symbols and {len(gen.solver.assertions())} constraints."
TypeError: object of type 'NoneType' has no len()

The main features are refactored in #33 with a primary focus on extensibility, code quality, and maintainability (e.g., ci & PyPI). This thread tracks some follow-up (or P1~2) enhancements and features that we are going to work on in the next few weeks.

0.0.0

https://pypi.org/project/nnsmith/0.0.0/

• Refactoring verbose printing with module-level loggers. #44
• Auto versioning #40
• Dtype testing for model exportation; #40
• Make pygraphviz an optional dependency as it requires system-level constraint #43

0.0.1

https://pypi.org/project/nnsmith/0.0.1/

• Stabilize nnsmith.fuzz via smoke testing in CI; #50
• Fix core.Gemm; #45
• Double-check and stabilize backend-oriented spec re-writing; #52
• generation exception safety. #52
• Well-form backend dispatching;
• GitHub HEAD install (tested);
• Fix XLA. #50
• Patchable bug filter. #53
• Crash safety in execution; #54

TBD

• Cleaning stale branches after a 5-month window size;
• nnsmith.fuzz TUI.
• Implement exclude set logics;
• Replace pygraphviz with graphviz to release the dependency constraint by pygraphviz?
• Simplifying nnsmith arith by extending z3.ArithRef. Z3Prover/z3#6309
• Well-form core operators and dialects; -- core operators should not change that frequently...
• Simplifying and bringing back attribute binning;
• CLI-level documentation;
• Tutorials on CLI, design, and extension.

Motivation

Does nnsmith support dynamic graphs? Just like RNN or Deformable Convolution?
If not, I think this will be a direction worth further investigation :)

Basic Example

RNN or Deformable Convolution

Drawbacks

Dynamic Graph?

Reference Issues

No response

Other context

No response

Contribution

yes

Tracking the issue of too many false positives, and recording the current status (probably won't touch for a while).

One proposal is being worked on in https://github.com/ise-uiuc/nnsmith/tree/Avoid-FP. It does the following:

• Constrain inputs for truncating operators to avoid cut-off edge. E.g., Floor(x), |x-round(x)| should not be too small (>1e-3). Affected ops: Floor, Ceil, Round, Cast(float->int)
• Constrain inputs for Sin/Cos to be small (in [-4pi, 4pi])

The result as of e05b1f5: on ORT CPU, the proportion of incorrect-result reports (hence false positives) has reduced from 24(Bug)/11504(Iter)=0.2% to 4/13470=0.03%, with a sacrifice on the input gen success rate that drops from 91.6% to 82.2%. One likely reason is because of graphs containing integers. These won't be differentiable, but now are more likely to violate the constraints.

Like in TensorRT's argmin requires input tensor's rank <= 4 that nnsmith should to test single op's dim availability as well. It can be orthogonal to dtype_test.

Hi, Jiawei @ganler . I'm sorry to bother you due to your busy schedule. I have some difficulty in collecting branch coverage of TVM.

I followed your instructions for Torch Cov and it worked.

I found that Tzer provides an instruction of TVM Memcov. I followed this, but unfortunately, it didn't work when I used the experiments scripts NNSmith provides. I couldn't get the profraw files.

I think this is due to the fact that the patch file is not specifically designed for branch cov but for edge cov. Do you still have a patch for branch cov for TVM? (or maybe some tutorials to help me make the patch)

Thx in advance :D

Sorry to bother you again, Jiawei :(

I felt confused after the replication of NNSmith.

In my experiment, NNSmith generated 7686, 9730, and 23812 testcases in 4 hours for torch, ort, and TVM respectively, which exits a big difference from neuri and the report in the paper. (In my experiment, Neuri can generate 125300 testcases for torch)

For NNSmith and Neuri, I use the docker file you provided. But I reproduced the experiment on the same linux server.

I can make sure I'm using the same hardware environment so I wonder if it's the version of NNSmith I'm using.

I found you provided two experimental instructions. One in nnsmith/experiments/README.md, the other one in docker file.

I want to know which version of the code can achieve the SOTA performance of NNSmith？

In addition, if I want to fully understand NNSmith on code aspect. Which version of the source code should I read?

Thx in advance, looking forward to your reply :D

https://colab.research.google.com/drive/1cbLiDvDLDY_MQfH2flNYbCP9aMNlFqLX#scrollTo=X9ZqNaW6o0dE

It is a bit strange that nnsmith/**/*.yaml won't be packaged if we do:

pip install --user "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade

in Colab directly.

While doing that with an extra --user (as well as extending corresponding PATH and PYTHONPATH) can resolve it, it is still annoying. It could be a bug in the packager.

Though I set the options.package_data which should work at all scenarios...

Check here for how to detect numerical invalidity:

Though we can keep some old code for debugging older bug reports, we should start using this kind of form to make things lightweight (so that we don't have to write repeated numeric checkers for each framework).

There are a few engineering efforts that should be paid to adapt it completely. I will update ASAP.