ise-uiuc / nnsmith Goto Github PK
View Code? Open in Web Editor NEWAutomatic DNN generation for fuzzing and more
Home Page: https://pypi.org/project/nnsmith/
License: Apache License 2.0
Automatic DNN generation for fuzzing and more
Home Page: https://pypi.org/project/nnsmith/
License: Apache License 2.0
Sorry to bother you again, Jiawei :(
I felt confused after the replication of NNSmith.
In my experiment, NNSmith generated 7686
, 9730
, and 23812
testcases in 4 hours
for torch
, ort
, and TVM
respectively, which exits a big difference from neuri and the report in the paper. (In my experiment, Neuri can generate 125300
testcases for torch)
For NNSmith and Neuri, I use the docker file you provided. But I reproduced the experiment on the same linux server.
I can make sure I'm using the same hardware environment so I wonder if it's the version of NNSmith I'm using.
I found you provided two experimental instructions. One in nnsmith/experiments/README.md, the other one in docker file.
I want to know which version of the code can achieve the SOTA performance of NNSmith?
In addition, if I want to fully understand NNSmith on code aspect. Which version of the source code should I read?
Thx in advance, looking forward to your reply :D
experiments/README.md
.https://colab.research.google.com/drive/1cbLiDvDLDY_MQfH2flNYbCP9aMNlFqLX#scrollTo=X9ZqNaW6o0dE
It is a bit strange that nnsmith/**/*.yaml
won't be packaged if we do:
pip install --user "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade
in Colab directly.
While doing that with an extra --user
(as well as extending corresponding PATH
and PYTHONPATH
) can resolve it, it is still annoying. It could be a bug in the packager.
Though I set the options.package_data
which should work at all scenarios...
Line 40 in bdc2747
Since both latest PyTorch and TensorFlow are deprecating Python 3.7. Maybe we should also embrace Python 3.8+ to make use of more features.
There are a list of changes where we can make it mandatory:
python_version>=3.8
cc: @Co1lin
The main features are refactored in #33 with a primary focus on extensibility, code quality, and maintainability (e.g., ci & PyPI). This thread tracks some follow-up (or P1~2) enhancements and features that we are going to work on in the next few weeks.
https://pypi.org/project/nnsmith/0.0.0/
pygraphviz
an optional dependency as it requires system-level constraint #43https://pypi.org/project/nnsmith/0.0.1/
nnsmith.fuzz
via smoke testing in CI; #50nnsmith.fuzz
TUI.pygraphviz
with graphviz
to release the dependency constraint by pygraphviz?z3.ArithRef
. Z3Prover/z3#6309Hi, apologies for this issue, I am working with NNSmith (and Neuri) and have several generated bug reports from fuzzing. However, when looking at bug reports developed as a result of NNSmith like this one here there's a minimal code sample that can be used to trigger and reproduce the bug encountered. However, I can't seem to figure out how to generate such examples utilizing NNSmith. Is there an additional utility, or a technique to do so? I can see the model.pth and pkl files, but would like a nudge or some guidance in how to utilize them to create a reproducible code sample if that makes sense?
Sorry again for the call for help.
I used nnsmith to generate a set of models and extracted features from them. In a model, I found a tensor with a very strange shape, as follows:
I don't quite understand what this li64, 40i64 is.
pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade
# Paste the commands or python script for reproducing the issue.
![DESCRIPTION](LINK.png)
No response
onnx
No response
tvm
No response
No response
tf
branch.pip installl icecream dill
export TF_FORCE_GPU_ALLOW_GROWTH=true
python nnsmith/graph_gen.py --max_nodes 5 --viz --framework tensorflow --seed 384525877
Then we can see this error:
2022-08-25 13:42:21.370475: E tensorflow/stream_executor/cuda/cuda_blas.cc:197] failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
2022-08-25 13:42:21.370486: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:438 : INTERNAL: Failed initializing math mode
Traceback (most recent call last):
File "/home/colin/code/nnsmith/nnsmith/graph_gen.py", line 1326, in <module>
out_eager = model.run_eagerly(inputs)
File "/home/colin/code/nnsmith/nnsmith/materialize/tensorflow/__init__.py", line 133, in run_eagerly
return self.net(**inputs)
File "/home/colin/miniconda3/envs/py39/lib/python3.9/site-packages/tensorflow/python/util/traceback_utils.py", line 153, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/home/colin/code/nnsmith/nnsmith/materialize/tensorflow/tfnet.py", line 91, in __call__
out_tensors = instr.fwd_fn(
File "/home/colin/miniconda3/envs/py39/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
raise e.with_traceback(filtered_tb) from None
tensorflow.python.framework.errors_impl.InternalError: Exception encountered when calling layer "dense" (type Dense).
Failed initializing math mode [Op:MatMul]
Call arguments received by layer "dense" (type Dense):
• inputs=tf.Tensor(shape=(1, 1, 1, 1), dtype=float32)
Discussions about similar issues on the Internet:
export TF_FORCE_GPU_ALLOW_GROWTH=1
. But this will not resolve this issue.Using the seed above 384525877
, we can get this graph:
Manually building this graph using the program below will not lead to the same error.
from typing import List, Tuple, Dict, Callable
import os
import tensorflow as tf
from tensorflow import keras
# import tensorflow.keras as keras
from keras import layers
from tqdm import tqdm
from icecream import ic
def randn_from_specs(specs: Dict[str, tf.TensorSpec]) -> Dict[str, tf.Tensor]:
return {
name: tf.cast(tf.random.normal(shape=spec.shape), dtype=spec.dtype)
for name, spec in specs.items()
}
def get_dense(units: int) -> layers.Dense:
return layers.Dense(units=units, dtype=tf.float32, autocast=False)
class MyModule(tf.Module):
def __init__(self) -> None:
self.mlist: List[Callable] = []
# self._7_linear = layers.Dense(units=2, dtype=tf.float32, autocast=False)
# self._3_linear = layers.Dense(units=1, dtype=tf.float32, autocast=False)
# self._5_linear = layers.Dense(units=1, dtype=tf.float32, autocast=False)
print('init 0...')
self.mlist.append(get_dense(2))
print('init 1...')
self.mlist.append(get_dense(1))
print('init 2...')
self.mlist.append(get_dense(1))
@tf.function # (jit_compile=False)
def __call__(self, *args, **kwargs) -> Dict[str, tf.Tensor]:
key2tensor: Dict[int, tf.Tensor] = {}
key2tensor[0] = kwargs['i0']
key2tensor[1] = self.mlist[0](key2tensor[0])
key2tensor[2] = self.mlist[1](key2tensor[1])
key2tensor[3] = tf.add(key2tensor[1], key2tensor[2])
key2tensor[4] = self.mlist[2](key2tensor[3])
key2tensor[5] = tf.add(key2tensor[2], key2tensor[2])
return {
"o0": key2tensor[5],
"o1": key2tensor[4],
}
if __name__ == '__main__':
net = MyModule()
input_dict = randn_from_specs({
"i0": tf.TensorSpec(shape=[1, 1, 1, 1], dtype=tf.float32),
})
ic(input_dict)
ic(net(**input_dict))
Seems to me the multi-input head and the constant nodes get unconnected quite often (almost always).
On the one hand, it helps us get new bugs on frameworks that poorly support this unreal models;
However, I feel that it produces a specific kind of bug in the very beginning all the time that might mask other bugs...
For example, 80% of TVM bugs look like this which shuts down the compilation in the very beginning.
That said, our current implementation of multi-input & constant node insertion might bias the model generation because we did not apply any strategies to consume these unconnected nodes. esp. for constant nodes.
I would recommend either reducing the probability of such kinds of nodes since they seldom connect prior nodes; or generating constant nodes on non-unary operators. For example, Add(Var, Cons) etc.
Before open-sourcing the nnsmith project, I want to simplify and standardize a bit the development and user accessibility. Below is a tracking list of TODOs for @ganler to make the repository a better open-source one in the next few months.
pre-commit
;black
formatting (within pre-commit);nnsmith/abstract
);The thing is that the coverage tool is extended to be used by multiple libraries.
When fuzzing TVM, we should avoid importing onnxruntime
as it will mess with the coverage...
However, the NaN checker relies on onnxruntime
which makes things much harder...
So @lazycal do you think there is other options to replace onnxruntime
for NaN checking? Anyhow, I would say not using tvm
and onnxruntime
as "utility" because it will mess with the coverage. Maybe we can consider PyTorch (JIT)?
experiments/README.md
.None
None
No response
No response
yes
I created an nnsmith environment using Conda and installed the required libraries according to the installation tutorial "python3- m pip install" nnsmith [torch, onnx, tvm, onnxruntime] "-- upgrade" in cli.md. I can generate the onnx model normally, but I encountered an issue when trying to debug this model locally.
When I used the command "nnsmith.model_exec model.type=onnx backend.type=onnxruntime model.path=nnsmith_output/model.onnx", an AttributeError has occurred
module 'onnx' has no attribute '_serialize'
This seems to be an issue with the onnx version, so I checked the onnx version and found that the current onnx version is 1.15.0, then I reinstalled version 1.14.0 of onnx, then I run the same command again, the onnx model will compile and run smoothly.
Hi, Jiawei @ganler . I'm sorry to bother you due to your busy schedule. I have some difficulty in collecting branch coverage of TVM
.
I followed your instructions for Torch Cov and it worked.
I found that Tzer provides an instruction of TVM Memcov. I followed this, but unfortunately, it didn't work when I used the experiments scripts NNSmith provides. I couldn't get the profraw files.
I think this is due to the fact that the patch file is not specifically designed for branch cov
but for edge cov
. Do you still have a patch for branch cov
for TVM? (or maybe some tutorials to help me make the patch)
Thx in advance :D
Recently I am trying to make our generator stable and bug-free. The only left sanity error that comes to my eye is:
This is obviously caused by the z3 timeout... Since we use sub-processes to control the timeout, why do we still need:
https://github.com/ise-uiuc/nnsmith/blob/main/nnsmith/graph_gen.py#L257
I have been knowing that z3.set_params
can be buggy from some z3 issues. @lazycal
Currently assertions are all caught and silently ignored. I think it might be better to have a specific new type of exceptions for this purpose just like RequiredDimNotFound
.
Lines 243 to 244 in 11127cc
Hi. I am currently learning to use nnsmith. I would like to know if I can customize the generated graph as a graph with only a single input variable and a single output variable.
Is there any way to quickly implement this?
Hi, I'm studying the implementation of NNSmith. Is there any tutorial telling developers how to add a new operator for GIR? I want to add a Split
operator to the GIR, but still try to find a way. I think a tutorial would be very helpful.
As far as I understand, one should add a new class for the operator in nnsmith/abstract/op.py
, and add some functions in nnsmith/backends/*
. Do I understand it correctly?
None.
None.
No response
No response
I can partially help.
I encountered the following error when using tvm.relay.frontend.onnx to load an onnx model generated by nnsmith.
pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade
# Paste the commands or python script for reproducing the issue.
![DESCRIPTION](LINK.png)
No response
onnx
No response
tvm
onnx-1.14.0
No response
Hi, I'm seeking a way to generate runable python code from bug report directory, so that I can reproduce bugs without NNSmith's code. I found that there is a class named Render
that may help me to do this. However, It does not work well when I want to emit a TorchModel
. It seems that fx.symbolic_trace
cannot generate the model declaration in TorchMdel.emit_def()
. Is the Render
supposed to work but I did not use it right? Or they are just some deprecated code?
A lot of inconsistency bugs show something similar below:
/home/ganler/miniconda3/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:55: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
warnings.warn("Specified provider '{}' is not in available provider names."
Process Process-12:
Traceback (most recent call last):
File "/home/ganler/miniconda3/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/ganler/miniconda3/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "nnsmith/fuzz.py", line 310, in difftest
assert_allclose(
File "/home/ganler/Documents/nnsmith/nnsmith/difftest.py", line 22, in assert_allclose
raise IncorrectResult(
nnsmith.error.IncorrectResult: torch v.s. ort-opt have different output tensor names: {'o1', 'o2', 'o3', 'o0'} ~ {'73', 'o2', 'o3', 'o0'}
'73' is not an expected output name.
Excuse me. I encountered a series of errors while converting the onnx model generated by nnsmith to tvm.relay. I would like to ask if this is a bug caused by an internal error in TVM or an error in the model generated by NNSmith.
The command I am using is "tvm.relay.frontend.onxx.from_onnx(onnx_model)".
I am migrating my coverage guidance update into the latest branch. Since @lazycal largely changed my graph generation implementation to support multi-input graphs, I am opening a new thread to ask a few questions about the change as a "post-review" section, though some of them might be not quite related to my merged update.
I would also recommend having a PR in the future if one made a significant change (say 500+ line new feature) to a codebase previously implemented by the other which is helpful to avoid inconsistency and bugs in our implementation. Just like what Fabian did. :-)
How do I know if this model has caused an error after executing the "nnsmith.model_exec model.type=onnx backend.type=onnxruntime model.path=nnsmith_output/model.onnx", " nnsmith.model_exec model.type=onnx
backend.type=onnxruntime
model.path=nnsmith_output/model.onnx
cmp.with='{type:tvm, optmax:true, target:cpu}'"?
tvm-optimized
v.s. tvm-debug
;
ort-optimized
v.s. ort-debug
;
Line 211 in db48367
This is not quite effective as we are going to compile the same model twice. The compilation is kinda expensive so...
As we use PyTorch to generate models, we can simply use PyTorch as the oracle.
So basically when doing input generation, we want to verify that outputs do not have (or have a lot of) NaN values. The <input, output> pair can be used as data samples to test the target framework. We can simply use IPC to transfer the data in the form of Tuple[Dict[str, np.array], Dict[str, np.array]]
or simply Tuple[List[np.array], List[np.array]]
.
Would like any volunteers to handle this as I am working on other features. Suggestions are welcomed.
Does nnsmith support dynamic graphs? Just like RNN or Deformable Convolution?
If not, I think this will be a direction worth further investigation :)
RNN or Deformable Convolution
Dynamic Graph?
No response
No response
yes
nnsmith/nnsmith/graph_input_gen.py
Line 93 in 2a487d5
It is not a good workaround to use check_call
here which:
We can discuss better solutions in this thread.
Previously my implementation makes the generation phase in-process which works well. It seems there are some issues regarding timeout and generation failure.
check_call
to allow stateful fuzzing;We can discuss more in this thread.
Check here for how to detect numerical invalidity:
Line 914 in 6812455
Though we can keep some old code for debugging older bug reports, we should start using this kind of form to make things lightweight (so that we don't have to write repeated numeric checkers for each framework).
There are a few engineering efforts that should be paid to adapt it completely. I will update ASAP.
Like in TensorRT's argmin requires input tensor's rank <= 4 that nnsmith should to test single op's dim availability as well. It can be orthogonal to dtype_test
.
Motivation: Test a few more operators.
We should focus on operators which are often common and optimizable (of course supported by PyTorch and can be directly mapped to ONNX). For example, we don't want to add more trivial element-wise layers (it's hard to implement it wrong, right?).
torch.tril
torch.triu
torch.nn.ConvTranspose1d
torch.nn.ConvTranspose2d
torch.nn.ConvTranspose3d
torch.nn.Conv1d
torch.nn.Conv3d
torch.cumsum
torch.nn.functional.one_hot
Hello, sorry for disturbing you.
I am currently testing a machine learning compiler that only accepts sequential models. I would like to know if there is a straightforward method to configure NNSmith in a way that it exclusively generates sequential models. (The sequntial model here means the model can be implemented by only using one torch.nn.Sequential())
Looking foward to your response!
Thanks a lot!
I've install nnsmith by using pip3.8 install "nnsmith[torch,onnx]" --upgrade
and install successfully. But when I try to excute
nnsmith.fuzz fuzz.time=86400s model.type=onnx backend.type=tvm fuzz.root=fuzz_report_1 debug.viz=true mgen.max_nodes=100
I met fuzz error. The part of the fuzz.log is as follow:
[2023-02-14 10:14:45,357][dtest][INFO] - Loading topset from /root/.cache/nnsmith/ONNXModel_tvm_llvm -keys=cpu .yaml.
[2023-02-14 10:14:45,357][dtest][INFO] - To regenerate the topset, delete the cache file and restart.
[2023-02-14 10:14:45,358][dtest][INFO] - rm /root/.cache/nnsmith/ONNXModel_tvm_llvm -keys=cpu .yaml
[2023-02-14 10:14:45,359][fuzz][INFO] - Test success info supressed -- only showing logs for failed tests
[2023-02-14 10:14:45,360][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,360][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,361][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3943882080 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,361][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,361][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,361][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=627982728 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,362][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,362][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,362][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=2470370689 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,362][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,362][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,363][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=2671980766 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,363][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,363][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,363][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3215383915 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,364][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,364][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,364][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=1079210586 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,364][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,364][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,365][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=1807477749 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,365][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,365][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,365][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3642741048 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,366][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,366][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,366][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3396546204 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,366][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,367][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,367][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=395719398 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,367][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,367][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,367][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3371485272 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,368][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,368][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,368][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=3200508025 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,368][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,369][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
[2023-02-14 10:14:45,369][fuzz][ERROR] - repro with: nnsmith.model_gen mgen.seed=759925757 mgen.max_nodes=100 model.type=onnx backend.target=cpu
[2023-02-14 10:14:45,369][fuzz][ERROR] - `make_testcase` failed. It could be a NNSmith bug or Generator bug (e.g., onnx).
[2023-02-14 10:14:45,369][fuzz][ERROR] - Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 178, in run
testcase = self.make_testcase(seed)
File "/usr/local/lib/python3.8/site-packages/nnsmith/cli/fuzz.py", line 136, in make_testcase
gen = random_model_gen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 1069, in random_model_gen
gen = PureSymbolGen(
File "/usr/local/lib/python3.8/site-packages/nnsmith/graph_gen.py", line 96, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
Could anyone tell me what should I do next?
Hi @lazycal , could you kindly improve our reporter by also logging inputs if there is an inconsistency error? I found it hard to debug w/o this info. Our current impl only record <model, err message>.
There has been a rich set of improvements since v0.0.1
(the preview version). a8307f2...8cee6f6 (until 03/08/2022). Here, some noticeable changes are listed, as part of the release for v0.1.0
. Additional non-documentation improvements will also be listed and targeted for v0.1.0
.
Following standard open-source convention, a release candidate will be pre-released and later "sealed" until no issues are found in two weeks.
iree
(#74)torchjit
. @Co1linHi
I ran into this error, trying to run basic instruction written in Quick Start section
Any idea what might cause this error?
Traceback (most recent call last):
File "/venv/test/bin/nnsmith.model_gen", line 5, in <module>
from nnsmith.cli.model_gen import main
File "/venv/test/lib/python3.8/site-packages/nnsmith/cli/model_gen.py", line 16, in <module>
@hydra.main(version_base=None, config_path="../config", config_name="main")
TypeError: main() got an unexpected keyword argument 'version_base'
The command I used:
nnsmith.model_gen model.type=onnx debug.viz=true
Here is my env/lib settings/version:
python: 3.8.7
nnsmith: 0.0.1
z3-solver: 4.11.2
hydra-core: 1.1.2
hydra_colorlog: 1.0.0
multipledispatch: 0.6.0
appdirs: 1.4.4
Those logs are so frequent and will mess up with rich
console printer.
Submitted an issue to PyTorch and let's track it here.
Hi there, when I invoke nnsmith.model_gen ... mgen.save=gen/
, it warns me the report folder already exists and let me choose, even if such directory doesn't exist in $pwd
.
If my guess is right, now it will warns me the reports folder exists if nnsmith has seen this path before.
It will facilitate the integration of nnsmith for automation, like, I was not warned to choose Y/N every single run. Is there a way to disable this reminder? :)
Tracking the issue of too many false positives, and recording the current status (probably won't touch for a while).
One proposal is being worked on in https://github.com/ise-uiuc/nnsmith/tree/Avoid-FP. It does the following:
The result as of e05b1f5: on ORT CPU, the proportion of incorrect-result reports (hence false positives) has reduced from 24(Bug)/11504(Iter)=0.2% to 4/13470=0.03%, with a sacrifice on the input gen success rate that drops from 91.6% to 82.2%. One likely reason is because of graphs containing integers. These won't be differentiable, but now are more likely to violate the constraints.
I wanted to excute nnsmith.model_gen mgen.max_nodes=5 mgen.dtype_choices="[f32]" model.type=onnx debug.viz=false mgen.save=$path
. But it reurned :
Could not override 'mgen.dtype_choices'.
To append to your config use +mgen.dtype_choices=[f32]
Key 'dtype_choices' is not in struct
full_key: mgen.dtype_choices
object_type=dict
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.
When I excuted without mgen.dtype_choices="[f32]"
, it can work correctly. Could anyone tell me what should I do?
pip install "nnsmith[torch,onnx]" --upgrade
# Paste the commands or python script for reproducing the issue.
![DESCRIPTION](LINK.png)
No response
No response
No response
No response
No response
No response
Is there any approach/setting to control the search space for value of some specific OP attributes?
For example, any approach to control kernel_size of Conv2d to be in range [1, 15]?
I tried to generate a model with conv2d and relu only, with max_nodes set to 100.
If the timeout_ms was set to 10000 (relativly small number) ms, the solver seems to find nothing, then model_gen failed with following errors.
If timeout_ms set to 3000000 (a large number) ms, the entire model_gen process takes a very long time to generate a single model
File "/venv/test/bin/nnsmith.model_gen", line 8, in <module>
sys.exit(main())
File "/venv/test/lib/python3.8/site-packages/hydra/main.py", line 90, in decorated_main
_run_hydra(
File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 394, in _run_hydra
_run_app(
File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 457, in _run_app
run_and_report(
File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 222, in run_and_report
raise ex
File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 219, in run_and_report
return func()
File "/venv/test/lib/python3.8/site-packages/hydra/_internal/utils.py", line 458, in <lambda>
lambda: hydra.run(
File "/venv/test/lib/python3.8/site-packages/hydra/_internal/hydra.py", line 132, in run
_ = ret.return_value
File "/venv/test/lib/python3.8/site-packages/hydra/core/utils.py", line 260, in return_value
raise self._return_value
File "/venv/test/lib/python3.8/site-packages/hydra/core/utils.py", line 186, in run_job
ret.return_value = task_function(task_cfg)
File "/venv/test/lib/python3.8/site-packages/nnsmith/cli/model_gen.py", line 57, in main
f"{len(gen.last_solution)} symbols and {len(gen.solver.assertions())} constraints."
TypeError: object of type 'NoneType' has no len()
I am getting a bit upset after using hydra
for a long time. I gradually realized that it actually has more cons than its pros...
Here are the main concerns in my mind that make it hard to use for both active developers and new users:
argparse
-- while click
allows us to make things argparse
-likea=b
style just cannot let you do auto path completion...hydra
but it is somehow unimplemented for years...--help
information for non-hydra argumentsOf course hydra
has its own edge:
history
main.yaml
file -- this is also main reason why I found click
attractiveProposing click
:
.yaml
which requires lots of extra configurations in package buildingAlso cc'ing active contributors for any comments: @Co1lin @Kristoff-starling
I am planning to take this into the v0.2.0
release when we fully merged NeuRI into our main repo.
ONNXModelCPU_tvm_0.9.0_cpu.yaml
file was not generated properly, resulting in the following error message
opset: {}
�[31mERROR �[0m �[35mfuzz �[0m - Traceback (most recent call last):
File "/home/vincy/nnsmith/nnsmith/cli/fuzz.py", line 224, in run
testcase = self.make_testcase(seed)
File "/home/vincy/nnsmith/nnsmith/cli/fuzz.py", line 179, in make_testcase
gen = model_gen(
File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 796, in model_gen
gen = SymbolicGen(opset, seed, symbolic_init=symbolic_init, **kwargs)
File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 467, in __init__
super().__init__(opset, seed, **kwargs)
File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 42, in __init__
assert len(opset) > 0, "opset must not be empty"
tvm version: 0.9.0
nnsmith: 0.2.0.dev16
pip install "git+https://github.com/ise-uiuc/nnsmith@main#egg=nnsmith[torch,onnx]" --upgrade
# Paste the commands or python script for reproducing the issue.
command: `python nnsmith/cli/fuzz.py fuzz.time=1s fuzz.root=${PATH_TO_REPORT} model.type=onnx backend.type=tvm filter.type="[nan,inf,dup]" fuzz.save_test=${PATH_TO_SAVE_TESTS} `
or
command: `pytest -s tests/tvm/ `
![DESCRIPTION](LINK.png)
�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((float16, float16)) => (float16,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((float32, float32)) => (float32,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((float64, float64)) => (float64,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((int8, int8)) => (int8,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((int16, int16)) => (int16,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((int32, int32)) => (int32,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((int64, int64)) => (int64,)�[0m
============= Diagnostic Run torch.onnx.export version 2.0.1+cu117 =============
verbose: False, log level: Level.ERROR
======================= 0 NONE 0 NOTE 0 WARNING 0 ERROR ========================
�[33mWARNING�[0m �[35mdtest �[0m - =====> [Failure] at torch.PTMatMul((uint8, uint8)) => (uint8,)�[0m
�[32mINFO �[0m �[35mfuzz �[0m - Test success info supressed -- only showing logs for failed tests�[0m
�[32mINFO �[0m �[35mfuzz �[0m - Saving all intermediate testcases to PATH_TO_SAVE_TESTS�[0m
�[33mWARNING�[0m �[35mcore �[0m - Report folder already exists. Press [Y/N] to continue or exit...�[0m
�[31mERROR �[0m �[35mfuzz �[0m - `make_testcase` failed with seed 2091741040. It can be NNSmith or Generator (onnx) bug.�[0m
�[31mERROR �[0m �[35mfuzz �[0m - Traceback (most recent call last):
File "/home/vincy/nnsmith/nnsmith/cli/fuzz.py", line 224, in run
testcase = self.make_testcase(seed)
File "/home/vincy/nnsmith/nnsmith/cli/fuzz.py", line 179, in make_testcase
gen = model_gen(
File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 796, in model_gen
gen = SymbolicGen(opset, seed, symbolic_init=symbolic_init, **kwargs)
File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 467, in __init__
super().__init__(opset, seed, **kwargs)
File "/home/vincy/.local/lib/python3.10/site-packages/nnsmith/graph_gen.py", line 42, in __init__
assert len(opset) > 0, "opset must not be empty"
AssertionError: opset must not be empty
onnx
onnx-1.14.0 onnxruntime-1.15.1
tvm
tvm-0.9.0
The onnxruntime back-end engine is working fine
Hi, authors.
When I tried to reproduce the experiment, I had a problem in this step: Step 2: Run NNSmith over instrumented SUT
When I wanted to run the fuzz.py below:
python nnsmith/cli/fuzz.py fuzz.time=4h fuzz.root=${PATH_TO_REPORT} \
model.type=torch backend.type=pt2 \
filter.type="[nan,inf,dup]" \
fuzz.save_test=${PATH_TO_SAVE_TESTS}
A relative import error occurred.
To be honest, I'm not quite sure if there's something wrong with the structure of project directory or my operation.
Here is the error log:
Traceback (most recent call last):
File "nnsmith/cli/fuzz.py", line 13, in <module>
from nnsmith.abstract.extension import activate_ext
ModuleNotFoundError: No module named 'nnsmith.abstract.extension'
Looking forward to your reply.
Thx in advance :)
Source file without edits
python nnsmith/cli/fuzz.py fuzz.time=4h fuzz.root=${PATH_TO_REPORT} \
model.type=torch backend.type=pt2 \
filter.type="[nan,inf,dup]" \
fuzz.save_test=${PATH_TO_SAVE_TESTS}
### Screenshots
_No response_
### Logs
_No response_
### Front-end framework
_No response_
### Version of the front-end framework
_No response_
### Back-end engine
_No response_
### Version of the front-end engine
_No response_
### Other context
_No response_
Hi!
When trying to run NNSmith on TorchJIT, I encountered errors like
ERROR fuzz - `make_testcase` failed with seed 1412986392. It can be NNSmith or Generator (torch) bug.
ERROR fuzz - Traceback (most recent call last):
File "nnsmith/cli/fuzz.py", line 215, in run
testcase = self.make_testcase(seed)
File "nnsmith/cli/fuzz.py", line 187, in make_testcase
model.refine_weights() # either random generated or gradient-based.
File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/__init__.py", line 40, in refine_weights
_, inputs = searcher.search(
File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/input_gen.py", line 56, in search
res = self.search_one(cur_input, max_time_ms)
File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/input_gen.py", line 110, in search_one
return SamplingSearch.search_one(self, start_inp, timeout_ms)
File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/input_gen.py", line 70, in search_one
_ = self.net(**start_inp)
File "/home/su/accdiff/thirdparty/pytorch/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/symbolnet.py", line 341, in forward
output_tensors = inst(*input_tensors)
File "/home/su/accdiff/thirdparty/pytorch/torch/nn/modules/module.py", line 1501, in _call_impl
return forward_call(*args, **kwargs)
File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/proxy_grad.py", line 108, in forward
return PGReLUFunc.apply(x)
File "/home/su/accdiff/thirdparty/pytorch/torch/autograd/function.py", line 506, in apply
return super().apply(*args, **kwargs) # type: ignore[misc]
File "/home/su/accdiff/thirdparty/nnsmith/nnsmith/materialize/torch/proxy_grad.py", line 40, in forward
return torch.relu(input)
RuntimeError: "clamp_min_scalar_cpu" not implemented for 'Half'
Source file without edits
yes | python nnsmith/cli/fuzz.py --config-dir nnsmith/config/ model.type=torch backend.type=torchjit fuzz.root=fuzz_report debug.viz=true backend.target=cuda
![DESCRIPTION](LINK.png)
No response
No response
torch-2.0.0a0+git062380d
torchjit
No response
No response
Hi. Thanks for developing this amazing project!
Just curious if nnsmith supports generating tests with dtype float16? Directly adding float16 to abstract/dtype.py
seems not working.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.