Is the use of reflection APIs in the fuzzing harness okay? about libprotobuf-mutator HOT 6 CLOSED

Figured I would add my 2 cents even though I'm not a framework author.
I've definitely used reflection in my harnesses. I don't see what could go wrong with doing this.

What do you mean by state by the way? This approach is stateless and I beleive it solves your problem:

std::string DeserializeA(A& a_msg) {
  return DeserializeB(a_msg.b(), a_msg.c())
} 

std::string DeserializeB(B& b_msg, bool is_c) {
  if (!is_c) {
    return std::string(b_msg.d());
  }
  return std::string(b_msg.d()) + b_msg.e(); 
}

from libprotobuf-mutator.

@jonathanmetzman The harness you shared is really helpful, thank you :)

I think a visitor pattern might be sufficient for my use case. What I meant by state is for example, writing the proto converter such that the conversion of a given proto message happens differently based on the context that message appears in.

But there's also another notion of state that is important in the context of compiler fuzzing and that is related to making the proto converter respect programming language scoping rules e.g., only reference a previously declared variable.

from libprotobuf-mutator.

What I meant by state is for example, writing the proto converter such that the conversion of a given proto message happens differently based on the context that message appears in.

Ah so I guess my "stateless" solution doesn't qualify under this definition.

But there's also another notion of state that is important in the context of compiler fuzzing and that is related to making the proto converter respect programming language scoping rules e.g., only reference a previously declared variable.

Ah I've run into similar problems myself when exploring the possibility of writing LPM for different formats (such as glsl shaders). I don't have any great ideas for how to solve this problem (maybe LPM is better for "stateless" formats while a fully-custom mutator works better for "stateful" ones). It can get even nastier if we worry about types as well.
If you do implement a nice solution please share!

I don't quite remember how this fuzzer works, but it is a successful LPM fuzzer for SQL queries in SQLite. Maybe it uses some neat tricks for dealing with state?

from libprotobuf-mutator.

maybe LPM is better for "stateless" formats while a fully-custom mutator works better for "stateful" ones

This is definitely something worth pondering over. I remember @kcc mentioning something about the pros and cons of using LPM natively versus defining a custom mutator.

Just thinking out loud here:

• A proto spec is like a regular expression
• A program in a programming language has context-free grammar but in reality is context-sensitive (e.g., due to scoping rules etc.)
• This means a proto spec has weaker expressivity than a program
• It follows that what we may be trying to do is the "impossible" in theory but "hackable" in practice :)
• If LPM natively (without a custom mutation function tailored to the format under test) mutates a proto spec, it might generate trees that don't obey scoping rules. I just don't know how often it won't
• Either we need a custom stateful mutator or a stateful proto converter
• A custom mutator is probably more efficient because meaningless programs are not generated in the first place

If you do implement a nice solution please share!

Will do :) Also, thank you for the sqlite fuzzer, I will check it out. I can imagine sql queries also depend on context (like some sub-queries make sense for a specific set of queries or so), so I can learn from this example.

If you know of stateful custom mutators in the chromium project, please share! Thanks once again.

from libprotobuf-mutator.

Sorry for the late (and unhelpful =) response to this.

I agree with your "out loud" thoughts.

If you know of stateful custom mutators in the chromium project, please share! Thanks once again.

Not off the top of my head.

from libprotobuf-mutator.

Closing because the original question was answered

from libprotobuf-mutator.