Differences between `DirectRng` and `ForcedRng` about bolero HOT 11 CLOSED

Direct mode simply maps the byte patterns produced by the rng into the type and rejects the value if that mapping doesn't succeed. This is useful for drivers that record coverage metrics and how they relate to the input. Forced mode instead tries its best to get the generator to succeed.

But as you've seen, DirectRng is almost never what you want, especially when the type is really picky about what is valid. That being said, I think the biggest benefit is being able to sample, let's say 10000 calls, and produce the probability that a valid value is generated. This would give you a "pickiness score" for your type that may help you change it to be less picky.

from bolero.

Direct mode simply maps the byte patterns produced by the rng into the type and rejects the value if that mapping doesn't succeed. This is useful for drivers that record coverage metrics and how they relate to the input. Forced mode instead tries its best to get the generator to succeed.

Thank you for your reply. But I'd like more clarification on this. IMO, RNG should produce bytes by the demand of the generator instead of generating a fixed amount of bytes ahead of the generation like what the fuzzer does. Unless we adopt something like generate-and-test, I think generation with RNG should almost certainly succeed.

from bolero.

RNG should produce bytes by the demand of the generator instead of generating a fixed amount of bytes ahead of the generation like what the fuzzer does.

This is exactly what happens in both modes. Neither mode limits the length of the input. They just specify what to do when the bytes don't validate the constraints for the type.

from bolero.

This is exactly what happens in both modes. Neither mode limits the length of the input. They just specify what to do when the bytes don't validate the constraints for the type.

For the particular example in this issue, the constraints are the same, right? Put it another way, could you provide an example where the bytes don't validate the constraints for DirectRng and ForcedRng leads to different behavior?

from bolero.

Another example is the string generation example in #157. I don't see the generation results should differ given that both modes produce unlimited amount of random bytes.

from bolero.

For both of these examples it doesn't matter how long the bytes input is. Consider the first one: you've said "generate a usize that is always 5". The DirectRng will generate a usize and only return Some(usize) if the value it picked was 5 (very unlikely). The ForcedRng will ensure that the usize that it generated is coerced and always produces Some(5).

For the String example, it's the same thing. The String generator says "generate N random chars". In Direct mode, the char generator says "generate a random u32 and return Some(char) if the u32 value is a valid codepoint". If any of those fail, the whole String generator will fail. In the case of Forced mode, it'll make sure that u32 value is always mapped somehow to a valid char and will always return Some(char).

from bolero.

Direct mode generator for char - should only succeed if the generated u32 directly maps to a valid char:

Forced mode generator for char - should always succeed:

from bolero.

For the String example, it's the same thing. The String generator says "generate N random chars". In Direct mode, the char generator says "generate a random u32 and return Some(char) if the u32 value is a valid codepoint".

I'm sorry I'm still confused. IIUC, DirectRng does generate-and-test. For instance, gen_usize(Bound::Included(&5), Bound::Excluded(&5)) means generating a usize and test if it satisfies the constraint (==5). Following the same reasoning, gen_char(Bounded::Included(&'a'), Bounded::Included(&'z')) should generate a char and test if it's a lower-case letter. If we don't provide any bounds, it should produce a valid char, just like gen_u8(Bound::Unbounded, Bound::Unbounded) should produce a u8?

from bolero.

I think I see where you're getting hung up. The problem we run into is there's a difference between u8 and char. For u8 any 8 bit pattern is a valid u8. This means it can easily map random input into a valid u8. OTOH, char isn't valid for any bit pattern so it can't just map arbitrary bytes to one. So by nature of using a char you've placed constraints on the underlying bytes that you don't have with u8. Does that make sense?

You can see this same thing with Kani. The primitive integers just construct arbitrary bit patterns and they're done: https://github.com/model-checking/kani/blob/06c0fbe60d793f8b47fadc5fc5a83932e96f2389/library/kani/src/arbitrary.rs#L29-L67

In contrast, the char implementation needs to place constraints on the bit pattern to ensure it's valid: https://github.com/model-checking/kani/blob/06c0fbe60d793f8b47fadc5fc5a83932e96f2389/library/kani/src/arbitrary.rs#L85-L95

from bolero.

Thank you for your patience. I think I got the idea. That being said, I think users familiar with property based testing but not with fuzzing could be easily confused. Maybe it's a good idea to add more description to the Rust doc about them.

from bolero.

Closed this issue as my questions have been answered.

from bolero.