Magic number in to_fp() about cmake-raytracer HOT 2 CLOSED

I think it's just a remnant from a time when I had the scale set differently (most likely to 1000000), and hardcoded in the number 6 as a temporary hack.

All that line is doing is making sure that if we try to convert a number like 0.0000000000001 to fixed point, it ignores everything past the first few digits of the fractional part, so it just becomes 0. You can see in the code below that the expression frac_digits - frac_length would become negative if frac_length > frac_digits, which we want to avoid, so chopping off anything past the first few characters is a simple way to ensure this doesn't happen (effectively performing frac_length = min(frac_length, 6)).

Instead of 6 it should be frac_digits now (which has value 8 at current scale). There should be no difference between rounding to 6dp vs 8dp here, which is why I didn't notice this line was wrong until you mentioned it.

from cmake-raytracer.

That also did cross my mind (though the commit that introduced that line had scale set to 10^8 already), but thanks for clarifying!

from cmake-raytracer.