[fpv] Investigate prim and top-level IP FPV results and figure out what to do about opentitan HOT 9 CLOSED

For the failing pinmux tests, PR #22754 should get them running again.

from opentitan.

@rswarbrick: How much effort is left just to get pinmux numbers above the limit, what do you think?

from opentitan.

I spent some time looking at this yesterday and the answer is "quite a bit, for coverage" and also "but maybe the limit is wrong".

In more detail, across all the FPV tests in the issue:

• Pass rates are mostly high.
  • The pinmux_fpv pass rates from last weekend were low, but these should have been fixed by #23216 (which was merged on Tuesday)
  • Other pass rates are high, except for prim_fifo_sync_fpv and prim_fifo_async_sram_adapter_fpv. These are both tooling issues: probably worth working around in each case, but neither shows meaningful missing test coverage.
• Coverage numbers for the IP FPV results (including for pinmux) are currently well under the 90% limit
  • I think they are currently better than they were when we taped out the ES chip(!), so we probably weren't too worried then...
  • There is some sensible work that we could do to include the coverage numbers, but it can probably reasonably go an the back burner.

from opentitan.

Thanks for your investigations @rswarbrick. IIRC, FPV is run weekly, so we'll get an update for #23216 next Monday.

IIRC you've also investigated the coverage gaps and reported that most of these gaps were due the COIs of our assertions not touching the signals in question, in particualr inside u_reg, and further that this most likely hasn't been covered before. Further, you've also been checking the Earlgrey-M2.5.2-RC0 tag again and the results look better now.

Based on this, I am supportive to conclude this work for Earlgrey-PROD / close this M4 issue. It would probably be worth to document the open gaps and outline the next steps for closing these. This would give us a starting point for the future and will speed up things when we decide we need to more work here. Let's discuss this in the issue triage meeting.

from opentitan.

Discussed in triage: No regressions w.r.t. Earlgrey-ES. We also have added top-level test cases to cover pinmux permutations. prims are a known gap. The PROVE_COV from prim_onehot_check looks concerning, though. @rswarbrick What are your thoughts on that one?

from opentitan.

It's not all that bad:

|+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++|
| Instance Name                                              | Formal                                   | Stimuli                                  | Checker                                  |
|                                                            | Coverage                                 | Coverage                                 | Coverage                                 |
|+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++|
|prim_onehot_check                                           |100.00% (3/3)                             |100.00% (3/3)                             |0.00% (0/0)                               |
|+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++|

Maybe 0/0 should appear in grey in some way, rather than being a red 0%. But there's no uncovered item here.

from opentitan.

@vogelpi to take a look

@rswarbrick future work would probably be to work around tooling issues (about prim). For pinmux it's just coverage.

@moidx much better than what we had before!

from opentitan.

The latest regression results look as follows.

• IPs:
  

• prims:
  

As previously mentioned, we're done here. I am closing this issue and will create a new one to track potential future work (low priority). Thanks @rswarbrick for the hard work here!

from opentitan.

The new issue is here #23499

from opentitan.