1st of all, congratulations for great design goals: compiling Python code as is much better than having non portable extensions.

The speedup in your benchmarks is really exciting. However, if you don't mind I think it would be nice to benchmark mypyc against other tools that intend to improve Python performance, like Pyrex, for instance.

Dataclasses are popular so we should have at least microbenchmarks that exercise them.

Add benchmark that type checks mypy using mypy.

Try to make this work with earlier mypyc revisions so that we can generate some historical data as well.

A common use is testing performance between two mypyc revisions (usually between master and a local branch). Add an easy, documented way to do this. Currently the workflow is optimized for testing between compiled and interpreted mode.

Currently none of the benchmarks use bytes values much (outside microbenchmarks), even though manipulating binary data is a pretty common use case. Add one or more such benchmarks.

We run the benchmark suite once a day on a Linux desktop that I have running at home. It's pretty janky, since the CPU is from 2011 and it doesn't even have wired internet. I'm going to replace it with a new system with these features:

• Intel NUC (small form-factor PC)
• Core i5 laptop processor (11th gen)
• 16 GB RAM (same amount as before)
• Wired network (now possible since the small form factor makes placement flexible)

I expect that the relative performance of compiled code will be worse on the new system. The CPU is wider and has better branch prediction, among other things -- these should help interpreted code more. Still, I'd rather have "worse" results that better match current reality.

As a bonus it should consume much less power, which is nice since it's running 24/7.

Steps:

• Stop daily benchmark cron job
• Install latest updates to Ubuntu 20.04 (this may help with hardware compatibility)
• Shut down old system
• Install new system, move SSD to the new system
• Do some manual testing to ensure that things basically work
• Turn off turbo boost and hyperthreading (for more reliable timings)
• Check that timings actually are reliable
• Make sure auto updates are disabled
• Implement scaling of old timings to avoid abrupt changes when we switch to the new system
• Collect initial performance figures for the new system and make sure scaling works
• Enable cron job
• Monitor results for a while

For each new mypyc/mypy commit, run all benchmarks and collect the data in a repository. Generate reports based on the collected data that highlight changes in performance between mypyc versions.

Set up the scripts to be automatically run at least daily.

Add some benchmarks from https://github.com/python/pyperformance/tree/master/pyperformance/benchmarks (with type annotations):

• richards
• deltablue
• hexiom
• spectral norm
• nqueens

We have some support for enums, and since they are a popular feature, we should have some (micro)benchmarks that use them.

Currently non-native calls with keyword arguments are slow, but I don't think that we have many benchmarks that use these. Create a new benchmark focused on this use case, since it's a pretty important one and is probably a bottleneck. Call Python functions, C extension functions, Python classes, and extension type objects. It probably makes sense to split this into at least two separate benchmarks (for example, function vs type).

Run all benchmarks and type check using mypy in CI.

Maybe also run lint.

Named tuples are popular, at least in code that didn't initially target Python 3.7+ (dataclasses were added in 3.7), so we should have benchmarks that use them. Microbenchmarks are fine.

Does mypyc recognize numpy as a C-Compiled module or it falls back to regular python calls?

I know the docs say that Cython is better for numeric code, but I just wonder if mypyc can do this natively.

We currently run benchmarks a minimum of 5 times, which is too much when iterating on slow benchmarks. Add --num N to override the default logic.

Fix any issues and update documentation to say that macOS is supported.

No benchmark results have been collected in ~3 weeks since the GitHub ssh host key changed and broke the job.

String operations are common in Python code, but the current benchmarks don't use them much (other than some microbenchmarks). Add a string benchmark or two.

Also int_long_binary_ops. These only use bitwise ops, and they also use ~, which is unary.