The Pyston project published a set of macro benchmarks that they used. We can work on incorporating those into the pyperformance suite.

• https://github.com/pyston/python-macrobenchmarks (and blog)
• https://github.com/python/pyperformance (and docs)

TODO:

• changes to pyperformance
  • Support user-defined benchmark suites. python/pyperformance#109
  • any subsequent fixes
• get comparison of results on our bench machine w/ & w/o my branch
  • req-1636475547-esnow vs. req-1636488560-esnow
• changes to pyston-benchmarks
  • https://github.com/ericsnowcurrently/pyston-macrobenchmarks/tree/pyperformance
  • verify it works with pyperformance
  • verify pyston benchmarks branch preserves existing behavior
  • check in with pyston team
    • Use pyperformance to run the benchmarks. pyston/python-macrobenchmarks#2
  • PR (post after pyperformance PR)
    • Use pyperformance to run the benchmarks. pyston/python-macrobenchmarks#3
• combined manifest
  • add [includes] section to manifest
  • try out default + pyston
  • try it on the bench host (3.9 at least)
  • record steps
• ask Pablo about work needed to switch to new pyperformance (w/ combined manifest)
  • Add pyston benchmarks to speed.python.org. #253

Steps to incorporate Pyston benchmarks on a benchmarking host:

1. clone the Pyston benchmarks repo
2. create a manifest file with just an "includes" section, containing references to the manifests you're combining
3. add a "manifest=" entry to the config file you're passing to pyperformance compile ...
4. proceed as usual

More concretely:

#git clone --branch benchmark-management https://github.com/ericsnowcurrently/python-performance pyperformance
#git clone --branch pyperformance https://github.com/ericsnowcurrently/pyston-macrobenchmarks
git clone https://github.com/pyston/python-macrobenchmarks pyston-macrobenchmarks

cat > MANIFEST.combined << EOF
[includes]
<default>
pyston-macrobenchmarks/benchmarks/MANIFEST
EOF

cat > ./cpython-perf.ini << EOF
[config]
...

[scm]
...

[compile]
...

[run_benchmark]
manifest = MANIFEST.combined
...
EOF

PYTHONPATH=./pyperformance python3 -m pyperformance compile ./cpython-perf.ini main