Skip to content

Fix benchmark runner bin layout#75

Merged
govindchari merged 1 commit intomainfrom
fix/benchmark-runner-bin-layout
May 6, 2026
Merged

Fix benchmark runner bin layout#75
govindchari merged 1 commit intomainfrom
fix/benchmark-runner-bin-layout

Conversation

@govindchari
Copy link
Copy Markdown
Member

@govindchari govindchari commented May 6, 2026

No description provided.

@govindchari govindchari force-pushed the fix/benchmark-runner-bin-layout branch from f5d144f to 8f464a3 Compare May 6, 2026 00:57
@govindchari govindchari merged commit 4cfef1f into main May 6, 2026
24 checks passed
@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 6, 2026

Download Benchmark Artifacts

Benchmark Summary

Problems Solved

Dataset Main Solved Diff Solved Main Iters Diff Iters Main IR Iters Diff IR Iters
cutest 43 / 62 🟢 47 / 62 786 🟢 746 1556 🟢 1414
misc 🟢 1 / 3 0 / 3 0 0 0 0
mm 136 / 138 🟢 137 / 138 2547 🟢 2499 4200 🟢 2769
mpc 63 / 64 63 / 64 🟢 480 490 587 🟢 253

Differences in Solved Problems

  • Diff branch additionally solved: CMPC11, CMPC13, CMPC14, CMPC15, CMPC16, CMPC7, CMPC8, CVXQP1_L, CVXQP3_L
  • Main additionally solved: BOYD2, CLEUVEN6, LEUVEN1, TABLE1, cvxpy_lp5

Iteration Regressions (diff took more iterations)

  • ALLINQP: diff=14, main=12
  • AVGASA: diff=9, main=8
  • AVGASB: diff=9, main=8
  • CBS: diff=34, main=33
  • CLEUVEN7: diff=34, main=19
  • GMNCASE2: diff=9, main=8
  • GMNCASE3: diff=9, main=8
  • HIER163D: diff=27, main=26
  • HS35I: diff=7, main=6
  • CVXQP2_S: diff=9, main=8
  • DTOC3: diff=8, main=4
  • DUAL1: diff=11, main=10
  • DUAL2: diff=9, main=8
  • DUAL4: diff=9, main=8
  • DUALC5: diff=13, main=10
  • DUALC8: diff=14, main=12
  • EXDATA: diff=17, main=16
  • GOULDQP3: diff=8, main=6
  • HS268: diff=9, main=5
  • HS35: diff=6, main=5
  • HS35MOD: diff=11, main=10
  • LISWET1: diff=65, main=64
  • LISWET10: diff=71, main=68
  • LISWET11: diff=59, main=58
  • LISWET12: diff=100, main=93
  • LISWET8: diff=97, main=91
  • LISWET9: diff=98, main=91
  • MOSARQP2: diff=9, main=8
  • Q25FV47: diff=31, main=27
  • QADLITTL: diff=15, main=11
  • QAFIRO: diff=16, main=15
  • QBANDM: diff=23, main=21
  • QBEACONF: diff=17, main=12
  • QBORE3D: diff=29, main=18
  • QE226: diff=21, main=18
  • QETAMACR: diff=29, main=26
  • QGFRDXPN: diff=29, main=25
  • QGROW15: diff=21, main=18
  • QGROW22: diff=25, main=22
  • QGROW7: diff=21, main=19
  • QISRAEL: diff=31, main=27
  • QPCBLEND: diff=17, main=16
  • QPTEST: diff=7, main=5
  • QSCAGR25: diff=21, main=17
  • QSCAGR7: diff=21, main=15
  • QSCRS8: diff=34, main=23
  • QSCSD8: diff=12, main=11
  • QSCTAP2: diff=16, main=14
  • QSCTAP3: diff=17, main=16
  • QSHARE1B: diff=30, main=27
  • QSHELL: diff=38, main=28
  • QSHIP04L: diff=21, main=14
  • QSHIP04S: diff=21, main=14
  • QSHIP08L: diff=23, main=14
  • QSHIP08S: diff=64, main=14
  • QSHIP12L: diff=29, main=18
  • QSHIP12S: diff=24, main=18
  • QSIERRA: diff=34, main=24
  • QSTANDAT: diff=26, main=24
  • S268: diff=9, main=5
  • STCQP2: diff=8, main=7
  • YAO: diff=66, main=63
  • ballOnPlate_4: diff=9, main=8
  • dcMotor_1: diff=8, main=7
  • dcMotor_2: diff=9, main=8
  • dcMotor_3: diff=10, main=8
  • dcMotor_4: diff=9, main=8
  • dcMotor_6: diff=7, main=6
  • doubleInvertedPendulum_1: diff=7, main=5
  • doubleInvertedPendulum_2: diff=10, main=8
  • doubleInvertedPendulum_3: diff=8, main=5
  • helicopter_1: diff=6, main=4
  • helicopter_2: diff=7, main=6
  • helicopter_3: diff=7, main=5
  • pendulum_3: diff=6, main=5
  • quadcopter_4: diff=12, main=10
  • robotArm_1: diff=8, main=5
  • robotArm_2: diff=12, main=10
  • shell_1: diff=7, main=6
  • spacecraft_1: diff=10, main=9
  • spacecraft_2: diff=10, main=9
  • springMass_3: diff=6, main=5
  • tripleInvertedPendulum_1: diff=8, main=7
  • tripleInvertedPendulum_2: diff=9, main=7
  • tripleInvertedPendulum_3: diff=10, main=8

Iteration Improvements (diff took fewer iterations)

  • BTS4: diff=31, main=34
  • CONT5-QP: diff=31, main=35
  • CVXQP2: diff=10, main=22
  • DALE: diff=32, main=34
  • GMNCASE4: diff=6, main=10
  • HIE1327D: diff=27, main=29
  • HIE1372D: diff=26, main=27
  • HIER13: diff=17, main=20
  • HIER133A: diff=27, main=30
  • HIER133D: diff=26, main=29
  • HIER133E: diff=26, main=29
  • HIER16: diff=15, main=18
  • JJTABEL3: diff=36, main=37
  • NINE12: diff=17, main=20
  • NINE5D: diff=16, main=19
  • NINENEW: diff=17, main=20
  • RDW2D51U: diff=4, main=6
  • RDW2D52U: diff=4, main=6
  • TABLE7: diff=26, main=28
  • TABLE8: diff=17, main=20
  • TWO5IN6: diff=17, main=19
  • BOYD1: diff=34, main=37
  • CONT-050: diff=7, main=10
  • CONT-100: diff=7, main=11
  • CONT-101: diff=6, main=10
  • CONT-200: diff=7, main=13
  • CONT-201: diff=7, main=12
  • CONT-300: diff=8, main=12
  • CVXQP1_M: diff=10, main=13
  • CVXQP2_L: diff=10, main=22
  • CVXQP2_M: diff=10, main=13
  • CVXQP3_M: diff=12, main=17
  • DUALC1: diff=16, main=21
  • DUALC2: diff=13, main=18
  • HUES-MOD: diff=4, main=13
  • LISWET5: diff=7, main=8
  • LISWET6: diff=9, main=10
  • LISWET7: diff=8, main=60
  • PRIMALC1: diff=16, main=25
  • PRIMALC2: diff=14, main=24
  • PRIMALC5: diff=12, main=20
  • PRIMALC8: diff=19, main=21
  • QBRANDY: diff=18, main=19
  • QCAPRI: diff=38, main=40
  • QFFFFF80: diff=40, main=65
  • QFORPLAN: diff=23, main=25
  • QPCBOEI2: diff=25, main=26
  • QPCSTAIR: diff=22, main=25
  • QPILOTNO: diff=38, main=105
  • QRECIPE: diff=21, main=23
  • QSC205: diff=15, main=16
  • QSCFXM1: diff=24, main=26
  • QSCFXM2: diff=28, main=31
  • QSCFXM3: diff=29, main=31
  • QSEBA: diff=25, main=30
  • QSHARE2B: diff=16, main=19
  • QSTAIR: diff=19, main=23
  • STADAT1: diff=10, main=20
  • STADAT2: diff=14, main=17
  • STADAT3: diff=11, main=16
  • UBH1: diff=6, main=7
  • aircraft_11: diff=8, main=11
  • aircraft_12: diff=8, main=11
  • aircraft_13: diff=10, main=14
  • binaryDistillationColumn_1: diff=7, main=9
  • binaryDistillationColumn_2: diff=7, main=9
  • fiordosExample_1: diff=7, main=8
  • fiordosExample_3: diff=9, main=10
  • nonlinearCstr_1: diff=8, main=9
  • nonlinearCstr_2: diff=8, main=9
  • pendulum_2: diff=7, main=8
  • quadcopter_1: diff=6, main=7
  • shell_3: diff=9, main=10
  • springMass_1: diff=9, main=10
  • springMass_4: diff=6, main=7
  • toyExample_3: diff=5, main=6
  • toyExample_4: diff=8, main=9
  • toyExample_5: diff=8, main=9

IR Iteration Regressions (diff used more IR iterations)

  • CLEUVEN7: diff=280, main=149
  • HIER163D: diff=59, main=58
  • TABLE3: diff=79, main=76
  • TARGUS: diff=80, main=78
  • CONT-050: diff=3, main=2
  • CONT-100: diff=9, main=2
  • CONT-101: diff=11, main=0
  • CONT-200: diff=20, main=4
  • CONT-201: diff=19, main=0
  • CONT-300: diff=26, main=0
  • DTOC3: diff=22, main=0
  • QBEACONF: diff=43, main=35
  • QBORE3D: diff=51, main=32
  • QSCAGR25: diff=31, main=26
  • QSHELL: diff=61, main=48
  • forcesExample_3: diff=12, main=10
  • nonlinearCstr_1: diff=5, main=1
  • nonlinearCstr_2: diff=5, main=1
  • nonlinearCstr_3: diff=1, main=0
  • springMass_4: diff=2, main=1

IR Iteration Improvements (diff used fewer IR iterations)

  • BTS4: diff=87, main=117
  • CONT5-QP: diff=7, main=45
  • CVXQP2: diff=0, main=38
  • DALE: diff=37, main=39
  • HIE1327D: diff=50, main=56
  • HIE1372D: diff=38, main=42
  • HIER13: diff=33, main=42
  • HIER133A: diff=55, main=69
  • HIER133D: diff=52, main=68
  • HIER133E: diff=52, main=68
  • HIER16: diff=32, main=48
  • HIER163A: diff=61, main=62
  • JJTABEL3: diff=158, main=162
  • NINE12: diff=38, main=69
  • NINE5D: diff=43, main=61
  • NINENEW: diff=38, main=56
  • TABLE7: diff=36, main=38
  • TABLE8: diff=9, main=10
  • TWO5IN6: diff=36, main=51
  • BOYD1: diff=125, main=206
  • CVXQP1_M: diff=4, main=41
  • CVXQP1_S: diff=0, main=7
  • CVXQP2_L: diff=0, main=38
  • CVXQP2_M: diff=0, main=18
  • CVXQP3_M: diff=8, main=145
  • CVXQP3_S: diff=0, main=4
  • DUALC1: diff=20, main=110
  • DUALC2: diff=10, main=91
  • DUALC5: diff=8, main=36
  • DUALC8: diff=23, main=58
  • HUES-MOD: diff=8, main=34
  • HUESTIS: diff=12, main=50
  • PRIMALC1: diff=24, main=42
  • PRIMALC2: diff=20, main=41
  • PRIMALC5: diff=16, main=37
  • PRIMALC8: diff=33, main=35
  • QADLITTL: diff=14, main=18
  • QAFIRO: diff=8, main=12
  • QBANDM: diff=34, main=40
  • QBRANDY: diff=16, main=19
  • QCAPRI: diff=161, main=193
  • QETAMACR: diff=65, main=116
  • QFFFFF80: diff=390, main=552
  • QFORPLAN: diff=115, main=189
  • QGFRDXPN: diff=60, main=112
  • QGROW15: diff=29, main=32
  • QGROW22: diff=32, main=38
  • QGROW7: diff=26, main=31
  • QISRAEL: diff=0, main=6
  • QPCBLEND: diff=4, main=9
  • QPCBOEI1: diff=48, main=50
  • QPCBOEI2: diff=56, main=64
  • QPCSTAIR: diff=33, main=55
  • QPILOTNO: diff=317, main=338
  • QRECIPE: diff=36, main=60
  • QSCAGR7: diff=16, main=18
  • QSCFXM1: diff=75, main=98
  • QSCFXM2: diff=99, main=131
  • QSCFXM3: diff=104, main=149
  • QSCORPIO: diff=0, main=16
  • QSCRS8: diff=29, main=44
  • QSCTAP1: diff=0, main=9
  • QSCTAP2: diff=0, main=9
  • QSCTAP3: diff=7, main=14
  • QSEBA: diff=45, main=94
  • QSHARE1B: diff=27, main=36
  • QSHARE2B: diff=17, main=29
  • QSHIP04L: diff=12, main=27
  • QSHIP04S: diff=11, main=21
  • QSHIP08L: diff=3, main=32
  • QSHIP08S: diff=5, main=32
  • QSHIP12L: diff=7, main=40
  • QSHIP12S: diff=8, main=38
  • QSIERRA: diff=112, main=120
  • QSTAIR: diff=30, main=42
  • QSTANDAT: diff=22, main=28
  • STADAT1: diff=0, main=8
  • aircraft_1: diff=2, main=9
  • aircraft_10: diff=3, main=9
  • aircraft_11: diff=2, main=16
  • aircraft_12: diff=2, main=16
  • aircraft_13: diff=10, main=20
  • aircraft_2: diff=2, main=9
  • aircraft_3: diff=0, main=2
  • aircraft_4: diff=2, main=9
  • binaryDistillationColumn_1: diff=8, main=12
  • binaryDistillationColumn_2: diff=8, main=12
  • dcMotor_1: diff=10, main=29
  • dcMotor_2: diff=10, main=29
  • dcMotor_3: diff=12, main=31
  • dcMotor_4: diff=6, main=16
  • dcMotor_5: diff=18, main=58
  • dcMotor_6: diff=15, main=46
  • doubleInvertedPendulum_1: diff=0, main=4
  • doubleInvertedPendulum_2: diff=0, main=6
  • doubleInvertedPendulum_3: diff=0, main=4
  • fiordosExample_1: diff=2, main=9
  • fiordosExample_3: diff=14, main=16
  • forcesExample_1: diff=6, main=7
  • helicopter_2: diff=0, main=3
  • pendulum_1: diff=0, main=1
  • quadcopter_1: diff=6, main=8
  • quadcopter_2: diff=6, main=8
  • quadcopter_3: diff=6, main=8
  • robotArm_1: diff=0, main=4
  • robotArm_2: diff=0, main=21
  • shell_1: diff=8, main=10
  • shell_2: diff=7, main=10
  • shell_3: diff=6, main=14
  • springMass_1: diff=14, main=16
  • toyExample_1: diff=6, main=9
  • toyExample_2: diff=3, main=6
  • toyExample_3: diff=8, main=13
  • toyExample_4: diff=4, main=10
  • toyExample_5: diff=4, main=10
  • tripleInvertedPendulum_1: diff=0, main=12
  • tripleInvertedPendulum_2: diff=0, main=12
  • tripleInvertedPendulum_3: diff=0, main=12

@github-actions
Copy link
Copy Markdown

github-actions Bot commented May 6, 2026

Download Benchmark Artifacts

Benchmark Summary

Problems Solved

Dataset Main Solved Diff Solved Main Iters Diff Iters Main IR Iters Diff IR Iters
cutest 43 / 62 43 / 62 1069 1069 2301 2301
misc 1 / 3 1 / 3 15 15 2 2
mm 136 / 138 136 / 138 2615 2615 4420 4420
mpc 63 / 64 63 / 64 480 480 587 587

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@govindchari