ElectroBench

What I built

A reproducible, code-graded benchmark that asks models to generate Python solve(inputs) → dict solutions for electrical engineering simulations. Tasks span power systems (pandapower), control systems (python-control), and circuit modeling (SciPy-graded waveforms, with ngspice CLI integration smoke-tested in Docker). Candidate outputs are graded against frozen references with engineering tolerances, physical-validity checks, and separated Docker runner/grader environments.

Why it matters

Textual plausibility is not enough for simulation-heavy engineering work. A model can sound correct while using the wrong API, failing to converge, extracting the wrong metric, or violating a physical constraint. ElectroBench turns “does this sound right?” into “does this execute and pass numerical grading?” — the same failure modes that matter when AI assists real engineering workflows.

Tools and domains

• Power systems: IEEE 14/30 N-1 analysis, 10 kV ring power flow — pandapower
• Control systems: buck average model + PI, margins, poles, step response — python-control
• Circuits: rectifier, RLC, diode clipper waveform metrics — SciPy grading; ngspice CLI smoke only
• Infrastructure: immutable freeze hashes, Docker runner/grader isolation, structured failure taxonomy

Blind evaluation results

The frozen blind set has 12 never-tuned tasks (3 control, 5 power-system, 4 circuit), with 5 attempts per condition. The blind set was frozen before the first paid run.

• Blind baseline: 18.3% (11/60)
• Official documentation: 50.0% (30/60)
• Documentation + diagnostic repair: 60.0% (36/60)

Documentation strongly reduced API-navigation failures in power systems, but did not solve waveform-modeling and metric-extraction errors in the circuit family.

Limitations and next steps

• Five attempts per task — descriptive pilot, not a definitive performance estimate
• One model/provider configuration in this batch
• Circuit grading uses SciPy; ngspice is CLI smoke-tested, not the grading backend
• Hidden parameterized checks were disabled for this blind batch
• Docker separation is evidence of isolation design, not a claim of production hostile-code sandboxing
• Next: broader model coverage, stronger circuit grading, and formal external review artifacts