Chenhao Tan - Automating Mechanistic Interpretability [Alignment Workshop]

Chenhao Tan demonstrates an automated mechanistic interpretability agent, MechEvalAgent, and describes the current state of this work-in-progress. He argues that mechanistic interpretability represents "a dream problem for research agents" because experiments run in silico with causally testable findings. AI models like Claude and Gemini already conduct interpretability research when instructed, but in 2025, the bottleneck shifted from agents running experiments, to reviewing and trusting these agents. His MechEvalAgent framework validates automated mechanistic interpretability research using three dimensions: coherence (following plans), reproducibility (rerunning experiments), and generalizability (extending findings). The research exposes critical failure modes for automated evaluation agents, including hallucinations where agents claim ablation studies but code shows otherwise. He presents a core challenge that automation will not advance interpretability unless we automate evaluation to some extent, and yet, we do not yet have the tools to evaluate automated evaluation agents. Note: The opinions shared in this event are those of the speaker(s) and may not represent the views of FAR.AI or their affiliated organizations.