PIRAMID

Physics-Informed Research for Ambitious Mechanistic Interpretability Development

We believe that the pursuit of ambitious mechanistic interpretability (AMI) supported by a rigorous science of AI systems is essential for scalable AI alignment. Starting with the methods and frameworks of statistical physics, our research will simultaneously build up the scientific foundations of three pillars of AI safety.

Advancements in Learning Theory

We build theoretical frameworks that explain how neural networks organize and process information, and establish principled methods to bound and predict model behaviors, particularly in rare or worst-case scenarios. Projects include:

Direct Idealization of Toy Tasks, via comprehensive, though idealized, formal analysis, to guide heuristics that generalize empirically.
Indirect Idealization of Realistic Tasks, by empirically identifying hypotheses of realistic behaviors (e.g., in gpt2-small) and creating a model organism that isolates this behavior in a synthetic environment with improved theoretical tractability.

Recent Work

Interpretability Applications

We focus on developing the infrastructure (architectures, tooling) for scale-aware interpretability. The goal is to ambitiously model network internals in a way that is informed by theory and useful for AI safety researchers and engineers, grounded in fundamental understanding rather than ad-hoc evaluations and optimization practices. Current work aims to build and advance methods to:

Develop new unsupervised training methods – similar to SAEs or weight-based decomposition methods – to reverse-engineer SOTA AI systems
Explore training-time uses of interpretability tools for better safety guarantees
Design new neural network architectures which are interpretable by design

Recent Work

Feature identification with the empirical NTK — preprint, code
Mechanistically Eliciting Latent Behaviors in Language Models
Scaling Interpretable Transformers with Parity Bottleneck Layers

Data Models & Validation

We construct synthetic datasets that:

Encode increasingly complex but tractable properties of natural language data, and
As principled validation methods, can be used to assess current tools and phenomenologically guide theory construction.

Recent Work

These are deeply interconnected: theory underpins applications, which provide empirical support for theoretical predictions, and principled validation methods facilitate quick feedback between the two.

PIRAMID is a participating group in the PIAMI coordinated research program. Our current work focuses on how neural networks learn and leverage hierarchical structure in real-world data, grounded in several hypotheses:

Learned features are organized according to a hierarchical, scale-dependent notion of relevance.
Faithful interpretability tools leverage this hierarchy.
A renormalization-like framework can place principled, probabilistic bounds on cross-scale mechanistic behavior, potentially enabling worst-case guarantees.

We use these ideas a guiding heuristics, rather than strict prescriptions, and are open to using a wide variety of tools and methods.