Publications
Post-cohort publications, preprints, and research outputs from PrincInt, Fellows and Affiliates
Works are organized chronologically, with flagship contributions highlighted
2026
Andrew Mack, Kraig Yuheng Tou, Mark Henry, Zhengxun Wu, Lauren Greenspan
Scaling Interpretable Transformers with Parity Bottleneck Layers
arxiv • Machine Learning
Language models are thought to exhibit the phenomenon of superposition, representing many more features than dimensions in their residual streams. Sparse autoencoders (SAEs) are designed to recover such features post-hoc, but training models that are interpretable by construction has remained impractical, as a per-layer over-complete bottleneck is prohibitively expensive in both memory and compute. To overcome this issue, we introduce the ParityTransformer, a GPT-2-scale architecture whose intermediate representations are efficient and wide / sparse by design.
Aryeh Brill, Tom Ingebretsen Carlson
Critical Percolation as a Synthetic Data Model for Interpretability
arxiv • Machine Learning
Neural networks learn features that reflect the hierarchical, multi-scale structure of natural data. Synthetic datasets used to evaluate interpretability methods typically lack this structure, limiting their value as realistic toy models. To close this gap, we introduce a family of synthetic datasets consisting of hierarchical functions defined on critical mean-field percolation clusters embedded in a high-dimensional data space.
Andrew Mack, Nina Panickssery, Alexander Matt Turner
Mechanistically Eliciting Latent Behaviors in Language Models
arxiv • Machine Learning
We aim to discover diverse, generalizable perturbations of LLM internals that can surface hidden behavioral modes. Such perturbations could help reshape model behavior and systematically evaluate potential risks. We introduce Causal Perturbative Elicitation (CPE), an unsupervised method for discovering interpretable low-rank adapters (LoRAs) that can elicit these latent behaviors.
Dmitry Vaintrob, Lauren Greenspan
Mean field sequence: an introduction
LessWrong • Blog
This is the first post in a planned series about mean field theory. The posts are a combination of an explainer and some original research/ experiments. The goal of this series is to explain an approach to understanding and interpreting model internals which we informally denote “mean field theory” or MFT.
Dmitry Vaintrob, Kaarel Hänni
Width-Robust Learnability in Mean-Field Bayesian Neural Networks
arxiv, 2026 • Machine Learning
Infinite-width limits are a standard way to reason about neural networks, but it is not automatic that the limiting learner has the same complexity-theoretic inductive bias as large finite networks. We study this question for Bayesian neural networks at the mean-field, or critical feature-learning, scaling.
Lauren Greenspan, et. al
Towards Worst-Case Guarantees with Scale-Aware Interpretability
Cornell University, 2026 • High Energy Physics – Theory
Neural networks organize information according to the hierarchical, multi-scale structure of natural data. Methods to interpret model internals should be similarly scale-aware, explicitly tracking how features compose across resolutions and guaranteeing bounds on the influence of fine-grained structure that is discarded as irrelevant noise. We posit that the renormalisation framework from physics can meet this need by offering technical tools that can overcome limitations of current methods. Moreover, relevant work from adjacent fields has now matured to a point where scattered research threads can be synthesized into practical, theory-informed tools. To combine these threads in an AI safety context, we propose a unifying research agenda — \emph{scale-aware interpretability} — to develop formal machinery and interpretability tools that have robustness and faithfulness properties supported by statistical physics.
Dmitry Vaintrob
A tale of three theories: sparsity, frustration, and statistical field theory
LessWrong • Blog
This post combines three aspects of interpretability: 1) Questions about computation in superposition; 2) The mean field approach to understanding neural nets; 3) The agenda at PIBBSS on multi-scale structure in neural nets, and to physical theories of renormalization that study when phenomena at one scale can be decoupled from another.
2025
Jennifer Lin
Feature Identification via the Empirical NTK
Lin, 2025 • Machine Learning
We provide evidence that eigenanalysis of the empirical neural tangent kernel (eNTK) can surface feature directions in trained neural networks. Across three increasingly realistic settings — a 1-layer MLP trained on modular addition, a 1-layer Transformer trained on modular addition and the pretrained language model Gemma-3-270M — we show that top eigenspaces of the eNTK align with ground-truth or interpretable features. In the modular arithmetic examples, top eNTK eigenspaces align with the Fourier features used by the MLP and the Fourier features at seed-dependent frequencies used by the Transformer to implement known ground-truth algorithms. Moreover, the alignment of the relevant subspaces evolves over training, with its first derivative peaking near the onset of grokking. For Gemma-3-270M, we compute top eNTK eigendirections on a dataset of TinyStories context windows and check their alignment with an automatically-generated set of parts-of-speech and other grammatical feature directions. We find that the alignment of eNTK eigendirections with grammar features outperforms a same-budget baseline of PCA on model activations. These results suggest that eNTK eigenanalysis may provide a new handle towards identifying features in trained models for mechanistic interpretability.
Mel Andrews
A Model for Scaling Laws of General Intelligence
OpenReview, 2025 • Scaling Laws
This philosophical examination challenges prevailing narratives about AI disrupting scientific practice and knowledge production. Andrews argues that many claims about AI transforming science rest on a confused understanding of scientific objectivity. Rather than AI enabling fundamentally new forms of objectivity or disrupting how science operates, the paper demonstrates that such claims mistake the instrumental role of AI for a conceptual revolution. The work developed during Andrews’ PIBBSS fellowship but was only published years later.
Mel Andrews
AI, Objectivity, and the Limits of Scientific Disruption
Erkenntnis, 2025 • Philosophy of Science
This philosophical examination challenges prevailing narratives about AI disrupting scientific practice and knowledge production. Andrews argues that many claims about AI transforming science rest on a confused understanding of scientific objectivity. Rather than AI enabling fundamentally new forms of objectivity or disrupting how science operates, the paper demonstrates that such claims mistake the instrumental role of AI for a conceptual revolution. The work developed during Andrews’ PIBBSS fellowship but was only published years later.
Eleni Angelou
AI Safety as an Emerging Paradigm
Routledge/CRC Book Chapter, 2025 • Paradigm Analysis
Angelou examines AI safety through the lens of paradigm formation, asking whether AI safety constitutes an emerging scientific paradigm in the Kuhnian sense. The chapter analyzes the field’s conceptual foundations, methodological commitments, and community structures, identifying both paradigmatic features (shared problem sets, research programs) and pre-paradigmatic characteristics (competing frameworks, lack of consensus on fundamentals). This meta-level analysis helps situate AI safety research within broader philosophy and history of science, providing perspective on the field’s maturity and development trajectory.
Holly Elmore
Genes did misalignment first: comparing gradient hacking and meiotic drive
April 2025 • Evolutionary Biology / AI Safety Analogy
Elmore’s work draws a parallel between AI gradient hacking and meiotic drive in biology, suggesting natural selection solved alignment issues millennia ago. Selfish genetic elements, like gradient-hacking model components, prioritize their own transmission over system fitness. Meiosis, a “governance mechanism,” prevents alleles from predicting their genomic context, aligning their incentives with organismal health. This implies AI alignment might need similar “randomization” to prevent components from optimizing for self-preservation over task performance. This PIBBSS fellowship-inspired work offers a biological case study for alignment problems.
2024
Shai et al
Transformers Represent Belief State Geometry in Their Residual Stream
NeurIPS 2024
This paper presents compelling empirical evidence that transformer language models encode belief states—probability distributions over world states—in interpretable geometric structures within their residual streams. Drawing on computational mechanics, the authors demonstrate that transformers maintain representations of uncertainty and track competing hypotheses about underlying generative processes. This work suggests that interpretability research could benefit from treating neural activations as embedded probability distributions rather than merely feature vectors. The finding that belief geometry is preserved through layers provides a potential foundation for understanding how models reason under uncertainty.
Gabriel Weil
Tort Law as a Tool for Mitigating Catastrophic Risk from Artificial Intelligence
SSRN, January 2024 • Initial work from 2023 Fellowship • Vox coverage • Law & Economics / AI Governance
Weil’s paper addresses the challenge of AI governance where catastrophic harm precludes traditional tort law. The core idea is to “pull forward” expected liability from catastrophic scenarios into recoverable damages in sub-catastrophic cases. This requires doctrinal changes like punitive damages without malice and treating AI development as an abnormally dangerous activity with strict liability. The paper also suggests expanding foreseeability doctrines, reconsidering the value of human life in tort law, and legislative interventions such as mandatory liability insurance for AI developers, diverting punitive damages to an AI safety fund, and pre-announcing liability rules to shape incentives. Weil acknowledges tort law’s limitations, including developer solvency, truly unforeseeable risks, and the speed of AI development. This work represents sophisticated institutional design, well-received by the AI safety community and media.
Aron Vallinder
Cultural Evolution of Cooperation among LLM Agents
arXiv, December 2024; AAMAS 2025 Extended Abstract • Multi-agent Systems
Vallinder investigates whether large language model agents can develop and maintain cooperative norms through cultural evolution mechanisms. The paper demonstrates that LLM-based agents, when placed in repeated social dilemmas, exhibit norm formation dynamics analogous to those observed in human societies. This includes the emergence of punishment mechanisms, in-group favoritism, and stable cooperation even in the absence of explicit coordination. The work bridges evolutionary game theory with modern AI systems, suggesting that insights from cultural evolution could inform multi-agent AI safety. The findings raise both opportunities (norms as an alignment mechanism) and risks (harmful norm formation in AI systems).
Matthew Clarke
Compositionality and Ambiguity: Latent Co-occurrence and Interpretable Subspaces
LessWrong, December 2024 • Interpretability
Clarke examines how sparse autoencoders (SAEs) handle compositional features and latent co-occurrence patterns in language models. The work identifies systematic patterns in how SAE features activate together, suggesting that interpretability research needs richer frameworks beyond single-feature analysis. Clarke argues that understanding feature composition—how multiple features combine to represent complex concepts—is essential for scaling interpretability to real-world model behaviors. The paper proposes viewing interpretable subspaces not as collections of independent features but as structured spaces where co-occurrence patterns carry semantic meaning. This represents an important direction for next-generation interpretability methods.
Magdalena Wache
Factored Space Models: Towards Causality Between Levels of Abstraction
arXiv, July 2024 • Causality / Representation Learning
Wache develops a mathematical framework for reasoning about causal relationships between different levels of abstraction, addressing a fundamental challenge in both AI interpretability and scientific explanation. The paper introduces “factored space models” that formalize how high-level causal structures relate to low-level mechanisms, providing tools for understanding when abstractions preserve causal information. This work has implications for interpretability (when can we trust high-level explanations of neural networks?) and for alignment (how do we ensure safety properties hold across abstraction levels?). The framework offers conceptual clarity on long-standing philosophical questions about reduction and emergence, grounding them in formal causal modeling.
Dombrowski & Guillaume Corlouer
An Information-Theoretic Study of Lying in LLMs
ICML 2024 Workshop (LLMs and Cognition) • 2024 Affiliate Project
This paper applies information-theoretic methods to understand and detect deception in large language models. Corlouer and Dombrowski examine lying not just as generating false statements but as systematically distorting information flows, using mutual information and entropy measures to characterize deceptive behavior. The work demonstrates that information-theoretic signatures can distinguish lying from errors or ignorance, potentially enabling better deception detection mechanisms.
Guillaume Corlouer
Degeneracies Are Sticky for SGD
AI Alignment Forum, June 2024 • Optimization / Singular Learning Theory
Corlouer investigates how stochastic gradient descent interacts with degeneracies—regions of parameter space where multiple configurations produce identical behavior. Drawing on singular learning theory, the paper argues that SGD exhibits “stickiness” near degenerate regions, with training dynamics spending disproportionate time near these high-symmetry configurations. This has implications for understanding neural network training: degeneracies may act as attractors that shape the solutions SGD finds. The work contributes to mechanistic understanding of optimization, which matters for alignment if training dynamics systematically favor certain solution types over others.
Ninell Oldenburg & Zhi-Xuan
Bayesian Learning of Social Norms
AAMAS 2024 • Started during 2023 Fellowship
Oldenburg and Zhi-Xuan model social norm learning as Bayesian inference, proposing computational mechanisms for how agents infer normative rules from observed behavior. The paper demonstrates that social norms can emerge from rational inference under uncertainty rather than requiring specialized social cognition modules. This work has implications for multi-agent AI systems: if norms are learnable through Bayesian updating, then norm-following AI could potentially be achieved through appropriate observational learning rather than explicit programming. The paper also identifies risks—mislearned or adversarially manipulated norms—that could arise from these mechanisms.
2023
Patrick Butlin et al. (George Deane)
Consciousness in Artificial Intelligence: Insights from the Science of Consciousness
arXiv Report, August 2023 • AI Consciousness Assessment
This multi-author report applies neuroscience and philosophy of consciousness frameworks to assess whether AI systems could be conscious. Butlin and collaborators evaluate leading theories of consciousness (global workspace theory, integrated information theory, higher-order theories) and what they would predict for current and future AI architectures. The report concludes that while current systems almost certainly lack consciousness, certain architectural features could plausibly support conscious states. The work aims to ground AI consciousness discussions in scientific rather than purely speculative frameworks, providing researchers and policymakers with empirically-grounded assessment criteria.