Conference Agenda

We all remember cramming for a test, scrambling to prepare in the final stretch by specifically targeting what we knew would be covered. When a benchmark catches on, it incentivizes model builders to do the engineering equivalent of cramming for the test, preparing models to excel on the specific benchmark. Although not a form of cheating, this potent practice of training on the test task confounds model comparisons and threatens benchmark validity.

But what if the problem also charted a path forward? If adapting to the test distribution is so effective, why not push the logic to the extreme: adapt the model to each test instance? Doing so is, in fact, the essence of test-time training, an evolving conceptual toolkit to improve models at test-time by turning each test instance into its own learning problem.

In this talk, I’ll speculate about a connection between the problem of training on the test task and test-time training. The natural limit point for both is instance-optimal adaptation, pointing at a confluence between model testing and model training: As test-time compute budgets grow, the line between training and testing diminishes. I’ll conclude by imagining a future of machine learning that fully embraces the tyranny of testing.

Automated recognition of emotions and facial expressions is a key technology for effective human-computer interaction. It is the prerequisite for overcoming the "10-hour barrier"—a critical threshold where interaction often fails once the initial novelty effect wears off and the system fails to provide social depth.

The current state of the art in purely data-driven AI models faces a fundamental problem: their visual representations often diverge massively from human perception. They are highly susceptible to spurious grounding—learning irrelevant correlations (e.g., identity traits instead of actual facial movements)—which severely limits their ecological validity in real, dynamic interactions. While these models "recognize" statistical patterns, they fail to anchor them in semantically meaningful concepts.

Our research addresses this limitation through a neuro-symbolic approach that bridges the statistical learning capabilities of neural networks (perception) with the consistent structure of symbolic logic (reasoning). The central mechanism driving this is Abductive Learning, implemented through an iterative cycle:

1. Perception: A neural network (e.g., ResNet-152 or Vision Transformer) generates predictions based on sequential, visual facial expression data from standard research datasets (e.g., BP4D-Spontaneous, DISFA).
2. Logical Validation: These predictions are cross-referenced against a knowledge base formalized in First-Order Logic (e.g., utilizing rules from the Facial Action Coding System).
3. Abductive Correction: Whenever inconsistencies arise (e.g., physiologically implausible muscle combinations for a specific emotion), the system infers the most probable explanation (hypothesis).

The abduced solution then acts as a direct feedback signal to correct and optimize the neural network or the knowledge base itself. This systematic process forces the system to align its internal representations with valid, human-centric concepts. Ultimately, this yields data-efficient models that do not merely reproduce statistical patterns, but contextually "understand" them, making their decision-making processes logically transparent and explainable.

Vision foundation models exhibit generalization and adaptation capabilities across a wide range of downstream applications. In language, it has been empirically demonstrated that parts of network representations arise from the linear composition of independent “concepts” which can be extracted by sparse autoencoders (SAEs). Here we propose a series of techniques to discover, annotate, and curate human-interpretable “concepts” from vision foundation models.

We introduce PatchSAE (Lim et al., ICLR 2025) to extract interpretable visual concepts at granular levels (e.g., shape, color, or object semantics) and their patch-wise spatial attributions by representing the original activation using a sparse latent representation from a CLIP foundation model. Using this framework, we explore which concepts the model captures and how these concepts influence model outputs in downstream image classification tasks. We study how model performance improves under prompt-based adaptation by comparing the concepts used for classification in prompt-adapted models, where learnable prompts are concatenated with the input, with those used in the base models. While concept activations change only slightly between the two settings, we find that most performance gains on common adaptation tasks can be explained by concepts already present in the original foundation model.

On top of PatchSAE, we introduce ConceptScope (Choi et al., NeurIPS 2025), a scalable and automated framework to systematically distinguish concepts that are important for the class label from those that frequently co-occur with the label but are not directly related. ConceptScope identifies causally important concepts for the class as well as frequently occurring but non-causal concepts that reflect dataset collection biases. We show that ConceptScope reliably detects known biases (e.g., background bias in Waterbirds dataset) and uncovers previously unannotated ones (e.g, co-occurring objects in ImageNet), offering a practical tool for dataset auditing. Our evaluation shows concept presence detection performance across six fine-grained classification datasets, achieving an F1 score of 0.72. By applying ConceptScope to real-world datasets, including ImageNet, SUN397, and Food101, we identified an average of 2.45 biased concepts per class.

Together, these approaches ground learned representations in human semantics and allow auditing datasets for bias, paving the way towards more transparent and explainable AI systems.

Medical image analysis plays a central role in diagnosis, yet image-based analysis alone cannot capture all patient-specific factors that influence disease risk and progression. Although structured clinical variables such as family history or laboratory results can improve predictive performance, they are often incomplete or missing, limiting the applicability of conventional multimodal fusion methods that assume fixed inputs.

In this work, we propose TACTIC (Tabular-Attribute Conditioned Transformer for Image Classification), a prompt-based framework that adaptively incorporates clinical information into whole-body MRI feature representations. As shown in Figure 1, TACTIC encodes available clinical attributes as semantic prompts that condition image-derived representations, enabling clinical context to directly guide visual feature aggregation. This design addresses a key limitation of traditional multimodal fusion strategies, which often degrade when clinical data is partially or entirely missing.

We evaluate TACTIC on five disease classification tasks: (1) diabetes, (2) chronic obstructive pulmonary disease (COPD), (3) breast cancer, (4) prostate cancer, and (5) metastasis in our cancer cohorts. Our method consistently improves image-based predictions when clinical information is introduced and remains effective even when only a limited subset of clinical variables is available. Figure 2 shows that, compared with standard fusion approaches, TACTIC shows greater robustness to missing clinical data and more stable performance as the amount of tabular information varies. Furthermore, unlike the standard fusion approaches, our method (purple) is the only one that surpasses the tabular-only baseline (dashed line) on every level of tabular data availability.

These results suggest that conditioning visual representations through prompting enables more effective use of complementary clinical signals than direct feature fusion. Overall, our work highlights prompt-based conditioning as a practical and robust strategy for multimodal integration in medical imaging, particularly in settings where clinical data availability is heterogeneous.

Learning meaningful representations from sparse and noisy observations is a central challenge in scientific machine learning, particularly in domains where the underlying processes follow well-defined physical laws and symmetries. In this work, we propose a controlled benchmark to study how different machine learning paradigms recover the latent geometric structure of a simple yet physically motivated problem: the reconstruction of noisy helical particle trajectories in a magnetic field. Rather than focusing solely on prediction accuracy, we investigate whether learned representations capture the intrinsic low-dimensional manifold and symmetries governing the trajectories.

We compare three complementary learning paradigms: supervised sequence modeling with Transformers, simulation-based inference (SBI) for posterior estimation of trajectory parameters, and self-supervised representation learning using Joint Embedding Predictive Architectures (JEPA). Using a lightweight simulator that generates helical tracks with configurable observation conditions, we train each approach to infer or represent the underlying trajectory parameters and analyze the geometry of the resulting latent embeddings.

To progressively approach realistic particle tracking scenarios encountered in collider experiments at CERN, the benchmark includes multiple data-generation regimes of increasing complexity. These regimes introduce varying levels of measurement noise, variable numbers of hits per trajectory, missing hits, and occasional outliers. We also consider different detector observation models, ranging from ideal continuous sampling of trajectories to discretized measurements on cylindrical detector layers. Mild deviations from the idealized physical model, such as slightly non-uniform magnetic fields, are also introduced to assess robustness.

Our evaluation goes beyond standard regression metrics and focuses on representation structure. We probe embeddings for the linear accessibility of physical parameters, analyze the alignment between latent and physical distances, evaluate neighborhood preservation between the latent space and the parameter manifold, and test whether models recover the rotational symmetry of the problem, corresponding to the (SO(2)) invariance of trajectories in the transverse plane.

This study provides a minimal yet interpretable testbed to compare how different learning paradigms organize physical information in their internal representations.