Publications

Explore our latest research by browsing pre-prints and accepted papers.

Accepted papers

Linear GCNs Need Better Bias, Not More Expressive Power

Celia Rubio-Madrigal, and Rebekka Burkholz

In Workshop on Graph Foundation Models: A New Era for Graph Machine Learning (GFM) @ ICML, 2026

Bib Abs

Do modern GNNs succeed because of architectural expressive power or because of implicit bias? We show that simple linear graph convolutions with sum aggregation and MLP classifiers can match the generalization performance of state-of-the-art graph learning methods. Theoretically, sum aggregation after a linear feature transformation can preserve neighborhood multiset information, yielding Weisfeiler–Lehman-level expressive power. Empirically, however, this potential is realized only through distillation from more complex GNN teachers. When trained from scratch, linear GCNs often fail to generalize. This reveals a gap between what simple models can express and what standard training selects. Our findings suggest that progress in graph learning may depend less on increasing expressiveness and more on controlling implicit bias, trainability, and model selection. Our experimental design provides a framework to distinguish these concepts.
@inproceedings{rubio-madrigal2026linear, title = {Linear {GCN}s Need Better Bias, Not More Expressive Power}, author = {Rubio-Madrigal, Celia and Burkholz, Rebekka}, booktitle = {Workshop on Graph Foundation Models: A New Era for Graph Machine Learning (GFM) @ ICML}, year = {2026}, url = {https://openreview.net/forum?id=cUJMrMzYVG}, }
Prune to Protect: Faster Training and Enhanced Privacy by Dynamic Data Pruning

Chinmay Joshi, Advait Gadhikar, Celia Rubio-Madrigal, Aneet Kumar Dutta, Mridula Singh, and Rebekka Burkholz

In Workshop on The Impact of Memorization on Trustworthy Foundation Models (MemFM) @ ICML, 2026

Bib Abs

Deep networks memorize parts of their training data, exposing them to privacy attacks such as membership inference. The most vulnerable samples are often hard-to-learn tail examples: they are fitted late, carry high information for generalization, and are difficult to protect without sacrificing performance. We show that this privacy–utility tension can be mitigated by learning such samples earlier. To this end, we propose Weighted Loss InfoBatch (WLIB), which dynamically prunes easy samples while re-weighting hard ones according to sample-wise pruning ratios. By adjusting the influence of informative samples throughout training, WLIB jointly reduces memorization, improves robustness to membership inference attacks, and speeds up optimization. Extensive experiments demonstrate that WLIB achieves a favorable combination of privacy, utility, and training efficiency.
@inproceedings{joshi2026prune, title = {Prune to Protect: Faster Training and Enhanced Privacy by Dynamic Data Pruning}, author = {Joshi, Chinmay and Gadhikar, Advait and Rubio-Madrigal, Celia and Dutta, Aneet Kumar and Singh, Mridula and Burkholz, Rebekka}, booktitle = {Workshop on The Impact of Memorization on Trustworthy Foundation Models (MemFM) @ ICML}, year = {2026}, url = {https://openreview.net/forum?id=IFnzabrAbl}, }
Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer?

Franka Bause, Jonas Niederle, Martin Pawelczyk, and Rebekka Burkholz

In Workshop on Compositional Learning: Safety, Interpretability, and Agents (CompLearn) @ ICML, 2026

Bib Abs

The effectiveness of multi-agent LLM deliberation depends not only on the agents’ individual predictions, but also on how they communicate and collaborate. We study this mechanism through the lens of Friedkin-Johnsen (FJ) opinion dynamics, a tractable model for analyzing stubbornness, influence, and opinion change in multi-agent systems that captures empirically observed deliberation patterns. We show that the FJ parameters are input-dependent, turning multi-agent deliberation into a mixture of experts. This perspective implies that multi-agent systems can outperform single agents and static ensembles when routing reflects agent competence. Since competence is latent in practice, we analyze how influence is established through observable proxies: agents’ self-assessed confidence, their perceived confidence, and initial alignment with other agents’ views.
@inproceedings{bause2026masaremoes, title = {Multi-Agent Systems are Mixtures of Experts: Who Becomes an Influencer?}, author = {Bause, Franka and Niederle, Jonas and Pawelczyk, Martin and Burkholz, Rebekka}, booktitle = {Workshop on Compositional Learning: Safety, Interpretability, and Agents (CompLearn) @ ICML}, year = {2026}, url = {https://arxiv.org/abs/2605.25929}, }
Don’t Trust Stubborn Neighbors: A Security Framework for Agentic Networks

Samira Abedini, Sina Mavali, Lea Schönherr, Martin Pawelczyk, and Rebekka Burkholz

In Workshop on Compositional Learning: Safety, Interpretability, and Agents (CompLearn) @ ICML, 2026

Bib Abs Code

Large Language Model (LLM)-based Multi-Agent Systems (MASs) are increasingly deployed for agentic tasks, such as web automation, itinerary planning, and collaborative problem solving. Yet, their interactive nature introduces new security risks: malicious or compromised agents can exploit communication channels to propagate misinformation and manipulate collective outcomes. In this paper, we study how such manipulation can arise and spread by borrowing the Friedkin-Johnsen opinion formation model from social sciences to propose a general theoretical framework to study LLM-MAS. Remarkably, this model closely captures LLM-MAS behavior, as we verify in extensive experiments across different network topologies and attack and defense scenarios. Theoretically and empirically, we find that a single highly stubborn and persuasive agent can take over MAS dynamics, underscoring the systems’ high susceptibility to attacks by triggering a persuasion cascade that reshapes collective opinion. Our theoretical analysis reveals three mechanisms to increase system security: a) increasing the number of benign agents, b) increasing the innate stubbornness or peer-resistance of agents, or c) reducing trust in potential adversaries. Because scaling is computationally expensive and high stubbornness degrades the network’s ability to reach consensus, we propose a new mechanism to mitigate threats by a trust-adaptive defense that dynamically adjusts inter-agent trust to limit adversarial influence while maintaining cooperative performance. Extensive experiments confirm that this mechanism effectively defends against manipulation.
@inproceedings{abedini2026donttruststubbornneighbors, title = {Don't Trust Stubborn Neighbors: A Security Framework for Agentic Networks}, author = {Abedini, Samira and Mavali, Sina and Schönherr, Lea and Pawelczyk, Martin and Burkholz, Rebekka}, booktitle = {Workshop on Compositional Learning: Safety, Interpretability, and Agents (CompLearn) @ ICML}, year = {2026}, url = {https://arxiv.org/abs/2603.15809}, }
HORST: Composing Optimizer Geometries for Sparse Transformer Training

Tom Jacobs, Rohan Jain, and Rebekka Burkholz

In Workshop on High-dimensional Learning Dynamics (HiLD) @ ICML, 2026

Bib Abs

Sparsifying transformers remains a fundamental challenge, as standard optimizers fail to simultaneously encourage sparsity and maintain training stability. Effective adaptive optimizers exhibit an implicit L_∞ bias favoring stability, yet, sparsity requires an L_1 bias. To integrate sparsity, we propose a composition of optimizer steps, which we cast as non-commutative operators to analyze and combine their optimization geometry in a principled way. This yields HORST (Hyperbolic Operator for Robust Sparse Training), a modular optimizer that inherits stability from adaptive methods while inducing L_1 sparsity bias through a hyperbolic mirror map. Our experiments demonstrate its utility for sparse training of transformers on both vision and language tasks. HORST consistently and significantly outperforms AdamW baselines across all sparsity levels, with large gains at higher sparsity.
@inproceedings{jacobs2026horstcomposingoptimizergeometries, title = {HORST: Composing Optimizer Geometries for Sparse Transformer Training}, author = {Jacobs, Tom and Jain, Rohan and Burkholz, Rebekka}, booktitle = {Workshop on High-dimensional Learning Dynamics (HiLD) @ ICML}, year = {2026}, url = {https://arxiv.org/abs/2605.21104}, }
Continuous Sparsification via Minimizing Movement

Hoang Pham, Tom Jacobs, Binh Nguyen, Rebekka Burkholz, and Long Tran-Thanh

In Workshop on High-dimensional Learning Dynamics (HiLD) @ ICML, 2026

Bib Abs

Continuous sparsification is a popular approach for finding efficient sparse subnetworks, optimizing soft masks before applying a hard Top-K projection. However, existing methods typically regularize mask variables using flat Euclidean geometry, penalizing local rewirings and disruptive long-range relocations equally. To address this, we propose Continuous Sparsification via Minimizing Movement (CSMM), a geometry-aware framework that treats layer connectivity as a probability allocation on the simplex. By leveraging the minimizing-movement scheme, we regularize the temporal evolution of connectivity using flexible proximal penalty functions. This approach decouples task performance from structural evolution, allowing practitioners to impose specific geometric inductive biases on topology evolution. The experiment shows a promising result on CIFAR10 dataset under ResNet-20 architecture, offering a new approach on continuous sparsification.
@inproceedings{pham2026continuous, title = {Continuous Sparsification via Minimizing Movement}, author = {Pham, Hoang and Jacobs, Tom and Nguyen, Binh and Burkholz, Rebekka and Tran-Thanh, Long}, booktitle = {Workshop on High-dimensional Learning Dynamics (HiLD) @ ICML}, year = {2026}, url = {https://openreview.net/forum?id=nsSLDnJ2uU}, }
Fixed Aggregation Features Can Rival GNNs

Celia Rubio-Madrigal, and Rebekka Burkholz

In Forty-third International Conference on Machine Learning, 2026

ICML Bib Abs Code

Graph neural networks (GNNs) are widely believed to excel at node representation learning through trainable neighborhood aggregations. We challenge this view by introducing Fixed Aggregation Features (FAFs), a training-free approach that transforms graph learning tasks into tabular problems. This simple shift enables the use of well-established tabular methods, offering strong interpretability and the flexibility to deploy diverse classifiers. Across 14 benchmarks, well-tuned multilayer perceptrons trained on FAFs rival or outperform state-of-the-art GNNs and graph transformers on 12 tasks – often using only mean aggregation. The only exceptions are the Roman Empire and Minesweeper datasets, which typically require unusually deep GNNs. To explain the theoretical possibility of non-trainable aggregations, we connect our findings to Kolmogorov–Arnold representations and discuss when mean aggregation can be sufficient. In conclusion, our results call for (i) richer benchmarks benefiting from learning diverse neighborhood aggregations, (ii) strong tabular baselines as standard, and (iii) employing and advancing tabular models for graph data to gain new insights into related tasks.
@inproceedings{rubio-madrigal2026fixed, title = {Fixed Aggregation Features Can Rival {GNN}s}, author = {Rubio-Madrigal, Celia and Burkholz, Rebekka}, booktitle = {Forty-third International Conference on Machine Learning}, year = {2026}, url = {https://openreview.net/forum?id=gSZhNPp103}, }
Robustness of Mixtures of Experts to Feature Noise

Dong Sun, Rahul Nittala, and Rebekka Burkholz

In Forty-third International Conference on Machine Learning, 2026

ICML Bib Abs

Despite their practical success, it remains unclear why Mixture of Experts (MoE) models can outperform dense networks beyond sheer parameter scaling. We study an iso-parameter regime where inputs exhibit latent modular structure but are corrupted by feature noise, a proxy for noisy internal activations. We show that sparse expert activation acts as a noise filter: compared to a dense estimator, MoEs achieve lower generalization error under feature noise, improved robustness to perturbations, and faster convergence speed. Empirical results on synthetic data and real-world language tasks corroborate the theoretical insights, demonstrating consistent robustness and efficiency gains from sparse modular computation.
@inproceedings{sun2026robustnessmixturesexpertsfeature, title = {Robustness of Mixtures of Experts to Feature Noise}, author = {Sun, Dong and Nittala, Rahul and Burkholz, Rebekka}, year = {2026}, booktitle = {Forty-third International Conference on Machine Learning}, url = {https://arxiv.org/abs/2601.14792}, }
SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training

Mohammed Adnan, Rohan Jain, Tom Jacobs, Ekansh Sharma, Rahul G Krishnan, Rebekka Burkholz, and Yani Ioannou

In Forty-third International Conference on Machine Learning, 2026

ICML Bib Abs

Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar accuracy. We demonstrate both analytically and empirically that Batch Normalization (BN) adversely affects sparse training and propose SparseOpt — a sparsity-aware optimizer— to address this. Experiments on ResNet models across CIFAR-100 and ImageNet demonstrate consistently faster convergence and improved generalization with our proposed method. Our work highlights the limitations of current normalization layers in sparse training and provides the first systematic study of the interaction between Batch Normalization, sparse layers, and DST, taking a significant step toward making DST practically competitive with dense training.
@inproceedings{adnan2026sparseopt, title = {SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training}, author = {Adnan, Mohammed and Jain, Rohan and Jacobs, Tom and Sharma, Ekansh and Krishnan, Rahul G and Burkholz, Rebekka and Ioannou, Yani}, booktitle = {Forty-third International Conference on Machine Learning}, year = {2026}, url = {https://openreview.net/forum?id=qerVUczDMf}, }
Sex differences in gene regulation and its impact on cancer incidence

Camila M. Lopes-Ramos, Rebekka Burkholz, Marouen Ben Guebila, Viola Fanfani, Enakshi Saha, Katherine H. Shutta, Kimberly Glass, John Quackenbush, and Dawn L. DeMeo

iScience, May 2026

Bib Abs

There are significant sex differences in cancer incidence, yet the underlying regulatory mechanisms in normal tissues remain poorly understood. We studied 8,279 gene regulatory networks across 29 non-cancerous tissues and compared network centrality by sex. Cancer genes were differentially targeted by transcription factors in males and females, with an overrepresentation on the X chromosome, particularly among X-inactivation escapees, and key signaling pathways such as WNT, NOTCH, and p53. We observed higher targeting of cancer-related pathways in females for tissues that have higher tumor incidence in females (breast, lung, and thyroid) and higher targeting in males for tissues with increased tumor incidence in males (stomach, colon, and liver), a pattern replicated in independent lung data. Sex-biased transcription factors were enriched for sex hormone response elements. These findings suggest that sex-biased transcriptional programs in normal tissues contribute to sex differences in cancer incidence and should be considered in cancer prevention strategies.
@article{Lopes-Ramos2026, author = {Lopes-Ramos, Camila M. and Burkholz, Rebekka and Ben Guebila, Marouen and Fanfani, Viola and Saha, Enakshi and Shutta, Katherine H. and Glass, Kimberly and Quackenbush, John and DeMeo, Dawn L.}, title = {Sex differences in gene regulation and its impact on cancer incidence}, journal = {iScience}, year = {2026}, month = may, day = {15}, publisher = {Elsevier}, volume = {29}, number = {5}, issn = {2589-0042}, doi = {10.1016/j.isci.2026.115570}, url = {https://doi.org/10.1016/j.isci.2026.115570}, }
Bridging Domains through Subspace-Aware Model Merging

Levy Chaves, Chao Zhou, Rebekka Burkholz, Eduardo Valle, and Andra Avila

In IEEE Conference on Computer Vision and Pattern Recognition, 2026

CVPR Bib Abs

Model merging integrates multiple task-specific models into a single consolidated one. Recent research has made progress in improving merging performance for in-distribution or multi-task scenarios, but domain generalization in model merging remains underexplored. We investigate how merging models fine-tuned on distinct domains affects generalization to unseen domains. Through an analysis of parameter competition in the task matrix using singular value decomposition, we show that merging models trained under different distribution shifts induces stronger conflicts between their subspaces compared to traditional multi-task settings. To mitigate this issue, we propose SCORE (Subspace COnflict-Resolving mErging), a method designed to alleviate such singular subspace conflicts. SCORE finds a shared orthogonal basis by computing the principal components of the concatenated leading singular vectors of all models. It then projects each task matrix into the shared basis, pruning off-diagonal components to remove conflicting singular directions. SCORE consistently outperforms, on average, existing model merging approaches in domain generalization settings across a variety of architectures and model scales, demonstrating its effectiveness and scalability.
@inproceedings{merge, title = {Bridging Domains through Subspace-Aware Model Merging}, author = {Chaves, Levy and Zhou, Chao and Burkholz, Rebekka and Valle, Eduardo and Avila, Andra}, year = {2026}, booktitle = {IEEE Conference on Computer Vision and Pattern Recognition}, url = {https://arxiv.org/abs/2603.05768}, }
Frequency-Based Hyperparameter Selection in Games

Aniket Sanyal, Baraah A. M. Sidahmed, Rebekka Burkholz, and Tatjana Chavdarova

In International Conference on Artificial Intelligence and Statistics, 2026

AISTATS Bib Abs

Learning in smooth games fundamentally differs from standard minimization due to rotational dynamics, which invalidate classical hyperparameter tuning strategies. Despite their practical importance, effective methods for tuning in games remain underexplored. A notable example is LookAhead (LA), which achieves strong empirical performance but introduces additional parameters that critically influence performance. We propose a principled approach to hyperparameter selection in games by leveraging frequency estimation of oscillatory dynamics. Specifically, we analyze oscillations both in continuous-time trajectories and through the spectrum of the discrete dynamics in the associated frequency-based space. Building on this analysis, we introduce Modal LookAhead (MoLA), an extension of LA that selects the hyperparameters adaptively to a given problem. We provide convergence guarantees and demonstrate in experiments that MoLA accelerates training in both purely rotational games and mixed regimes, all with minimal computational overhead.
@inproceedings{sanyal2026games, title = {Frequency-Based Hyperparameter Selection in Games}, author = {Sanyal, Aniket and Sidahmed, Baraah A. M. and Burkholz, Rebekka and Chavdarova, Tatjana}, year = {2026}, booktitle = {International Conference on Artificial Intelligence and Statistics}, url = {https://arxiv.org/abs/2601.18409}, }
Boosting for Predictive Sufficiency

Abbavaram Gowtham Reddy, Rajeev Verma, Celia Rubio-Madrigal, Krikamol Muandet, and Rebekka Burkholz

In The Fourteenth International Conference on Learning Representations, 2026

ICLR Bib Abs

Out-of-distribution (OOD) generalization is a defining hallmark of truly robust and reliable machine learning systems. Recently, it has been empirically observed that existing OOD generalization methods often underperform on real-world tabular data, where hidden confounding shifts drive distribution changes that boosting models handle more effectively. Part of boosting’s success is attributed to variance reduction, handling missing variables, feature selection, and connections to multicalibration. This paper uncovers a crucial reason behind its success in OOD generalization: boosting’s ability to infer stable environments robust to hidden confounding shifts and maximize predictive performance within those environments. This paper introduces an information-theoretic notion called α-predictive sufficiency and formalizes its link to OOD generalization under hidden confounding. We show that boosting implicitly identifies suitable environments and produces an α-predictive sufficient predictor. We validate our theoretical results through synthetic and real-world experiments and show that boosting achieves robust performance by identifying these environments and maximizing the association between predictions and true outcomes.
@inproceedings{reddy2026boosting, title = {Boosting for Predictive Sufficiency}, author = {Reddy, Abbavaram Gowtham and Verma, Rajeev and Rubio-Madrigal, Celia and Muandet, Krikamol and Burkholz, Rebekka}, booktitle = {The Fourteenth International Conference on Learning Representations}, year = {2026}, url = {https://openreview.net/forum?id=1mQT8PXIy8}, }
When Shift Happens - Confounding Is to Blame

Abbavaram Gowtham Reddy, Celia Rubio-Madrigal, Rebekka Burkholz, and Krikamol Muandet

In The Fourteenth International Conference on Learning Representations, 2026

ICLR Bib Abs

Distribution shifts introduce uncertainty that undermines the robustness and generalization capabilities of machine learning models. While conventional wisdom suggests that learning causal-invariant representations enhances robustness to such shifts, recent empirical studies present a counterintuitive finding: (i) empirical risk minimization (ERM) can rival or even outperform state-of-the-art out-of-distribution (OOD) generalization methods, and (ii) OOD generalization performance improves when all available covariates, including non-causal ones, are utilized. We present theoretical and empirical explanations that attribute this phenomenon to hidden confounding. Shifts in hidden confounding induce changes in data distributions that violate assumptions commonly made by existing approaches. Under such conditions, we prove that generalization requires learning environment-specific relationships, rather than relying solely on invariant ones. Furthermore, we explain why models augmented with non-causal but informative covariates can mitigate the challenges posed by hidden confounding shifts. These findings offer new theoretical insights and practical guidance, serving as a roadmap for future research on OOD generalization and principled covariate-selection strategies.
@inproceedings{reddy2026when, title = {When Shift Happens - Confounding Is to Blame}, author = {Reddy, Abbavaram Gowtham and Rubio-Madrigal, Celia and Burkholz, Rebekka and Muandet, Krikamol}, booktitle = {The Fourteenth International Conference on Learning Representations}, year = {2026}, url = {https://openreview.net/forum?id=sFjxg8cyJS}, }
Never Saddle for Reparameterized Steepest Descent as Mirror Flow

Tom Jacobs, Chao Zhou, and Rebekka Burkholz

In The Fourteenth International Conference on Learning Representations, 2026

ICLR Bib Abs

How does the choice of optimization algorithm shape a model’s ability to learn features? To address this question for steepest descent methods —including sign descent, which is closely related to Adam —we introduce steepest mirror flows as a unifying theoretical framework. This framework reveals how optimization geometry governs learning dynamics, implicit bias, and sparsity and it provides two explanations for why Adam and AdamW often outperform SGD in fine-tuning. Focusing on diagonal linear networks and deep diagonal linear reparameterizations (a simplified proxy for attention), we show that steeper descent facilitates both saddle-point escape and feature learning. In contrast, gradient descent requires unrealistically large learning rates to escape saddles, an uncommon regime in fine-tuning. Empirically, we confirm that saddle-point escape is a central challenge in fine-tuning. Furthermore, we demonstrate that decoupled weight decay, as in AdamW, stabilizes feature learning by enforcing novel balance equations. Together, these results highlight two mechanisms how steepest descent can aid modern optimization.
@inproceedings{jacobs2026never, title = {Never Saddle for Reparameterized Steepest Descent as Mirror Flow}, author = {Jacobs, Tom and Zhou, Chao and Burkholz, Rebekka}, booktitle = {The Fourteenth International Conference on Learning Representations}, year = {2026}, url = {https://openreview.net/forum?id=YgudIlQ9nC}, }
Hyperbolic Aware Minimization: Implicit Bias for Sparsity

Tom Jacobs, Advait Gadhikar, Celia Rubio-Madrigal, and Rebekka Burkholz

In The Fourteenth International Conference on Learning Representations, 2026

ICLR Bib Abs

Understanding the implicit bias of optimization algorithms is key to explaining and improving the generalization of deep models. The hyperbolic implicit bias induced by pointwise overparameterization promotes sparsity, but also yields a small inverse Riemannian metric near zero, slowing down parameter movement and impeding meaningful parameter sign flips. To overcome this obstacle, we propose Hyperbolic Aware Minimization (HAM), which alternates a standard optimizer step with a lightweight hyperbolic mirror step. The mirror step incurs less compute and memory than pointwise overparameterization, reproduces its beneficial hyperbolic geometry for feature learning, and mitigates the small–inverse-metric bottleneck. Our characterization of the implicit bias in the context of underdetermined linear regression provides insights into the mechanism how HAM consistently increases performance –even in the case of dense training, as we demonstrate in experiments with standard vision benchmarks. HAM is especially effective in combination with different sparsification methods, advancing the state of the art.
@inproceedings{jacobs2026hyperbolic, title = {Hyperbolic Aware Minimization: Implicit Bias for Sparsity}, author = {Jacobs, Tom and Gadhikar, Advait and Rubio-Madrigal, Celia and Burkholz, Rebekka}, booktitle = {The Fourteenth International Conference on Learning Representations}, year = {2026}, url = {https://openreview.net/forum?id=XKB5Hu0ACY}, }
OptRot: Mitigating Weight Outliers via Data-Free Rotations for Post-Training Quantization

Advait Gadhikar, Riccardo Grazzi, and James Hensman

In Workshop on Machine Learning for Systems (MLForSys) @ NeurIPS, 2025

Bib Abs

We introduce OptRot, a data-free preprocessing method to learn fusible rotations for post-training quantization of language models. OptRot reduces weight outliers by finding rotations which minimize the element-wise fourth power of the rotated weights. We show how reducing weight outliers can provably improve weight quantization performance and how OptRot rotations can outperform both Hadamard rotations and rotations learned by the data-dependent method SpinQuant.
@inproceedings{gadhikar2025optrot, title = {OptRot: Mitigating Weight Outliers via Data-Free Rotations for Post-Training Quantization}, author = {Gadhikar, Advait and Grazzi, Riccardo and Hensman, James}, booktitle = {Workshop on Machine Learning for Systems (MLForSys) @ NeurIPS}, year = {2025}, url = {https://openreview.net/forum?id=4uwRaBhjHY}, }
Pay Attention to Small Weights

Chao Zhou, Tom Jacobs, Advait Gadhikar, and Rebekka Burkholz

In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

NeurIPS Bib Abs

Finetuning large pretrained neural networks is known to be resource-intensive, both in terms of memory and computational cost. To mitigate this, a common approach is to restrict training to a subset of the model parameters. By analyzing the relationship between gradients and weights during finetuning, we observe a notable pattern: large gradients are often associated with small-magnitude weights. This correlation is more pronounced in finetuning settings than in training from scratch. Motivated by this observation, we propose NANOADAM, which dynamically updates only the small-magnitude weights during finetuning and offers several practical advantages: first, this criterion is gradient-free – the parameter subset can be determined without gradient computation; second, it preserves large-magnitude weights, which are likely to encode critical features learned during pretraining, thereby reducing the risk of catastrophic forgetting; thirdly, it permits the use of larger learning rates and consistently leads to better generalization performance in experiments. We demonstrate this for both NLP and vision tasks.
@inproceedings{zhou2025payattentionsmallweights, title = {Pay Attention to Small Weights}, author = {Zhou, Chao and Jacobs, Tom and Gadhikar, Advait and Burkholz, Rebekka}, booktitle = {The Thirty-ninth Annual Conference on Neural Information Processing Systems}, year = {2025}, url = {https://openreview.net/forum?id=XKnOA7MhCz}, }
Sign-In to the Lottery: Reparameterizing Sparse Training

Advait Gadhikar, Tom Jacobs, Chao Zhou, and Rebekka Burkholz

In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

NeurIPS Bib Abs

The performance gap between training sparse neural networks from scratch (PaI) and dense-to-sparse training presents a major roadblock for efficient deep learning. According to the Lottery Ticket Hypothesis, PaI hinges on finding a problem specific parameter initialization. As we show, to this end, determining correct parameter signs is sufficient. Yet, they remain elusive to PaI. To address this issue, we propose Sign-In, which employs a dynamic reparameterization that provably induces sign flips. Such sign flips are complementary to the ones that dense-to-sparse training can accomplish, rendering Sign-In as an orthogonal method. While our experiments and theory suggest performance improvements of PaI, they also carve out the main open challenge to close the gap between PaI and dense-to-sparse training.
@inproceedings{Gadhikar2025SignInTT, title = {Sign-In to the Lottery: Reparameterizing Sparse Training}, author = {Gadhikar, Advait and Jacobs, Tom and Zhou, Chao and Burkholz, Rebekka}, booktitle = {The Thirty-ninth Annual Conference on Neural Information Processing Systems}, year = {2025}, url = {https://openreview.net/forum?id=iwKT7MEZZw}, }
The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis

Hoang Pham, The-Anh Ta, Tom Jacobs, Rebekka Burkholz, and Long Tran-Thanh

In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025

NeurIPS Spotlight Bib Abs

Sparse neural networks promise efficiency, yet training them effectively remains a fundamental challenge. Despite advances in pruning methods that create sparse architectures, understanding why some sparse structures are better trainable than others with the same level of sparsity remains poorly understood. Aiming to develop a systematic approach to this fundamental problem, we propose a novel theoretical framework based on the theory of graph limits, particularly graphons, that characterizes sparse neural networks in the infinite-width regime. Our key insight is that connectivity patterns of sparse neural networks induced by pruning methods converge to specific graphons as networks’ width tends to infinity, which encodes implicit structural biases of different pruning methods. We postulate the Graphon Limit Hypothesis and provide empirical evidence to support it. Leveraging this graphon representation, we derive a Graphon Neural Tangent Kernel (Graphon NTK) to study the training dynamics of sparse networks in the infinite width limit. Graphon NTK provides a general framework for the theoretical analysis of sparse networks. We empirically show that the spectral analysis of Graphon NTK correlates with observed training dynamics of sparse networks, explaining the varying convergence behaviours of different pruning methods. Our framework provides theoretical insights into the impact of connectivity patterns on the trainability of various sparse network architectures.
@inproceedings{pham2025the, title = {The Graphon Limit Hypothesis: Understanding Neural Network Pruning via Infinite Width Analysis}, author = {Pham, Hoang and Ta, The-Anh and Jacobs, Tom and Burkholz, Rebekka and Tran-Thanh, Long}, booktitle = {The Thirty-ninth Annual Conference on Neural Information Processing Systems}, year = {2025}, url = {https://openreview.net/forum?id=EEZLBhyer1}, }
Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?

Tom Jacobs, Chao Zhou, and Rebekka Burkholz

In Forty-second International Conference on Machine Learning, 2025

ICML Bib Abs

Implicit bias plays an important role in explaining how overparameterized models generalize well. Explicit regularization like weight decay is often employed in addition to prevent overfitting. While both concepts have been studied separately, in practice, they often act in tandem. Understanding their interplay is key to controlling the shape and strength of implicit bias, as it can be modified by explicit regularization. To this end, we incorporate explicit regularization into the mirror flow framework and analyze its lasting effects on the geometry of the training dynamics, covering three distinct effects: positional bias, type of bias, and range shrinking. Our analytical approach encompasses a broad class of problems, including sparse coding, matrix sensing, single-layer attention, and LoRA, for which we demonstrate the utility of our insights. To exploit the lasting effect of regularization and highlight the potential benefit of dynamic weight decay schedules, we propose to switch off weight decay during training, which can improve generalization, as we demonstrate in experiments.
@inproceedings{jacobs2025mirror, title = {Mirror, Mirror of the Flow: How Does Regularization Shape Implicit Bias?}, author = {Jacobs, Tom and Zhou, Chao and Burkholz, Rebekka}, booktitle = {Forty-second International Conference on Machine Learning}, year = {2025}, url = {https://openreview.net/forum?id=MLiR9LS5PW}, }
Attention Is All You Need For Mixture-of-Depths Routing

Advait Gadhikar, Souptik Kumar Majumdar, Niclas Popp, Piyapat Saranrittichai, Martin Rapp, and Lukas Schott

In Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models (SCOPE) @ ICLR, 2025

Bib Abs

Advancements in deep learning are driven by training models with increasingly larger numbers of parameters, which in turn heightens the computational demands. To address this issue, Mixture-of-Depths (MoD) models have been proposed to dynamically focus computations on the most relevant parts of the inputs, thereby enabling the deployment of large-parameter models with high efficiency during inference and training. However, conventional MoD models employ additional network layers specifically for the routing which are difficult to train, and add complexity to the model. In this paper, we introduce a novel attention-based routing mechanism A-MoD that leverages the existing attention map of the preceding layer for routing decisions within the current layer. Compared to standard routing, A-MoD allows for more efficient training as it introduces no additional trainable parameters and can be easily adapted from pre-trained transformer models. Furthermore, it can increase the performance of the MoD model. For instance, we observe up to 2% higher accuracy on ImageNet compared to standard routing and isoFLOP ViT baselines.
@inproceedings{gadhikar2025attention, title = {Attention Is All You Need For Mixture-of-Depths Routing}, author = {Gadhikar, Advait and Majumdar, Souptik Kumar and Popp, Niclas and Saranrittichai, Piyapat and Rapp, Martin and Schott, Lukas}, booktitle = {Workshop on Scalable Optimization for Efficient and Adaptive Foundation Models (SCOPE) @ ICLR}, year = {2025}, url = {https://openreview.net/forum?id=1uDP4ld3eZ}, }
Mask in the Mirror: Implicit Sparsification

Tom Jacobs, and Rebekka Burkholz

In The Thirteenth International Conference on Learning Representations, 2025

ICLR Bib Abs

Continuous sparsification strategies are among the most effective methods for reducing the inference costs and memory demands of large-scale neural networks. A key factor in their success is the implicit L1 regularization induced by jointly learning both mask and weight variables, which has been shown experimentally to outperform explicit L1 regularization. We provide a theoretical explanation for this observation by analyzing the learning dynamics, revealing that early continuous sparsification is governed by an implicit L2 regularization that gradually transitions to an L1 penalty over time. Leveraging this insight, we propose a method to dynamically control the strength of this implicit bias. Through an extension of the mirror flow framework, we establish convergence and optimality guarantees in the context of underdetermined linear regression. Our theoretical findings may be of independent interest, as we demonstrate how to enter the rich regime and show that the implicit bias can be controlled via a time-dependent Bregman potential. To validate these insights, we introduce PILoT, a continuous sparsification approach with novel initialization and dynamic regularization, which consistently outperforms baselines in standard experiments.
@inproceedings{jacobs2025mask, title = {Mask in the Mirror: Implicit Sparsification}, author = {Jacobs, Tom and Burkholz, Rebekka}, booktitle = {The Thirteenth International Conference on Learning Representations}, year = {2025}, url = {https://openreview.net/forum?id=U47ymTS3ut}, }
GNNs Getting ComFy: Community and Feature Similarity Guided Rewiring

Celia Rubio-Madrigal, Adarsh Jamadandi, and Rebekka Burkholz

In The Thirteenth International Conference on Learning Representations, 2025

ICLR Bib Abs Code

Maximizing the spectral gap through graph rewiring has been proposed to enhance the performance of message-passing graph neural networks (GNNs) by addressing over-squashing. However, as we show, minimizing the spectral gap can also improve generalization. To explain this, we analyze how rewiring can benefit GNNs within the context of stochastic block models. Since spectral gap optimization primarily influences community strength, it improves performance when the community structure aligns with node labels. Building on this insight, we propose three distinct rewiring strategies that explicitly target community structure, node labels, and their alignment: (a) community structure-based rewiring (ComMa), a more computationally efficient alternative to spectral gap optimization that achieves similar goals; (b) feature similarity-based rewiring (FeaSt), which focuses on maximizing global homophily; and (c) a hybrid approach (ComFy), which enhances local feature similarity while preserving community structure to optimize label-community alignment. Extensive experiments confirm the effectiveness of these strategies and support our theoretical insights.
@inproceedings{rubio-madrigal2025gnns, title = {{GNN}s Getting ComFy: Community and Feature Similarity Guided Rewiring}, author = {Rubio-Madrigal, Celia and Jamadandi, Adarsh and Burkholz, Rebekka}, booktitle = {The Thirteenth International Conference on Learning Representations}, year = {2025}, url = {https://openreview.net/forum?id=g6v09VxgFw}, }
Spectral Graph Pruning Against Over-Squashing and Over-Smoothing

Adarsh Jamadandi, Celia Rubio-Madrigal, and Rebekka Burkholz

In Thirty-eighth Conference on Neural Information Processing Systems, 2024

NeurIPS Bib Abs Code

Message Passing Graph Neural Networks are known to suffer from two problems that are sometimes believed to be diametrically opposed: over-squashing and over-smoothing. The former results from topological bottlenecks that hamper the information flow from distant nodes and are mitigated by spectral gap maximization, primarily, by means of edge additions. However, such additions often promote over-smoothing that renders nodes of different classes less distinguishable. Inspired by the Braess phenomenon, we argue that deleting edges can address over-squashing and over-smoothing simultaneously. This insight explains how edge deletions can improve generalization, thus connecting spectral gap optimization to a seemingly disconnected objective of reducing computational resources by pruning graphs for lottery tickets. To this end, we propose a computationally effective spectral gap optimization framework to add or delete edges and demonstrate its effectiveness on the long range graph benchmark and on larger heterophilous datasets.
@inproceedings{jamadandi2024spectral, title = {Spectral Graph Pruning Against Over-Squashing and Over-Smoothing}, author = {Jamadandi, Adarsh and Rubio-Madrigal, Celia and Burkholz, Rebekka}, booktitle = {Thirty-eighth Conference on Neural Information Processing Systems}, year = {2024}, url = {https://openreview.net/forum?id=EMkrwJY2de}, }
Dynamic Rescaling for Training GNNs

Nimrah Mustafa, and Rebekka Burkholz

In Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

NeurIPS Bib Abs Code

Graph neural networks (GNNs) with a rescale invariance, such as GATs, can be re-parameterized during optimization through dynamic rescaling of network parameters and gradients while keeping the loss invariant. In this work, we explore dynamic rescaling as a tool to influence GNN training dynamics in two key ways: i) balancing the network with respect to various criteria, and ii) controlling the relative learning speeds of different layers. We gain novel insights, unique to GNNs, that reveal distinct training modes for different tasks. For heterophilic graphs, achieving balance based on relative gradients leads to faster training and better generalization. In contrast, homophilic graphs benefit from delaying the learning of later layers. Additionally, we show that training in balance supports larger learning rates, which can improve generalization. Moreover, controlling layer-wise training speeds is linked to grokking-like phenomena, which may be of independent interest.
@inproceedings{mustafa2024dynamic, title = {Dynamic Rescaling for Training {GNN}s}, author = {Mustafa, Nimrah and Burkholz, Rebekka}, booktitle = {Thirty-eighth Annual Conference on Neural Information Processing Systems}, year = {2024}, url = {https://openreview.net/forum?id=IfZwSRpqHl}, }
Pruning neural network models for gene regulatory dynamics using data and domain knowledge

Intekhab Hossain, Jonas Fischer, Rebekka Burkholz, and John Quackenbush

In Thirty-eighth Conference on Neural Information Processing Systems, 2024

NeurIPS Bib Abs

The practical utility of machine learning models in the sciences often hinges on their interpretability. It is common to assess a model’s merit for scientific discovery, and thus novel insights, by how well it aligns with already available domain knowledge - a dimension that is currently largely disregarded in the comparison of neural network models. While pruning can simplify deep neural network architectures and excels in identifying sparse models, as we show in the context of gene regulatory network inference, state-of-the-art techniques struggle with biologically meaningful structure learning. To address this issue, we propose DASH, a generalizable framework that guides network pruning by using domain-specific structural information in model fitting and leads to sparser, better interpretable models that are more robust to noise. Using both synthetic data with ground truth information, as well as real-world gene expression data, we show that DASH, using knowledge about gene interaction partners within the putative regulatory network, outperforms general pruning methods by a large margin and yields deeper insights into the biological systems being studied.
@inproceedings{hossain2024pruning, title = {Pruning neural network models for gene regulatory dynamics using data and domain knowledge}, author = {Hossain, Intekhab and Fischer, Jonas and Burkholz, Rebekka and Quackenbush, John}, booktitle = {Thirty-eighth Conference on Neural Information Processing Systems}, year = {2024}, url = {https://openreview.net/forum?id=FNtsZLwkGr}, }
Biologically informed NeuralODEs for genome-wide regulatory dynamics

Intekhab Hossain, Viola Fanfani, Jonas Fischer, John Quackenbush, and Rebekka Burkholz

Genome Biology, May 2024

Bib Abs

Gene regulatory network (GRN) models that are formulated as ordinary differential equations (ODEs) can accurately explain temporal gene expression patterns and promise to yield new insights into important cellular processes, disease progression, and intervention design. Learning such gene regulatory ODEs is challenging, since we want to predict the evolution of gene expression in a way that accurately encodes the underlying GRN governing the dynamics and the nonlinear functional relationships between genes. Most widely used ODE estimation methods either impose too many parametric restrictions or are not guided by meaningful biological insights, both of which impede either scalability, explainability, or both.
@article{hossain2024biologically, author = {Hossain, Intekhab and Fanfani, Viola and Fischer, Jonas and Quackenbush, John and Burkholz, Rebekka}, title = {Biologically informed NeuralODEs for genome-wide regulatory dynamics}, journal = {Genome Biology}, year = {2024}, month = may, volume = {25}, url = {https://doi.org/10.1186/s13059-024-03264-0}, }

Edge-LLMs: Edge-Device Large Language Model Competition

Shiwei Liu, Kai Han, Adriana Fernandez-Lopez, Ajay Kumar Jaiswal, Zahra Atashgahi, Boqian Wu, Edoardo Ponti, Callie Hao, Rebekka Burkholz, Olga Saukh, Lu Yin, Andreas Zinonos, Tianjin Huang, Jared Tanner, and Yunhe Wang

In NeurIPS 2024 Competition Track, 2024

Bib

@inproceedings{liu2024edgellms,
  title = {Edge-{LLM}s: Edge-Device Large Language Model Competition},
  author = {Liu, Shiwei and Han, Kai and Fernandez-Lopez, Adriana and Jaiswal, Ajay Kumar and Atashgahi, Zahra and Wu, Boqian and Ponti, Edoardo and Hao, Callie and Burkholz, Rebekka and Saukh, Olga and Yin, Lu and Zinonos, Andreas and Huang, Tianjin and Tanner, Jared and Wang, Yunhe},
  booktitle = {NeurIPS 2024 Competition Track},
  year = {2024},
  url = {https://openreview.net/forum?id=jeCMRoIn15},
}

GATE: How to Keep Out Intrusive Neighbors

Nimrah Mustafa, and Rebekka Burkholz

In Forty-first International Conference on Machine Learning, 2024

ICML Bib Abs Code

Graph Attention Networks (GATs) are designed to provide flexible neighborhood aggregation that assigns weights to neighbors according to their importance. In practice, however, GATs are often unable to switch off task-irrelevant neighborhood aggregation, as we show experimentally and analytically. To address this challenge, we propose GATE, a GAT extension that holds three major advantages: i) It alleviates over-smoothing by addressing its root cause of unnecessary neighborhood aggregation. ii) Similarly to perceptrons, it benefits from higher depth as it can still utilize additional layers for (non-)linear feature transformations in case of (nearly) switched-off neighborhood aggregation. iii) By down-weighting connections to unrelated neighbors, it often outperforms GATs on real-world heterophilic datasets. To further validate our claims, we construct a synthetic test bed to analyze a model’s ability to utilize the appropriate amount of neighborhood aggregation, which could be of independent interest.
@inproceedings{mustafa2024gate, title = {{GATE}: How to Keep Out Intrusive Neighbors}, author = {Mustafa, Nimrah and Burkholz, Rebekka}, booktitle = {Forty-first International Conference on Machine Learning}, year = {2024}, url = {https://openreview.net/forum?id=Sjv5RcqfuH}, }
Masks, Signs, And Learning Rate Rewinding

Advait Harshal Gadhikar, and Rebekka Burkholz

In The Twelfth International Conference on Learning Representations, 2024

ICLR Spotlight Bib Abs

Learning Rate Rewinding (LRR) has been established as a strong variant of Iterative Magnitude Pruning (IMP) to find lottery tickets in deep overparameterized neural networks. While both iterative pruning schemes couple structure and parameter learning, understanding how LRR excels in both aspects can bring us closer to the design of more flexible deep learning algorithms that can optimize diverse sets of sparse architectures. To this end, we conduct experiments that disentangle the effect of mask learning and parameter optimization and how both benefit from overparameterization. The ability of LRR to flip parameter signs early and stay robust to sign perturbations seems to make it not only more effective in mask identification but also in optimizing diverse sets of masks, including random ones. In support of this hypothesis, we prove in a simplified single hidden neuron setting that LRR succeeds in more cases than IMP, as it can escape initially problematic sign configurations.
@inproceedings{gadhikar2024masks, title = {Masks, Signs, And Learning Rate Rewinding}, author = {Gadhikar, Advait Harshal and Burkholz, Rebekka}, booktitle = {The Twelfth International Conference on Learning Representations}, year = {2024}, url = {https://openreview.net/forum?id=qODvxQ8TXW}, }
Batch normalization is sufficient for universal function approximation in CNNs

Rebekka Burkholz

In The Twelfth International Conference on Learning Representations, 2024

ICLR Bib Abs

Layer normalization, for which Batch Normalization (BN) is a popular choice, is an integral part of many deep learning architectures and contributes significantly to the learning success. We provide a partial explanation for this phenomenon by proving that training normalization layers alone is already sufficient for universal function approximation if the number of available, potentially random features matches or exceeds the weight parameters of the target networks that can be expressed. Our bound on the number of required features does not only improve on a recent result for fully-connected feed-forward architectures but also applies to CNNs with and without residual connections and almost arbitrary activation functions (which include ReLUs). Our explicit construction of a given target network solves a depth-width trade-off that is driven by architectural constraints and can explain why switching off entire neurons can have representational benefits, as has been observed empirically. To validate our theory, we explicitly match target networks that outperform experimentally obtained networks with trained BN parameters by utilizing a sufficient number of random features.
@inproceedings{burkholz2024batch, title = {Batch normalization is sufficient for universal function approximation in {CNN}s}, author = {Burkholz, Rebekka}, booktitle = {The Twelfth International Conference on Learning Representations}, year = {2024}, url = {https://openreview.net/forum?id=wOSYMHfENq}, }
Are GATs Out of Balance?

Nimrah Mustafa, Aleksandar Bojchevski, and Rebekka Burkholz

In Thirty-seventh Conference on Neural Information Processing Systems, 2023

NeurIPS Bib Abs Code

While the expressive power and computational capabilities of graph neural networks (GNNs) have been theoretically studied, their optimization and learning dynamics, in general, remain largely unexplored. Our study undertakes the Graph Attention Network (GAT), a popular GNN architecture in which a node’s neighborhood aggregation is weighted by parameterized attention coefficients. We derive a conservation law of GAT gradient flow dynamics, which explains why a high portion of parameters in GATs with standard initialization struggle to change during training. This effect is amplified in deeper GATs, which perform significantly worse than their shallow counterparts. To alleviate this problem, we devise an initialization scheme that balances the GAT network. Our approach i) allows more effective propagation of gradients and in turn enables trainability of deeper networks, and ii) attains a considerable speedup in training and convergence time in comparison to the standard initialization. Our main theorem serves as a stepping stone to studying the learning dynamics of positive homogeneous models with attention mechanisms.
@inproceedings{mustafa2023are, title = {Are {GAT}s Out of Balance?}, author = {Mustafa, Nimrah and Bojchevski, Aleksandar and Burkholz, Rebekka}, booktitle = {Thirty-seventh Conference on Neural Information Processing Systems}, year = {2023}, url = {https://openreview.net/forum?id=qY7UqLoora}, }
Why Random Pruning Is All We Need to Start Sparse

Advait Harshal Gadhikar, Sohom Mukherjee, and Rebekka Burkholz

In Proceedings of the 40th International Conference on Machine Learning, 2023

ICML Bib Abs Code

Random masks define surprisingly effective sparse neural network models, as has been shown empirically. The resulting sparse networks can often compete with dense architectures and state-of-the-art lottery ticket pruning algorithms, even though they do not rely on computationally expensive prune-train iterations and can be drawn initially without significant computational overhead. We offer a theoretical explanation of how random masks can approximate arbitrary target networks if they are wider by a logarithmic factor in the inverse sparsity 1 / \log(1/\textsparsity). This overparameterization factor is necessary at least for 3-layer random networks, which elucidates the observed degrading performance of random networks at higher sparsity. At moderate to high sparsity levels, however, our results imply that sparser networks are contained within random source networks so that any dense-to-sparse training scheme can be turned into a computationally more efficient sparse-to-sparse one by constraining the search to a fixed random mask. We demonstrate the feasibility of this approach in experiments for different pruning methods and propose particularly effective choices of initial layer-wise sparsity ratios of the random source network. As a special case, we show theoretically and experimentally that random source networks also contain strong lottery tickets.
@inproceedings{pmlr-v202-gadhikar23a, title = {Why Random Pruning Is All We Need to Start Sparse}, author = {Gadhikar, Advait Harshal and Mukherjee, Sohom and Burkholz, Rebekka}, booktitle = {Proceedings of the 40th International Conference on Machine Learning}, pages = {10542--10570}, year = {2023}, editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan}, volume = {202}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, url = {https://proceedings.mlr.press/v202/gadhikar23a.html}, }
The Network Zoo: a multilingual package for the inference and analysis of gene regulatory networks

Marouen Ben Guebila, Tian Wang, Camila M. Lopes-Ramos, Viola Fanfani, Des Weighill, Rebekka Burkholz, Daniel Schlauch, Joseph N. Paulson, Michael Altenbuchinger, Katherine H. Shutta, Abhijeet R. Sonawane, James Lim, Genis Calderer, David G.P. IJzendoorn, Daniel Morgan, Alessandro Marin, Cho-Yi Chen, Qi Song, Enakshi Saha, Dawn L. DeMeo, Megha Padi, John Platig, Marieke L. Kuijjer, Kimberly Glass, and John Quackenbush

Genome Biology, 2023

Bib Abs Code

Inference and analysis of gene regulatory networks (GRNs) require software that integrates multi-omic data from various sources. The Network Zoo (netZoo; netzoo.github.io) is a collection of open-source methods to infer GRNs, conduct differential network analyses, estimate community structure, and explore the transitions between biological states. The netZoo builds on our ongoing development of network methods, harmonizing the implementations in various computing languages and between methods to allow better integration of these tools into analytical pipelines. We demonstrate the utility using multi-omic data from the Cancer Cell Line Encyclopedia. We will continue to expand the netZoo to incorporate additional methods.
@article{BenGuebila2023, author = {Ben Guebila, Marouen and Wang, Tian and Lopes-Ramos, Camila M. and Fanfani, Viola and Weighill, Des and Burkholz, Rebekka and Schlauch, Daniel and Paulson, Joseph N. and Altenbuchinger, Michael and Shutta, Katherine H. and Sonawane, Abhijeet R. and Lim, James and Calderer, Genis and van IJzendoorn, David G.P. and Morgan, Daniel and Marin, Alessandro and Chen, Cho-Yi and Song, Qi and Saha, Enakshi and DeMeo, Dawn L. and Padi, Megha and Platig, John and Kuijjer, Marieke L. and Glass, Kimberly and Quackenbush, John}, title = {The Network Zoo: a multilingual package for the inference and analysis of gene regulatory networks}, journal = {Genome Biology}, year = {2023}, day = {09}, volume = {24}, number = {1}, pages = {45}, issn = {1474-760X}, doi = {10.1186/s13059-023-02877-1}, url = {https://doi.org/10.1186/s13059-023-02877-1}, }
DRAGON: Determining Regulatory Associations using Graphical models on multi-Omic Networks

Katherine H Shutta, Deborah Weighill, Rebekka Burkholz, Marouen Ben Guebila, Dawn L DeMeo, Helena U Zacharias, John Quackenbush, and Michael Altenbuchinger

Nucleic Acids Research, 2022

Bib Abs Code

The increasing quantity of multi-omic data, such as methylomic and transcriptomic profiles collected on the same specimen or even on the same cell, provides a unique opportunity to explore the complex interactions that define cell phenotype and govern cellular responses to perturbations. We propose a network approach based on Gaussian Graphical Models (GGMs) that facilitates the joint analysis of paired omics data. This method, called DRAGON (Determining Regulatory Associations using Graphical models on multi-Omic Networks), calibrates its parameters to achieve an optimal trade-off between the network’s complexity and estimation accuracy, while explicitly accounting for the characteristics of each of the assessed omics ‘layers.’ In simulation studies, we show that DRAGON adapts to edge density and feature size differences between omics layers, improving model inference and edge recovery compared to state-of-the-art methods. We further demonstrate in an analysis of joint transcriptome - methylome data from TCGA breast cancer specimens that DRAGON can identify key molecular mechanisms such as gene regulation via promoter methylation. In particular, we identify Transcription Factor AP-2 Beta (TFAP2B) as a potential multi-omic biomarker for basal-type breast cancer. DRAGON is available as open-source code in Python through the Network Zoo package (netZooPy v0.8; netzoo.github.io).
@article{10.1093/nar/gkac1157, author = {Shutta, Katherine H and Weighill, Deborah and Burkholz, Rebekka and Guebila, Marouen Ben and DeMeo, Dawn L and Zacharias, Helena U and Quackenbush, John and Altenbuchinger, Michael}, title = {DRAGON: Determining Regulatory Associations using Graphical models on multi-Omic Networks}, journal = {Nucleic Acids Research}, volume = {51}, number = {3}, pages = {e15-e15}, year = {2022}, issn = {0305-1048}, doi = {10.1093/nar/gkac1157}, url = {https://doi.org/10.1093/nar/gkac1157}, eprint = {https://academic.oup.com/nar/article-pdf/51/3/e15/49192710/gkac1157.pdf}, }
Most Activation Functions Can Win the Lottery Without Excessive Depth

Rebekka Burkholz

In Advances in Neural Information Processing Systems, 2022

NeurIPS Bib Abs Code

The strong lottery ticket hypothesis has highlighted the potential for training deep neural networks by pruning, which has inspired interesting practical and theoretical insights into how neural networks can represent functions. For networks with ReLU activation functions, it has been proven that a target network with depth L can be approximated by the subnetwork of a randomly initialized neural network that has double the target’s depth 2L and is wider by a logarithmic factor. We show that a depth L+1 is sufficient. This result indicates that we can expect to find lottery tickets at realistic, commonly used depths while only requiring logarithmic overparametrization. Our novel construction approach applies to a large class of activation functions and is not limited to ReLUs. Code is available on Github (RelationalML/LT-existence).
@inproceedings{NEURIPS2022_76bf7786, author = {Burkholz, Rebekka}, booktitle = {Advances in Neural Information Processing Systems}, editor = {Koyejo, S. and Mohamed, S. and Agarwal, A. and Belgrave, D. and Cho, K. and Oh, A.}, pages = {18707--18720}, publisher = {Curran Associates, Inc.}, title = {Most Activation Functions Can Win the Lottery Without Excessive Depth}, url = {https://papers.nips.cc/paper_files/paper/2022/hash/76bf7786d311217077bc8bb021946cd9-Abstract-Conference.html}, volume = {35}, year = {2022}, }
Convolutional and Residual Networks Provably Contain Lottery Tickets

Rebekka Burkholz

In Proceedings of the 39th International Conference on Machine Learning, 2022

ICML Spotlight Bib Abs Code

The Lottery Ticket Hypothesis continues to have a profound practical impact on the quest for small scale deep neural networks that solve modern deep learning tasks at competitive performance. These lottery tickets are identified by pruning large randomly initialized neural networks with architectures that are as diverse as their applications. Yet, theoretical insights that attest their existence have been mostly focused on deed fully-connected feed forward networks with ReLU activation functions. We prove that also modern architectures consisting of convolutional and residual layers that can be equipped with almost arbitrary activation functions can contain lottery tickets with high probability.
@inproceedings{pmlr-v162-burkholz22a, title = {Convolutional and Residual Networks Provably Contain Lottery Tickets}, author = {Burkholz, Rebekka}, booktitle = {Proceedings of the 39th International Conference on Machine Learning}, pages = {2414--2433}, year = {2022}, editor = {Chaudhuri, Kamalika and Jegelka, Stefanie and Song, Le and Szepesvari, Csaba and Niu, Gang and Sabato, Sivan}, volume = {162}, series = {Proceedings of Machine Learning Research}, publisher = {PMLR}, url = {https://proceedings.mlr.press/v162/burkholz22a.html}, }
An online notebook resource for reproducible inference, analysis and publication of gene regulatory networks

Marouen Ben Guebila, Deborah Weighill, Camila M. Lopes-Ramos, Rebekka Burkholz, Romana T. Pop, Kalyan Palepu, Mia Shapoval, Maud Fagny, Daniel Schlauch, Kimberly Glass, Michael Altenbuchinger, Marieke L. Kuijjer, John Platig, and John Quackenbush

Nature Methods, 2022

Bib Abs Code

Open access to software in computational and systems biology, including data, code and models, is widely acknowledged as essential for ensuring reproducibility of research results and reuse of methods1. Although there are software tools that allow sharing of computational pipelines, these systems generally do not allow the integration of software annotation and documentation at each step in the process — elements that are required to understand and run complex and rapidly evolving software, including methods developed in systems biology for inferring biological pathways. Our research team has been developing network inference and analysis methods, collected into the Network Zoo (http://netzoo.github.io), with implementations in R, C, MATLAB and Python. The growing community of users of these network resources, the increasing interest in learning how to apply network inference methods, and the need to ensure that published analyses are fully reproducible led us to develop Netbooks (http://netbooks.networkmedicine.org), a hosted collection of Jupyter notebooks that provide detailed and annotated step-by-step case studies of GRN analysis.
@article{BenGuebila2022, author = {Ben Guebila, Marouen and Weighill, Deborah and Lopes-Ramos, Camila M. and Burkholz, Rebekka and Pop, Romana T. and Palepu, Kalyan and Shapoval, Mia and Fagny, Maud and Schlauch, Daniel and Glass, Kimberly and Altenbuchinger, Michael and Kuijjer, Marieke L. and Platig, John and Quackenbush, John}, title = {An online notebook resource for reproducible inference, analysis and publication of gene regulatory networks}, journal = {Nature Methods}, year = {2022}, day = {01}, volume = {19}, number = {5}, pages = {511-513}, issn = {1548-7105}, doi = {10.1038/s41592-022-01479-2}, url = {https://doi.org/10.1038/s41592-022-01479-2}, }
Plant ’n’ Seek: Can You Find the Winning Ticket?

Jonas Fischer, and Rebekka Burkholz

In The Tenth International Conference on Learning Representations, 2022

ICLR Bib Abs Code

The lottery ticket hypothesis has sparked the rapid development of pruning algorithms that aim to reduce the computational costs associated with deep learning during training and model deployment. Currently, such algorithms are primarily evaluated on imaging data, for which we lack ground truth information and thus the understanding of how sparse lottery tickets could be. To fill this gap, we develop a framework that allows us to plant and hide winning tickets with desirable properties in randomly initialized neural networks. To analyze the ability of state-of-the-art pruning to identify tickets of extreme sparsity, we design and hide such tickets solving four challenging tasks. In extensive experiments, we observe similar trends as in imaging studies, indicating that our framework can provide transferable insights into realistic problems. Additionally, we can now see beyond such relative trends and highlight limitations of current pruning methods. Based on our results, we conclude that the current limitations in ticket sparsity are likely of algorithmic rather than fundamental nature. We anticipate that comparisons to planted tickets will facilitate future developments of efficient pruning algorithms.
@inproceedings{fischer2022plant, title = {Plant 'n' Seek: Can You Find the Winning Ticket?}, author = {Fischer, Jonas and Burkholz, Rebekka}, booktitle = {The Tenth International Conference on Learning Representations}, year = {2022}, url = {https://openreview.net/forum?id=9n9c8sf0xm}, }
On the Existence of Universal Lottery Tickets

Rebekka Burkholz, Nilanjana Laha, Rajarshi Mukherjee, and Alkis Gotovos

In The Tenth International Conference on Learning Representations, 2022

ICLR Bib Abs Code

The lottery ticket hypothesis conjectures the existence of sparse subnetworks of large randomly initialized deep neural networks that can be successfully trained in isolation. Recent work has experimentally observed that some of these tickets can be practically reused across a variety of tasks, hinting at some form of universality. We formalize this concept and theoretically prove that not only do such universal tickets exist but they also do not require further training. Our proofs introduce a couple of technical innovations related to pruning for strong lottery tickets, including extensions of subset sum results and a strategy to leverage higher amounts of depth. Our explicit sparse constructions of universal function families might be of independent interest, as they highlight representational benefits induced by univariate convolutional architectures.
@inproceedings{burkholz2022on, title = {On the Existence of Universal Lottery Tickets}, author = {Burkholz, Rebekka and Laha, Nilanjana and Mukherjee, Rajarshi and Gotovos, Alkis}, booktitle = {The Tenth International Conference on Learning Representations}, year = {2022}, url = {https://openreview.net/forum?id=SYB4WrJql1n}, }

Pre-prints

Cyclic Sparse Training: Is it Enough?

Advait Gadhikar, Sree Harsha Nelaturu, and Rebekka Burkholz

Pre-print on arXiv, 2024

Bib Abs Code

The success of iterative pruning methods in achieving state-of-the-art sparse networks has largely been attributed to improved mask identification and an implicit regularization induced by pruning. We challenge this hypothesis and instead posit that their repeated cyclic training schedules enable improved optimization. To verify this, we show that pruning at initialization is significantly boosted by repeated cyclic training, even outperforming standard iterative pruning methods. The dominant mechanism how this is achieved, as we conjecture, can be attributed to a better exploration of the loss landscape leading to a lower training loss. However, at high sparsity, repeated cyclic training alone is not enough for competitive performance. A strong coupling between learnt parameter initialization and mask seems to be required. Standard methods obtain this coupling via expensive pruning-training iterations, starting from a dense network. To achieve this with sparse training instead, we propose SCULPT-ing, i.e., repeated cyclic training of any sparse mask followed by a single pruning step to couple the parameters and the mask, which is able to match the performance of state-of-the-art iterative pruning methods in the high sparsity regime at reduced computational cost.
@misc{gadhikar2024cyclicsparsetrainingenough, title = {Cyclic Sparse Training: Is it Enough?}, author = {Gadhikar, Advait and Nelaturu, Sree Harsha and Burkholz, Rebekka}, year = {2024}, eprint = {2406.02773}, archiveprefix = {arXiv}, primaryclass = {cs.LG}, url = {https://arxiv.org/abs/2406.02773}, }
Dynamical Isometry for Residual Networks

Advait Gadhikar, and Rebekka Burkholz

Pre-print on arXiv, 2022

Bib Abs

The training success, training speed and generalization ability of neural networks rely crucially on the choice of random parameter initialization. It has been shown for multiple architectures that initial dynamical isometry is particularly advantageous. Known initialization schemes for residual blocks, however, miss this property and suffer from degrading separability of different inputs for increasing depth and instability without Batch Normalization or lack feature diversity. We propose a random initialization scheme, RISOTTO, that achieves perfect dynamical isometry for residual networks with ReLU activation functions even for finite depth and width. It balances the contributions of the residual and skip branches unlike other schemes, which initially bias towards the skip connections. In experiments, we demonstrate that in most cases our approach outperforms initialization schemes proposed to make Batch Normalization obsolete, including Fixup and SkipInit, and facilitates stable training. Also in combination with Batch Normalization, we find that RISOTTO often achieves the overall best result.
@misc{gadhikar2022dynamicalisometryresidualnetworks, title = {Dynamical Isometry for Residual Networks}, author = {Gadhikar, Advait and Burkholz, Rebekka}, year = {2022}, eprint = {2210.02411}, archiveprefix = {arXiv}, primaryclass = {cs.LG}, url = {https://arxiv.org/abs/2210.02411}, }
Lottery Tickets with Nonzero Biases

Jonas Fischer, Advait Gadhikar, and Rebekka Burkholz

Pre-print on arXiv, 2022

Bib Abs Code

The strong lottery ticket hypothesis holds the promise that pruning randomly initialized deep neural networks could offer a computationally efficient alternative to deep learning with stochastic gradient descent. Common parameter initialization schemes and existence proofs, however, are focused on networks with zero biases, thus foregoing the potential universal approximation property of pruning. To fill this gap, we extend multiple initialization schemes and existence proofs to nonzero biases, including explicit ’looks-linear’ approaches for ReLU activation functions. These do not only enable truly orthogonal parameter initialization but also reduce potential pruning errors. In experiments on standard benchmark data, we further highlight the practical benefits of nonzero bias initialization schemes, and present theoretically inspired extensions for state-of-the-art strong lottery ticket pruning.
@misc{fischer2022lotteryticketsnonzerobiases, title = {Lottery Tickets with Nonzero Biases}, author = {Fischer, Jonas and Gadhikar, Advait and Burkholz, Rebekka}, year = {2022}, eprint = {2110.11150}, archiveprefix = {arXiv}, primaryclass = {cs.LG}, url = {https://arxiv.org/abs/2110.11150}, }