Shikhar Srivastava
PhD student in Computer Science
University of Rochester
Department of Computer Science
University of Rochester
I’m a Computer Science PhD student at the University of Rochester, advised by Christopher Kanan. I’m doing a PhD to enable large models to learn continually: I’m primarily interested in memory, learning dynamics, and enabling non-negative knowledge transfer across shared task structures in LLMs.
I also work with Catherine Arnett at EleutherAI, and Dhireesha Kudithipudi and Itamar Lerner at UTSA.
Previously, I graduated in Computer Science from MIT Manipal with the Institute’s Academic Excellence Award, then led data science and the AI Roadmap for DHL Ecommerce’s entire Smart-Trucking India business. I also did pre-doctoral work on continual learning for healthcare at MBZUAI & G42.
Publications
* Equal contribution-
Latest LayerRoPE: Dynamic Depth-wise Magnitude & Angular SuperpositionProposing depth encoding in LLMsLLaMA2-7B, norm weights by layer
A RoPE-style depth encoding in the residual stream. It improves compute scaling, depth scaling up to 512 layers, and widens the learning-rate basin, by learning to amplify residual variance rather than dampen it.
Preprint · COLM 2026, Actionable Interpretability
-
No Single Tokenizer Feature Reliably Predicts Downstream Language Model Performance
EMNLP 2026, Findings
TL;DRCan tokenizer quality be judged without training a model? Across 159 language models in nine languages, including 100 new VISTA models at 130M and 350M, none of 29 intrinsic tokenizer metrics reliably predicts downstream performance. Pooled correlations are weak (best r = 0.27) and flip sign across tasks, languages and model families.
No tokenizer metric predicts LLM performancePooled r, 29 tokenizer metrics -
SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition
CoLLAs 2026
TL;DRA streaming sequence learner that separates memory from prediction. Inspired by accelerated replay in sleeping rodents, offline “sleep” phases replay stored experience in compressed time to train a memory hierarchy whose levels tick 4× slower each, reaching a 1,024-step context while backpropagating through only 4 steps. It beats recurrent baselines on text8 and PG-19.
Hierarchical accelerated replay during sleepEach memory level ticks 4× slower -
The Unreasonable Benchmark
DMLR 2026
TL;DRA crowdsourced, living benchmark of everyday text and image questions that are easy for people but stump frontier LLMs, like moving three matchsticks so a fish swims the other way. Contributors tested 18 model variants, from GPT-4o and o3 to Claude 3.7 and Gemini 2.5 Pro, and a question retires once any model solves it. Only 21 have fallen; 100 remain open.
Problem: Move 3 sticks to flip the fishUnsolved by every model tested -
Improving Multimodal Large Language Models Using Continual Learning
CoLLAs 2025 · NeurIPS 2024, Scalable Continual Learning for Lifelong Foundation Models
TL;DRBuilding a multimodal LLM with the LLaVA recipe erodes the base LLM’s language skills. Treating this as continual learning, a study of 9 base LLMs (160M–7B) and five continual-learning methods finds one that cuts NLG forgetting from 9 points to about 1 at 2.8B, with nearly the same vision-language accuracy. Language skills also hold up as new vision-language tasks arrive in sequence.
Preventing linguistic forgetting in MLLMsPythia 2.8B; NLG drop vs. LLM only -
Revisiting Multi-Modal LLM Evaluation
CVPR 2025, Benchmarking & Expanding AI Multimodal Approaches · ICLR 2025, Navigating and Addressing Data Problems for Foundation Models; The Future of Machine Learning Data Practices and Repositories
TL;DRPopular VQA benchmarks are old, biased and too coarse for today’s multimodal LLMs. Testing 12 MLLMs, from LLaVA to GPT-4o, on four fine-grained datasets (TDIUC, TallyQA, DVQA, VQDv1) exposes unreported weaknesses: the best scores just 27 F1 at finding every object that matches a query, and models falter at complex counting, positional and chart reasoning.
All MLLMs struggle with counting & groundingVQDv1 F1: find all matching objects -
Continual Learning for Chest X-Ray Classification in Low-Resource Clinical Settings
MICCAI 2021, FAIR
TL;DRChest X-ray classifiers degrade as scanners and data sources drift, and low-resource clinics must adapt them under tight memory and privacy limits. Replaying compact vector-quantized features (no stored images, a ~3% buffer) adapts a model across an interleaved CheXpert → NIH → PadChest stream without knowing where domains shift, and beats fine-tuning, LwF and EWC on every domain.
Continual adaptation in clinical settingsFinal test AUC on each domain
Personal
I’m fascinated by memory. By the phenomena arising from our brains developing collectively through the phylogenetic tree. By old stories and old ideas.
I believe strongly in working towards the benefit of all.