Ronak Pradeep
Hi! I am a Research Scientist at Yupp AI working on benchmarking and evaluating LLMs. I completed my PhD in the David R. Cheriton School of Computer Science at the University of Waterloo, where I was advised by Jimmy Lin. During my PhD, I also had the chance to work at Google and Apple.
Previously, I completed my undergraduate studies at the University of Waterloo, where I majored in Computer Science and Combinatorics and Optimization. During my undergrad, I've had the chance to intern at Quebec Artificial Intelligence Institute (Mila), ContextLogic, and RBC Research.
Research Interests
My research interests lie at the intersection of Information Retrieval and Natural Language Processing. More specifically, I'm interested in tasks such as Evaluation, Benchmarking, Retrieval-Augmented Generation, Tool Use, Fact Verification, and Document Ranking. I have also been investigating the memory component of Large Language Models and the interplay between the inherent reasoning and memory modules, entangled in a single LLM or otherwise. I look forward to contributing to the next generation of reasoners capable of working with a constantly evolving ocean of both structured and unstructured data Some of my earlier work explores how to build neural search systems that promote correct and reliable information and work well in low-resource domains such as biomedical texts.
Latest Updates
Wrapped up my PhD at Waterloo.
We have five papers accepted at SIGIR 2025! See you all in Padova!
We had a successful first year of the TREC RAG 2024 Track with many submissions! Looking forward to Y2.
Organizing the TREC RAG 2024 Track! Do submit your systems :))
We introduced RankZephyr which garnered great community engagement!
I'm excited to visit Singapore for EMNLP 2023 to present our work "How Does Generative Retrieval Scale to Millions of Passages?"
I will be leading the TREC 2024 Retrieval-Augmented Generation Track! More information coming soon!
We introduced RankVicuna, the first zero-shot listwise reranker that leverages open-source LLMs!
Publications
NeuCLIRBench: A Modern Evaluation Collection for Monolingual, Cross-Language, and Multilingual Information Retrieval
Dawn Lawrie, James Mayfield, Eugene Yang, Andrew Yates, Sean MacAvaney, Ronak Pradeep, Scott Miller, Paul McNamee, Luca Soldani
RankLLM: A Python Package for Reranking with LLMs
Sahel Sharifymoghaddam, Ronak Pradeep, Andre Slavescu, Ryan Nguyen, Andrew Xu, Zijian Chen, Yilin Zhang, Yidi Chen, Jasper Xian, Jimmy Lin
Gosling Grows Up: Retrieval with Learned Dense and Sparse Representations Using Anserini
Jimmy Lin, Arthur Haonan Chen, Carlos Lassance, Xueguang Ma, Ronak Pradeep, Tommaso Teofili, Jasper Xian, Jheng-Hong Yang, Brayden Zhong, Vincent Zhong
Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judges
Nandan Thakur, Ronak Pradeep, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin
Chatbot Arena Meets Nuggets: Towards Explanations and Diagnostics in the Evaluation of LLM Responses
Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, Jimmy Lin
The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models
Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin
Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework
Ronak Pradeep, Nandan Thakur, Shivani Upadhyay, Daniel Campos, Nick Craswell, Jimmy Lin
A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Look
Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Daniel Campos, Nick Craswell, Ian Soboroff, Hoa Trang Dang, Jimmy Lin
A FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking
Zijian Chen, Ronak Pradeep, Jimmy Lin
Ragnarök: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track
Ronak Pradeep, Nandan Thakur, Sahel Sharifymoghaddam, Eric Zhang, Ryan Nguyen, Daniel Campos, Nick Craswell, Jimmy Lin
Prompts as Auto-Optimized Training Hyperparameters: Training Best-in-Class IR Models from Scratch with 10 Gold Labels
Jasper Xian, Saron Samuel, Faraz Khoubsirat, Ronak Pradeep, Md Arafat Sultan, Radu Florian, Salim Roukos, Avirup Sil, Christopher Potts, Omar Khattab
UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor
Shivani Upadhyay, Ronak Pradeep, Nandan Thakur, Nick Craswell, Jimmy Lin
ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models
Ronak Pradeep, Daniel Lee, Ali Mousavi, Jeffrey Pound, Yisi Sang, Jimmy Lin, Ihab Ilyas, Saloni Potdar, Mostafa Arefiyan, Yunyao Li
Entity Disambiguation via Fusion Entity Decoding
Junxiong Wang, Ali Mousavi, Omar Attia, Ronak Pradeep, Saloni Potdar, Alexander Rush, Umar Farooq Minhas, Yunyao Li
Zero-Shot Cross-Lingual Reranking with Large Language Models for Low-Resource Languages
Mofetoluwa Adeyemi, Akintunde Oladipo, Ronak Pradeep, Jimmy Lin
LiT and Lean: Distilling Listwise Rerankers Into Encoder-Decoder Models
Manveer Singh Tamber, Ronak Pradeep, Jimmy Lin
RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!
Ronak Pradeep, Sahel Sharifymoghaddam, Jimmy Lin
RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models
Ronak Pradeep, Sahel Sharifymoghaddam, Jimmy Lin
Vector Search with OpenAI Embeddings: Lucene Is All You Need
Jimmy Lin, Ronak Pradeep, Tommaso Teofili, Jasper Xian
naverloo @ TREC Deep Learning and NeuCLIR 2023: As Easy as Zero, One, Two, Three — Cascading Dual Encoders, Mono, Duo, and Listo for Ad-Hoc Retrieval
Carlos Lassance, Ronak Pradeep, Jimmy Lin
Towards Automated End-to-End Health Misinformation Free Search with a Large Language Model
Ronak Pradeep, Jimmy Lin
How Does Generative Retrieval Scale to Millions of Passages?
Ronak Pradeep, Kai Hui, Jai Gupta, Adam D Lelkes, Honglei Zhuang, Jimmy Lin, Donald Metzler, Vinh Q Tran
ReadProbe: A Demo of Retrieval-Enhanced Large Language Models to Support Lateral Reading
Dake Zhang, Ronak Pradeep
Zero-Shot Listwise Document Reranking with a Large Language Model
Xueguang Ma, Xinyu Zhang, Ronak Pradeep, Jimmy Lin
Pre-processing Matters! Improved Wikipedia Corpora for Open-Domain Question Answering
Manveer Singh Tamber, Ronak Pradeep, Jimmy Lin
PyGaggle: A Gaggle of Resources for Open-Domain Question Answering
Ronak Pradeep, Haonan Chen, Lingwei Gu, Manveer Singh Tamber, Jimmy Lin
Neural Query Synthesis and Domain-Specific Ranking Templates for Multi-Stage Clinical Trial Matching
Ronak Pradeep, Yilin Li, Yuetong Wang, Jimmy Lin
Document Expansion Baselines and Learned Sparse Lexical Representations for MS MARCO v1 and v2
Xueguang Ma, Ronak Pradeep, Rodrigo Nogueira, Jimmy Lin
Another Look at DPR: Reproduction of Training and Replication of Retrieval
Xueguang Ma, Kai Sun, Ronak Pradeep, Minghan Li, Jimmy Lin
New Nails for Old Hammers: Anserini and Pyserini at TREC 2021
Jimmy Lin, Haonen Chen, Chengcheng Hu, Sheng-Chieh Lin, Yilin Li, Xueguang Ma, Ronak Pradeep, Jheng-Hong Yang, Chuan-Ju Wang, Andrew Yates, Xinyu Zhang
Vera: Prediction Techniques for Reducing Harmful Misinformation In Consumer Health Search
Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, Jimmy Lin
Chatty Goose: A Python Framework for Conversational Search
Edwin Zhang, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, Rodrigo Nogueira, Jimmy Lin
Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations
Jimmy Lin, Xueguang Ma, Sheng-Chieh Lin, Jheng-Hong Yang, Ronak Pradeep, Rodrigo Nogueira
H₂oloo at TAC 2020: Epidemic Question Answering
Justin Borromeo, Ronak Pradeep, Jimmy Lin
Exploring Listwise Evidence Reasoning with T5 for Fact Verification
Kelvin Jiang, Ronak Pradeep, Jimmy Lin
H₂oloo at TREC 2020: When all you got is a Hammer... Deep Learning, Health Misinformation, and Precision Medicine
Ronak Pradeep, Xueguang Ma, Xinyu Zhang, Hang Cui, Ruizhou Xu, Rodrigo Nogueira, Jimmy Lin
Scientific Claim Verification with VerT5erini
Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, Jimmy Lin
A Replication Study of Dense Passage Retriever
Xueguang Ma, Kai Sun, Ronak Pradeep, Jimmy Lin
Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset
Edwin Zhang, Nikhil Gupta, Raphael Tang, Xiao Han, Ronak Pradeep, Kuang Lu, Yue Zhang, Rodrigo Nogueira, Kyunghyun Cho, Hui Fang, Jimmy Lin
The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models
Ronak Pradeep, Rodrigo Nogueira, Jimmy Lin
Document Ranking with a Pretrained Sequence-to-Sequence Model
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, Jimmy Lin
Foveated Down-Sampling Techniques
Parsa Torabian, Ronak Pradeep, Jeff Orchard, Bryan Tripp
... and a few more papers I might have potentially missed! See my Google Scholar for the complete list.
Mentorship
During my PhD journey, I had the privilege of mentoring over 40 undergraduate students! Below are some of the amazing students I've had the honor to work with (ordered lexicographically by last name):
Vikram Chandramohan
Haonan (Arthur) Chen
Steven Chen
Yidi Chen
Hang Cui
Lily Ge
Lingwei Gu
Qing Guo
Zibo (Daniel) Guo
Nikhil Gupta
Xiao (Johnson) Han
Yuxuan Ji
Faraz Khoubsirat
Nathan Kuissi
Sisi Li
Yilin (Larry) Li
Estella Liu
Yuqi Liu
Xueguang Ma
Jie (Stefan) Min
Ryan Nguyen
Nima Sadri
Andre Slavescu
Yuvan Sooryakumar
Kai Sun
Manveer Singh Tamber
Raghav Vasudeva
Yuetong Wang
Alvis Wong
Jasper Xian
Andrew Xu
Edward Xu
Ruizhou (Richard) Xu
Yicheng (Kevin) Xu
Jeongseop (Patrick) Yi
Edwin Zhang
Eric Zhang
Lijia (Lizzy) Zhang
Brayden Zhong
Vincent Zhong
Catherine Zhou
Joe Zou
Playlists
Thanks for making it to here :-) As a token of gratitude and since you asked nicely for it, I shall also introduce you to a few of my Spotify playlists.
ॐ
And everything under the sun is in tune. A music dump of sorts. Updated regularly.
A Day In The Life
An allusion to the Beatles song. Curated by a younger me for a someone who stole my heart. Not updated anymore.
Liebesträume
And what exactly is a dream of love? Here I take on Liszt and attempt to provide a longer answer to aid with my sleep. Updated semi-regularly.