John Y

Position: Humans are Missing from AI Coding Agent Research
John Yang*, Zora Z. Wang*, Kilian Lieret*, Alexa Tartaglini, Valerie Chen, Yuxiang Wei, Zijian Wang, Lingming Zhang, Karthik Narasimhan, Ludwig Schmidt, Graham Neubig, Daniel Fried, Diyi Yang
2026

ProgramBench: Can Language Models Rebuild Programs From Scratch?
John Yang*, Kilian Lieret*, Jeffrey Ma, Parth Thakkar, Dmitrii Pedchenko, Sten Sootla, Emily McMilin, Pengcheng Yin, Rui Hou, Gabriel Synnaeve, Diyi Yang, Ofir Press
2026 • NeurIPS (Datasets & Benchmarks)

CodeClash: Benchmarking Goal-Oriented Software Engineering
John Yang*, Kilian Lieret*, Joyce Yang, Carlos E. Jimenez, Muhtasham Oblokulov, Aryan Siddiqui, Ofir Press, Ludwig Schmidt, Diyi Yang
2026 • ICML

SWE-smith: Scaling Data for Software Engineering Agents
John Yang, Kilian Lieret, Carlos E. Jimenez, Alexander Wettig, Kabir Khandpur, Yanzhe Zhang, Binyuan Hui, Ofir Press, Ludwig Schmidt, Diyi Yang
2025 • NeurIPS (Datasets & Benchmarks) • Spotlight

SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?
John Yang*, Carlos E. Jimenez*, Alex L. Zhang, Kilian Lieret, Joyce Yang, Xindi Wu, Ori Press, Niklas Muennighoff, Gabriel Synnaeve, Karthik Narasimhan, Diyi Yang, Sida I. Wang, Ofir Press
2025 • ICLR

SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering
John Yang*, Carlos E. Jimenez*, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, Ofir Press
2024 • NeurIPS

SWE-bench: Can Language Models Resolve Real-World Github Issues?
Carlos E. Jimenez*, John Yang*, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, Karthik Narasimhan
2024 • ICLR • Oral

InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback
John Yang, Akshara Prabhakar, Karthik Narasimhan, Shunyu Yao
2023 • NeurIPS (Datasets & Benchmarks)

Language Agents as Hackers: Evaluating Cybersecurity Skills with Capture the Flag
John Yang, Akshara Prabhakar, Shunyu Yao, Kexin Pei, Karthik Narasimhan
2023 • Multi-Agent Security Workshop @ NeurIPS 2023 • Best Paper Award

Towards an Enhanced, Faithful, and Adaptable Web Interaction Environment
John Yang, Howard Chen, Karthik Narasimhan
2022 • Language & Reinforcement Learning Workshop @ NeurIPS 2022

SWE-chat: Coding Agent Interactions From Real Users in the Wild
Joachim Baumann, Vishakh Padmakumar, Kevin Xiang Li, John Yang, Diyi Yang, Sanmi Koyejo
2026 • COLM

SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents
Patrick Tser Jern Kon, Archana Pradeep, Ang Chen, Alexander P. Ellis, Warren Hunt, Zijian Wang, John Yang, Samuel Thompson
2026 • NeurIPS

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
Mike A. Merrill, Alexander G. Shaw, Nicholas Carlini, ..., John Yang, ..., Alex Dimakis, Andy Konwinski, Ludwig Schmidt
2026 • ICLR

OpenThoughts: Data Recipes for Reasoning Models
Etash Guha, Ryan Marten, Sedrick Keh, ..., John Yang, ..., Maheswaran Sathiamoorthy, Alexandros G. Dimakis, Ludwig Schmidt
2026 • ICLR • Oral

Collaborative Gym: A Framework for Enabling and Evaluating Human-Agent Collaboration
Yijia Shao, Vinay Samuel, Yucheng Jiang, John Yang, Diyi Yang
2026 • ICLR

EnIGMA: Interactive Tools Substantially Assist LM Agents in Finding Security Vulnerabilities
Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, Ofir Press
2025 • ICML

MMTEB: Massive Multilingual Text Embedding Benchmark
Kenneth Enevoldsen, Isaac Chung, Imene Kerboua, ..., John Yang, ..., Orion Weller, Siva Reddy, Niklas Muennighoff
2025 • ICLR

LongCodeBench: Evaluating Coding LLMs at 1M Context Windows
Stefano Rando*, Luca Romani*, Alessio Sampieri*, Luca Franco, John Yang, Yuta Kyuragi, Fabio Galasso, Tatsunori Hashimoto
2025 • COLM

Prompting Large Language Models to Tackle the Full Software Development Lifecycle: A Case Study
Bowen Li*, Wenhan Wu*, Ziwei Tang*, Lin Shi*, John Yang, Jinyang Li, Shunyu Yao, Chen Qian, Binyuan Hui, Qicheng Zhang, Zhiyin Yu, He Du, Ping Yang, Dahua Lin, Chao Peng, Kai Chen
2025 • COLING • Oral

Disentangled Prompt Learning for Transferable, Multimodal, Few-Shot Image Classification
John Yang, Alessandro Magnani, Binwei Yang
2024 • IEEE BigData

Introducing SWE-bench Verified
Neil Chowdhury*, James Aung*, Chan Jun Shern*, Oliver Jaffe*, Dane Sherburn*, Giulio Starace*, Evan Mays, Rachel Dias, Marwan Aljubeh, Mia Glaese, Carlos E. Jimenez, John Yang, Kevin Liu, Aleksander Madry
2024 • OpenAI Technical Blog

Referral Augmentation for Zero-Shot Information Retrieval
Michael William Tang, Shunyu Yao, John Yang, Karthik Narasimhan
2024 • ACL (Findings)

WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents
Shunyu Yao*, Howard Chen*, John Yang, Karthik Narasimhan
2022 • NeurIPS

Learning Language through Interactions with the Digital World
John Yang
2022 • M.S.E. Thesis | Princeton University

Quartz: A Framework for Engineering Secure Smart Contracts
John Kolb, John Yang, Randy H Katz, David E Culler
2020 • EECS Department, University of California, Berkeley, Tech. Rep. UCB/EECS-2020-178