New paper: Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with

[maykcaldas]

Paper: Can LLMs Generate Novel Research Ideas? A Large-Scale Human Study with

Authors: Chenglei Si, Diyi Yang, Tatsunori Hashimoto

Abstract: Recent advancements in large language models (LLMs) have sparked optimismabout their potential to accelerate scientific discovery, with a growing numberof works proposing research agents that autonomously generate and validate newideas. Despite this, no evaluations have shown that LLM systems can take thevery first step of producing novel, expert-level ideas, let alone perform theentire research process. We address this by establishing an experimental designthat evaluates research idea generation while controlling for confounders andperforms the first head-to-head comparison between expert NLP researchers andan LLM ideation agent. By recruiting over 100 NLP researchers to write novelideas and blind reviews of both LLM and human ideas, we obtain the firststatistically significant conclusion on current LLM capabilities for researchideation: we find LLM-generated ideas are judged as more novel (p < 0.05) thanhuman expert ideas while being judged slightly weaker on feasibility. Studyingour agent baselines closely, we identify open problems in building andevaluating research agents, including failures of LLM self-evaluation and theirlack of diversity in generation. Finally, we acknowledge that human judgementsof novelty can be difficult, even by experts, and propose an end-to-end studydesign which recruits researchers to execute these ideas into full projects,enabling us to study whether these novelty and feasibility judgements result inmeaningful differences in research outcome.

Link: https://arxiv.org/abs/2409.04109

Reasoning: produce the answer. We start by examining the title and abstract for any mention of language models or related terms. The title mentions "LLMs" which stands for large language models. The abstract discusses the capabilities of large language models (LLMs) in generating novel research ideas and compares them to human experts. It also mentions the evaluation of LLMs in the context of research ideation. Given these points, it is clear that the paper is focused on the application and evaluation of language models.