Tag: AI

Quantifying Human-AI Synergy

Christoph Riedl, Ben Weidmann

We introduce a novel Bayesian Item Response Theory framework to quantify human–AI synergy, separating individual and collaborative ability while controlling for task difficulty in interactive settings. Unlike standard static benchmarks, our approach models human–AI performance as a joint process, capturing both user-specific factors and moment-to-moment fluctuations. We validate the framework by applying it to human–AI benchmark data (n=667) and find significant synergy. We demonstrate that collaboration ability is distinct from individual problem-solving ability. Users better able to infer and adapt to others’ perspectives achieve superior collaborative performance with AI–but not when working alone. Moreover, moment-to-moment fluctuations in perspective taking influence AI response quality, highlighting the role of dynamic user factors in collaboration. By introducing a principled framework to analyze data from human-AI collaboration, interactive benchmarks can better complement current single-task benchmarks and crowd-assessment methods. This work informs the design and training of language models that transcend static prompt benchmarks to achieve adaptive, socially aware collaboration with diverse and dynamic human partners.

https://osf.io/preprints/psyarxiv/vbkmt_v1

Gabriele Scheler: From Verbal Thought to Neuron Computation | Brain Inspired

Gabriele Scheler co-founded the Carl Correns Foundation for Mathematical Biology. Carl Correns was her great grandfather, one of the early pioneers in genetics. Gabriele is a computational neuroscientist, whose goal is to build models of cellular computation, and much of her focus is on neurons.
We discuss her theoretical work building a new kind of single neuron model. She, like Dmitri Chklovskii a few episodes ago, believes we’ve been stuck with essentially the same family of models for a neuron for a long time, despite minor variations on those models. The model Gabriele is working on, for example, respects the computations going on not only externally, via spiking, which has been the only game in town forever, but also the computations going on within the cell itself. Gabriele is in line with previous guests like Randy Gallistel, David Glanzman, and Hessam Akhlaghpour, who argue that we need to pay attention to how neurons are computing various things internally and how that affects our cognition. Gabriele also believes the new neuron model she’s developing will improve AI, drastically simplifying the models by providing them with smarter neurons, essentially.
We also discuss the importance of neuromodulation, her interest in wanting to understand how we think via our internal verbal monologue, her lifelong interest in language in general, what she thinks about LLMs, why she decided to start her own foundation to fund her science, what that experience has been like so far. Gabriele has been working on these topics for many years, and as you’ll hear in a moment, she was there when computational neuroscience was just starting to pop up in a few places, when it was a nascent field, unlike its current ubiquity in neuroscience.

Listen at: braininspired.co

Fact-checking information from large language models can decrease headline discernment

Matthew R. DeVerna, Harry Yaojun Yan, Kai-Cheng Yang, and Filippo Menczer

PNAS 121 (50) e2322823121

This study explores how large language models (LLMs) used for fact-checking affect the perception and dissemination of political news headlines. Despite the growing adoption of AI and tests of its ability to counter online misinformation, little is known about how people respond to LLM-driven fact-checking. This experiment reveals that even LLMs that accurately identify false headlines do not necessarily enhance users’ abilities to discern headline accuracy or promote accurate news sharing. LLM fact checks can actually reduce belief in true news wrongly labeled as false and increase belief in dubious headlines when the AI is unsure about an article’s veracity. These findings underscore the need for research on AI fact-checking’s unintended consequences, informing policies to enhance information integrity in the digital age.

Read the full article in PNAS: doi:10.1073/pnas.2322823121