AIs encode language like the brain – opening a window into human conversations

Language allows people to convey thoughts to one another because each person’s brain responds similarly to the meaning of words. In our newly published researchmy colleagues and me developed a framework for modeling speakers’ brain activity while they conversed face-to-face.

We recorded the electrical activity of two people’s brains while they had an impromptu conversation. Previous research has shown that when two people talk, their brain activity becomes coupled, or aligned, and that the degree of neural coupling is associated with a better understanding of the speaker’s message.

A neural code refers to specific patterns of brain activity associated with distinct words in their contexts. We discovered that speakers’ brains are aligned in a shared neural code. Importantly, the brain’s neural code resembled the artificial neural code of large language models, or LLMs.

The neural patterns of words

A great language model is a machine learning program that can generate text by predicting which words are likely to follow others. Great language models are excellent at learning language. language structure, generating human text and maintaining conversations. They might even pass Turing Test, making it difficult for someone to discern whether they are interacting with a machine or a human. Like humans, LLMs learn to speak by reading or listening to texts produced by other humans.

By providing LLM with a transcript of the conversation, we were able to extract its “neural activations,” or how it translates words into numbers, as it “reads” the script. We then correlated the speaker’s brain activity with both LLM activations and the listener’s brain activity. We found that LLM activations could predict the shared brain activity of the speaker and listener.

To be able to understand each other, people have a common agreement on grammatical rules and meaning of words in context. For example, we know that we should use the past tense of a verb to talk about past actions, as in the sentence: “He visited the museum yesterday”. Furthermore, we intuitively understand that the same word can have different meanings in different situations. For example, the word cold in the sentence “you’re as cold as ice” can refer to body temperature or a personality trait, depending on the context. Due to the complexity and richness of natural language, until the recent success of great language models, we lacked an accurate mathematical model to describe it.

Our study found that large linguistic models can predict how linguistic information is encoded in the human brain, providing a new tool for interpreting human brain activity. The similarity between the linguistic code of the human brain and the linguistic code of the great language model has allowed us, for the first time, to trace how information in the speaker’s brain is encoded into words and transferred, word by word, to the listener’s brain. during the face-to-face conversation. face conversations. For example, we found that brain activity associated with the meaning of a word emerges in the speaker’s brain before articulating a word, and the same activity quickly resurfaces in the listener’s brain after hearing the word.

New powerful tool

Our study provided insights into the neural code for language processing in the human brain and how humans and machines can use this code to communicate. We found that large language models were better able to predict shared brain activity compared to different features of language, such as syntax, or the order in which words connect to form phrases and sentences. This is partly due to LLM’s ability to incorporate the contextual meaning of words, as well as to integrate multiple levels of the linguistic hierarchy into one model: from words to sentences to conceptual meaning. This suggests important similarities between the brain and artificial neural networks.

An important aspect of our research is using everyday recordings of natural conversations to ensure our findings capture real-life brain processing. This is called Ecological validity. In contrast to experiments where participants are told what to say, we gave up control of the study and let participants talk as naturally as possible. This loss of control makes data analysis difficult because each conversation is unique and involves two interacting individuals who speak spontaneously. Our ability to model neural activity as people engage in everyday conversations testifies to the power of great language models.

Other dimensions

Now that we have developed a framework for assessing the neural code shared between brains during everyday conversations, we are interested in knowing what factors drive or inhibit this coupling. For example, does linguistic coupling increase if the listener better understands the speaker’s intention? Or perhaps complex language, like jargon, could reduce neural coupling.

Another factor that can influence linguistic coupling can be the relationship between speakers. For example, you can convey a lot of information in a few words to a good friend, but not to a stranger. Or you may be better neurally coupled to political allies rather than rivals. This is because differences in how we use words between groups can make it easier to align and connect with people within rather than outside our social groups.

This article was republished from The conversation, an independent, nonprofit news organization that brings you trusted facts and analysis to help you understand our complex world. It was written by: Zaid Zada, Princeton University

See more information:

Zaid Zada ​​does not work for, consult with, own shares in or receive funding from any company or organization that would benefit from this article, and has disclosed no relevant affiliations beyond his academic appointment.