One of the decisive moments in my understanding of #LLMs and their limitations was when, last autumn, @emilymbender walked me through her Thai Library thought experiment.
She's now written it up as a Medium post, and you can read it here. The value comes from really pondering the question she poses, so take the time to think about it. What would YOU do in the situation she outlines?
@ct_bergstrom@emilymbender I am not convinced. I think the thought experiment relies too much on analogy to what a person would be capable of in that situation. A LLM is not a human brain and so, does not process information in exactly the same way.
A person would surely be maxed out of ability to maintain an overview of so many boys off information at once.
@ct_bergstrom@emilymbender spoiler:"But also: it’s not “intelligent”. Our only evidence for its “intelligence” is the apparent coherence of its output."
TIL the difference between, and relationship of, meaning and form. And now I finally get how words can truly be the mother of ten thousand things, dang.
@ct_bergstrom@emilymbender This article brings a great thought experiment, next level chinese room.
Clarifies how llm really don't have any concept of meaning.
@ct_bergstrom@emilymbender My brother’s opening sentence of his dissertation “A Phenomonology of the Listening Body” states, “Listening begins with the turn of the self’s attentive being in an attitude of receptivity toward the communicative gestures of the other.” 1/x
@ct_bergstrom@emilymbender By basic extension, a baby with total hearing loss would also not be able to understand that written language. By the sensor-fusion argument, no baby can acquire understanding of any language with an audiovisual homomorphism. Thus, no human understands any natural language.
A reasonable conclusion, then, is that humans don't understand anything. This matches both my experience and the typical Chinese Room analysis. Y'all want to remind us that LLMs aren't people, but you need to not glorify meatbags either. We are walking collections of memes, not necessarily acting according to any particular justification.
@ct_bergstrom@emilymbender@thedarktangent that's not really true though. While an AI might never taste an apple, the whole field of mathematics is entirely deducible from symbol patterns. There is no external thing that is a "two", it's a pattern, whether of apples or bits or Thai words. Programs that learn and can apply the rules can be considered to "understand" since the rules and patterns are the essence of math. Some generic pattern finding programs have even discovered new useful math.
@ct_bergstrom@emilymbender The only strategies I can come up with still rely on external knowledge (e.g. looking for maths textbooks). I find the argument to be very convincing though I was pretty strongly in the parrots camp to begin with. Also, an LLM not understanding meaning and not being intelligent doesn't preclude it from still being a very useful tool in many situations; it can however serve as a guideline as to which situations this tool should and should not be used in.
@ct_bergstrom (BTW, a bit out of subject, but kudos to archeologists who for some languages had to deal with settings looking like that. Mostly relying on the gotchas 3, 5 & 6 - i.e. relying on whatever external knowledge they could grasp and doing tons of statistics on it).
@ct_bergstrom@emilymbender@timnitGebru I wonder what the output of ChatGPT looks like when you give it a prompt that is "logical nonsense" - correct structure but no coherent meaning. 🤔
@ct_bergstrom@emilymbender Love the Thai Library thought experiment. How does it hold up with multimodal LLMs that train on images and video in addition to text? Does "knowing" what a horse looks like give meaning to the word "horse"? What if the model has "seen" people ride them?
@ct_bergstrom@emilymbender one thing that is hard for me to wrap my head around -- ok, imagine you're in the thai library, but you're not human, you're a very powerful statistical likelihood machine… then, at some point, you get introduced back out in the world, where you are presented with contexts very much like the text you were able to observe in the library. i feel like all of our intuitions break down about what happens next.
@ct_bergstrom@emilymbender Fantastic post. I was thinking there’s an even more general problem that goes beyond LLMs…the lack of contact with the physical world. Even if an AI has access to illustrations and other info, it would never be able to discern truth without the ability to interact with the physical world.
@ct_bergstrom@dpp@emilymbender this is a very interesting though experiment, but it will b/c very blurry with multimodal models which mixe text, sound, image. Likely possible to derive complex model of meaning with that (still missing first hand causal feedback loop, but hey, we can I least rich philosophy if not science).
Plus, chatgpt is already past the Thai library: it is fed with labeled data giving meaning/concept (in the lang of the creators) to string of words.
Thanks for the link!
@ct_bergstrom@emilymbender Does anybody ever do that? Isn't all understanding of language ultimately by association with something else you understand? Even our first language – baby learns what "bottle" means while you're feeding them with one?
Add comment