Ted Chiang is emphatic: LLMs are nowhere close to being conscious. “We don’t need to fully understand the nature of consciousness to definitively say that certain things are not conscious, and conversational transcripts fall in that category.”
Advertise here with Carbon Ads
This site is made possible by member support. 💞
Big thanks to Arcustech for hosting the site and offering amazing tech support.
When you buy through links on kottke.org, I may earn an affiliate commission. Thanks for supporting the site!
kottke.org. home of fine hypertext products since 1998.
Beloved by 86.47% of the web.
Comments 12
thread
latest
popular
I was perplexed by this story at first, because Ted seems to be arguing with a shadow. (Who is saying AI, in it's current manifestation, is conscious?)
But in the second half, he started to ask some questions that intrigued me. He posits that chatbots shouldn't use first person ("The use of first-person pronouns is dishonest"), or express selfhood in any way. Interesting! I'm not sure how it would work, but interesting.
I think he's arguing with those who say that LLMs are on their way to being conscious (by GPT 7 or Opus 8 or whatever), which Chiang deems an impossibility, no matter how complex or widely trained the model.
For folks suggesting that current AI systems might be conscious, seems it came up recently here: https://www.youtube.com/watch?v=NgDIG8u1-CA
Yeah, maybe I missed that.
This is a good example of a story that I wanted to "interact with." There are some big ideas in there; so big, they're blurry. Every paragraph led me to asking a follow-up question.
Dammit, I really don't want everything to just become a podcast.
Another note on the question of who tf thinks this stuff is actually conscious… Geoffrey Hinton is apparently in that camp:
https://garymarcus.substack.com/p/the-pope-appears-to-understand-ai
I can’t tell if his beliefs are genuine or just a grab for attention, but neither possibility is heartening.
This is also before we consider people who ostensibly don’t have an understanding of machine learning, such as Grimes, who got into a little back and forth with Gary Marcus on X-Twitter recently: https://x.com/GaryMarcus/status/2060346632433180706
Reading through it, I could clearly see how someone who doesn’t have at least a basic understanding of LLMs might be beguiled into thinking that they’re conscious. As Chalmers puts it in the video linked to above, we’re now having to wrangle with real-life philosophical zombies.
From the piece:
I love Ted Chiang's writings, and I do not believe that LLMs are conscious, and even so, I found much of this article unconvincing (and I wanted to be convinced!). I agree with Rex, however, the piece got more compelling towards the end. Definitely worth a read.
Thanks, good rec! Just listened. I’m into it.
Side note: Ultimately I think conscienceness ends up being too squishy of a term and we abandon it for something else. We’ve got underground networks of fungus that are “conscious” at this point.
Oops, I replied to the wrong person. Well, anyway, good pod!
I think AI is an evolutionary dead end. Ask yourself “What happens to the stuff (data, music, coding, discussions, etc.) that AI generates? Does it just evaporite into the atmosphere? No, it remains as data that needs to be stored somewhere. Hence the need for more and more so-called data centres. These massive storage units are being built just to keep up with the output that AI is producing. Sooner or later the storage systems will contain more AI generated data than human generated - if they aren’t already - and will eventually dwarf human output. Since we cannot keep building data centres forever a time will come when choices will have to be made about what data gets to stay and what gets to be “forgotten”. And that is a dead end for AI if not for humanity. Don’t say you weren’t warned.
Long before that point (storage is cheap and has low power requirements compared to the kind of compute that STEM needs), the machines will be ingesting so much of their own extrusions that they will collapse. This is a well-known phenomenon and is already being observed.
The current systems are conscious in the same way that a cloud or a tree is conscious. They're not, but humans will see faces in the clouds and invent nature spirits and gods where there are none: that's what we do and what we've done since before we were human.
The surprising thing is that the very person who famously wrote a book about this has now fallen into this very trap of imagining that there is a ghost in the machine. That should be a warning to everyone.
Hello! In order to comment or fave, you need to be a current kottke.org member. If you'd like to sign up for a membership to support the site and join the conversation, you can explore your options here.
Existing members can sign in here. If you're a former member, you can renew your membership.
Note: If you are a member and tried to log in, it didn't work, and now you're stuck in a neverending login loop of death, try disabling any ad blockers or extensions. Or try logging out and then back in. Still having trouble? Email me!
In order to comment or fave, you need to be a current kottke.org member. Check out your options for renewal.
If you feel like this comment goes against the grain of the community guidelines or is otherwise inappropriate, please let me know and I will take a look at it.
Hello! In order to leave a comment, you need to be a current kottke.org member. If you'd like to sign up for a membership to support the site and join the conversation, you can explore your options here.
Existing members can sign in here. If you're a former member, you can renew your membership.
Note: If you are a member and tried to log in, it didn't work, and now you're stuck in a neverending login loop of death, try disabling any ad blockers or extensions. Or try logging out and then back in. Still having trouble? Email me!