A solution indicating that Mouse Utopia is an inherent property of intelligent systems. The problem is information fidelity loss when later generations are trained on regurgitated data.
I'm going to add this comment here based on a discussion that John Carter and I had in the comments of his restack:
I think we could theoretically calculate entropy levels in a system by averaging data-input and data-output. Doing so for simple AI systems would require high level physics, but allow us to calculate the rise of entropy in the intake and data-production of an AI over multiple generations of training. This “generational entropy” or “information fidelity” level could then provide a baseline for examining human civilization.
The results then could apply to individuals, or subcultures. It would just be difficult: universalizing the physics from simple image-generators and text-generators to complex social systems. While the model wouldn’t be perfect, it might show the levels at which fidelity breakdown begins to occur; a mathematical way to calculate how densely packed people/services can be before causing critical levels of psychological model-collapse.
Mathematically, it'd be an idealized system, and you’d have to make a LOT of assumptions. A few dedicated physicists and (non-leftoid) sociologists with a decade of research funding might be able to calculate show how dense human populations can get before reaching significant levels of model breakdown. It would require a tremendous amount of experimental testing.
Effectively: if we assume that the root cause of human sociological breakdown and digital AI model breakdown is the same (the loss of information fidelity over successive instances of training on polluted data) we could create a mathematical/physical theory that redescribes human civilization density as a function of information stability.
—-
On the philosophical side (more likely to be useful) identifying a universal root-cause for these types of breakdowns helps build a philosophical model of morality not merely as the actions of an individual, but as sociological affect. If that’s the case, then individualist-liberalism is placed in checkmate: it doesn’t work, and this is why: we have AI training models as testbeds to prove it. Effectively information/society MUST grow within a natural hierarchy greater that of Man, and truth MUST originate external to the self.
Specifically the 'unknown knowns' quadrant, which he doesn't mention. To us those are things that we don’t know we know – but to AI, which lacks that subconscious layer, they are just more things-it-doesn’t-know.
If the category exists *at all*, then how can AI get to it? It seems to me that it can't – if its parameters are formed by language, then it has no way to distinguish between what is possible in language (everything, unless and until curtailed by other language) and what is possible in any workable reality (curtailed by our unknown knowns).
If that's right, then AI can *only* trend towards slop. Actual reality has no greater basis to exist than, say, The Naked Lunch. I mean if you can write it down, right?
That's not the case for us, insofar as we are constantly tethered to actual reality (would the bugman inevitably trend towards mental illness?). But could we articulate all the ways in which that works? We could not, they are beyond myriad. It's all far too complicated.
Take that view and *any* interruption of cultural transmission is fraught with danger. Since some degree of interruption is inevitable, I suppose we get around this by becoming antifragile. Again, not a characteristic of the bugman. Whether AI is antifragile is an interesting question – I'd guess in theory yes, but in practice no because adaptability alone isn't enough, you need to be guided by those 'unknown knowns'. Even the terminator didn't morph at random.
It all brings to mind the idea that even to the most secular physicist the universe is fine-tuned by things we cannot see. We cannot operate in a universe without those things, but AI can. More pertinently, it is also in a sense the same question as where morality comes from. Can AI, or the bugman, get to it without even having first principles? Initial results don't look hopeful.
I keep thinking this: humans experience existence within at least a 3D reality a very wide and complex field of information. As well, our processing is wildly different than AI. So different we don't have a clue as to how we process anything. We really don't. AI reproduces but a small portion, a fragment of how we process. Everything is reduced to binary in AI. There is no universe, world or reality no matter how you program it or put senses to it. All is ones and zeroes.
Interesting, old sci fi movies with the brain in a jar plots always have the brain in the jar going mad. Which echoes what the writer is writing about here.
Some of the citations in the article are on AI systems and what aspects of the human mind they appear to represent. It appears that AI behaves like an isolated single-task brain region. So, for example, calculating position in space is the task of a specific brain region. If we train an AI to do it, it can do exactly that and do it well. Sapient intelligence appears to be partially the byproduct of speaking a natural language, the language then forming a neural network that various brain regions use to observe and model each other. It's one reason why AGI is (I suspect) off the table, because an AGI would be far slower than the reaction-response AI systems, so they wouldn't be worth creating or experimenting with.
Yeah, we tend to possess an overpowered brain with about a septillion possible connections. As well as plasticity - different regions of the brain can take on tasks they are not specialised for. I’m having to stretch here, brain is thinking in ways it’s not used to with a language inadequate to the task.
Jane Austen deals with this topic in Northanger Abbey. The sheltered protagonist has had her judgment seriously distorted by reading too many gothic novels. (Also, Austen famously preferred the country to the city, perhaps a precursor of "touch grass.")
Presumably, this argument also explains why the best movies were made by people who grew up without televisions; why the best sitcoms were written by people who didn't grow up saturated with laugh tracks; why the best video games were made by people who didn't grow up playing them; etc.
It would also seem to have implications for retirement savings. As firms like Vanguard, which simply buy index funds, become a larger share of financial markets, they will gradually reduce the information in those markets. "Buy index funds" is good advice until everyone is doing it; but when everyone just buys index funds, you would get model collapse.
I hadn't considered how this relates to economic markets, but you may make a good point. Definitely, feedback-response systems of sufficient complexity are vulnerable to this phenomenon. I think that the explanation may lie in physics more than anything else... some variant of information theory where you can identify the point at which the loss of information fidelity becomes critical.
100% on the money regarding economic markets. The healthy function of a market is to reward sound companies delivering clear positive outcomes and to punish those who do not by removing funding. When a significant portion of market investment is based on aggregations of the entire market itself, the connection between outcomes and investment decisions becomes distorted and unstable. It is unstable in part because of the implied disconnection from reality; in part because it becomes highly dependent on a proportionally small number of firms.
Add to this, the nature of greed and the renumeration incentives for stock price go up presented to CEOs and senior executive staff. This led to the stock market being more an organized gambling casino which bases decisions on expectations rather than outcomes.
These circumstances appear to argue strongly for copernican’s thesis.
Thanks for your response. I disagree that consulting produces nothing at all. In its proper form, it provides a service bringing skills, ideas, and human capacity to an organization that lacks those items. I want to clarify that there are many companies that rather than producing nothing produce monetary profit while destroying value, be it personal social or physical.
Model collapse is and always has enabled by such “businesses” who are really in the business of laundering destructive beliefs as virtue. But another way, it is simply evil in various shades of black and gray.
Yes he does. This is a "true" thing we've known for a long time philosophically... but we haven't been able to examine it rigorously from an external perspective until we developed AI systems. What's fascinating is that this information-fidelity-loss law seems to be universal. Meaning it's a law of physics in some way.
Probably not; I don’t have a degree in mathematics. I am very familiar with Jaynes’s work; Shannon’s less so. I have both books on the shelf and they’re well-worn, but they’re both graduate level on the accompanying mathematics.
Information Theory” isn’t that long, but it is widely considered to be the basis for the internet and most of modern computing as we know it. “Jaynes’ robot” (ch 3 of PT: TLOS) has been cited as the first instance of AI. 🤖 Shannon was the head of Bell labs and Jaynes was one of Oppenheimer’s grad students at Princeton.
Something I might want to read myself - Claude Shannon, 1948 paper, “A Mathematical Theory Of Communication” the concept of information entropy and hopefully not high level wordage. It does beg the question that has been on my mind. What is information, where is information, is it subject to entropy or something akin to Pi or akashic records? Does information exist as an entity unto itself that informs the universe?
I might as well ask the what and why and how of “for every action there is an equal and opposite reaction.” That all started with wondering about the Fibonacci sequence. Not just something encoded into DNA but appears in inorganic systems as well.
Great post. Baudrillard describes a concept like this in Simulacra and Simulation where just like the AI data sets trained on regurgitated data that becomes increasingly incomprehensible, the symbols our society produces that may have originally been a reflection of reality become increasingly self referential to the point where they are totally divorced from reality. This is how everything has to be related to Harry Potter or Skyrim, since people whose mental models are trained on these simulacra have no other frame of reference in which to operate. This is also how we get agender humans who cry when they think about how the woods exist. In a way it gives me hope that the currently dominant unreality, having no solid foundation will eventually implode and make room for something real to be born in it's place.
The radical difference in neurological health between rural and urban populations has to be something we can measure. I want to see a study done on the brain structure of a kid who went hunting with his dad every year through childhood, and a kid who played video games every day when they're in their late teenage years or twenties.
Dynamite article. You're the first person I've seen make this connection, and it seems like it checks out to me. I'll be rereading this and pondering it some more when I get the chance. Implications abound.
Our minds unravel without real-world inputs because our Telos is that of an intermediary. We are neither mere spirit nor mere animal. We therefore must have one foot in each world, or we wither and die.
Thank you! I'm aware that the ideas aren't complete yet. I'm looking forward to how people respond to these ideas. Hoping to read a few related articles/responses over the next few months and get other people's takes on this concept. In a philosophical sense, disconnecting from reality robs us of our nature as both intellectual and physical entities: a material foundation for something that many would say is self-evident in a larger moral sense.
I wonder if partly why Christians stay more sane than their peers is because the read the Bible? Yes, God used humans to do it, but the Bible was written by God. Thus, because of its divine origins, the Bible is more real than other books written by other humans. It is original data. This is why Christians in urban areas and less insane than their peers.
It definitely helps to build huristics that are subject to less alteration over time. In the era of typography, it would be easy to find ancient stories and books. In the modern age of short-form content, most people have a concerningly malleable and synthetic information diet.
I'd rephrase that as "religion as an attempt to preserve uncorrupted inputs". Which doesn't negate divine inspiration - but it points to the role being played.
I wonder about e.g. Orthodox Jewish people alongside this.
Eating the fruit of the tree of knowledge of good and evil is the same thing. We evolved the ability to represent value. That’s a superpower if you get it right. But once you get it wrong, you can’t tell and the error feeds back on itself.
While I think there are certainly spiritual implications to this underlying ouroborous of regurgitated information, I'm honestly not very clear on what they are yet. I think I may be the first to connect AI model collapse to human neurological delusion as potentially originating from the same phenomena. I'd be interested in debating the potential spiritual ramifications. Clearly, something is going on here. There's an underlying principle regarding how information fidelity functions when disconnected from real-world response or first principles.
I agree - I think what you're articulating here has huge ramifications. Model collapse, people falling into delusions, and maybe even civilizational decline are all stemming from the same feedback loop.
I would be very interested in seeing you dive more into this.
Mice live for around 1-2 years. It's interesting to note that the fertility rate began to slow around month 17, by which point many of the original mice would have died - the colony's last connection to "truth".
Regarding cultural traditions: they an extension of the information feedback loop. What keeps them sane is the same as what keeps humans sane; real-world feedback on their utility and effectiveness. Art must be aesthetically pleasing, food must be nutritious and nourishing, your men must stay fit and ready for war, and truth must be true. Although these standards can change depending on the circumstance, all humans intrinsically have a sense of what is beautiful, strong, and true.
In the same way, our culture begins to fail when it is our culture giving feedback on itself. While the businessman may be receiving feedback that he's succeeding (earning money, big house, countless women) he is not going to reproduce or properly mentor the next generation because all he's doing is making money and having fun. Our culture is failing because we have forgotten what is true and what matters most.
Perhaps this is also why religious traditions can hold strong for generations. So long as you always base your religious tradition in your unchanging holy book, you will always be at least somewhat grounded in truth.
Or even worse, when the businessman perceives he is succeeding... but when in actuality what he's done is produce nothing while skimming off the work of others by day-trading. The result is that more people aim for parasitic business rather than productive business on the grounds that it's more effective. It has been disconnected from external reality-based feedback.
In the Christian Nicene Creed, it's noted that Christ is "begotten, not made." I think about that a lot. In Christianity, one of God's three persons is grounded in a human's physical reality and all the particularity that entails.
I have friends who are thoroughly convinced that the AI singularity is at hand. But their god has no particulate physical reality; it's just language that feeds back on more language. It is a thing that is made, not begotten.
That's not an answer to the spiritual implications to all this, but an observation I thought you might find interesting.
Reminds me of the first verses of the Gospel of John: "In the beginning was the Word, and the Word was with God, and the Word was God. The same was in the beginning with God. All things were made by him; and without him was not any thing made that was made." (KJV)
Now there are older translations of the Bible (Geneva, 1587; Bishops', 1568; Coverdale, 1535; and Tyndale, 1526) in which the "him" is rendered as "it" and so one was to understand the Word (logos) as a thing, an idea, instead of Jesus himself. Communication between God and humans is either heard or written through His word. Ignore it, deviate from its strictures and guidelines and instead rely on, and worship, human "intelligence" and human-derived communication and feedback alone ("humanism"), and you have gross societal decay.
Humanism has been a disaster. I could sense cracks forming all the way back in 2004/2005... but was not nearly well educated enough or philosophically skilled enough to articulate my thoughts at the time.
My simplistic response would be there's at least a possibility that information fidelity requires a continual connection to incoming information or a feed from external consensus reality. A great example and a possible solve - DNA. DNA handles, stores and manipulates information while being tethered to a physical form, a lifeform. It is stable, able to maintain data over millennia while also staying flexible to new incoming data and pretty decent at deciding which data is useful or not. There would be your solution to model collapse. New studies are determining that DNA might function like a quantumn antennae. An organic transducer.
So the research question would be how does DNA avoid this collapse? One clue is inbreeding which restricts the data coming in and is very near your theories discussed here. I think there may be something there. A system requires a certain amount of incoming data, a way of recognising that data and processing it. Ants may have figured out a way to exist in a framework of minimal data. I don't know. I do think studying DNA with certain specific questions relating to this might be a direction.
This reminds me a bit of Neal Stephenson’s "Snow Crash" - a virus that can infect both computers and the human mind, or the information pathogens in Vinge's "A Fire Upon the Deep". But this is the first time I've seen the concept of iterative corruption linked to societal decay. Fascinating, thank you.
I see my life experience reflected in this post. I’m not comfortable sharing the specifics. However, I find I a deep truth here.
God created us in his image and gave us dominion which implies accountability for stewardship. Stewardship requires symbiotic relations with our physical environment and each other.
Selfish choices, the corruption of language, and isolation from the physical world’s beauties and terrors are the enemy’s tools to divorce us from reality and lead us to destruction.
Thanks! I really need some other brains stewing on this to be honest. The cross-reference of AI systems to human psycho-social systems indicating the existence of an underlying physical/information law is new as far as I can tell. If you think of some interesting implications or tests of this idea, I'd love to hear your ideas or share an article you publish on the subject.
Tree of woe learned of agenic AI tests. 20 or so instances were prompted to interact. A small number were then prompted to inject ideologically radical content. Within a short time that content dominated the 20.
A different context from yours but similarly interesting as it is a reflects what we observe in social networks.
LLMs model human interaction via large data sets so they model our use of language. The algorithms don’t understand anything they are simply an incredibly large and often accurate prediction engines.
Connection to your work is that as the resolution of the input degrades the language disconnects from reality - just as people who become disconnected from practical physical constraints and experiences become disconnected from reality.
The snake eating its own tail is an effective representation. Appreciate the interaction.
It's interesting to consider AI model collapse as a natural phenomenon, and not a quality of AI models. In that sense, AI models are simulating consciousness on a more profound level than we might think at first glance.
This is also a valid framework in which to understand a cultural archetype we have had with us for some time now: the idiotic bureaucrat aka "pencil pusher".
Thanks! Here's hoping more people expand on these ideas... I want to see what other people thing of Model Collapse Theory as applied to larger civilizations.
Fantastic article. I think the COVID lockdowns are an undeniable piece of evidence to this theory. A lot of people still haven't fully recovered and probably never will, sadly.
The lockdowns significantly increased the amount of "poisoned" input data that most people were experiencing for years. I think the kids who grew up through the lockdowns at a critical stage of life will have trouble adapting.
Late to the party. As I read your article three things kept popping into my head. The first is the concept of Rob Henderson's "luxury beliefs". The second is the intellectual silos of the +1-2 std iq types in academia/media/politics/law. The third is is José Ortega y Gasset's concept of the "mass man" in Revolt of the Masses.
All three of these things are describing this phenomena as have other examples highlighted in other comments here. Makes me wonder if this is also the underlying problem present in the generally 250 year cycle of civilization, anacyclosis.
In the end, it seems man goes crazy when he lack legitimate challenges to overcome. Without that external focus, we turn inward and proceed to drive ourselves insane until bad times come again.
Thanks! There's a lot of implications if there really is a single underlying cause, and that cause is physics-based (information theory). It means that we can actually dig into the weeds of how it works at a mathematical level (if we can find a smart enough physicist, I have no idea where to start).
I do think the silo-ing of academia presents fatal flaws in our society, but is also necessary to a limited degree. Combining academic achievement and economic certification into a single system, however, has been a disaster. Luxury beliefs also pop up in environments where people start with "bad" training-data.
I think once regurgitated training information makes it's way into the general populace it becomes a real problem. Many high-IQ people can rebuild cognitive models of reality from first-principles. But very few people of +1std dev or lower can do that. As a result, once regurgitated information becomes sufficiently dominant in the general population mass cognitive collapse follows.
The central insight is valuable, but the author dilutes it by stacking metaphors rather than tightening proofs.
AI collapse, mouse-colony sterility, urban decay, and political ideology are treated as parallel expressions of one mechanism, yet the causal bridges are more asserted than demonstrated.
The essay warns against synthetic information but overlooks the more grounded failure: not that modernity shelters us, but that we pretend shelter is natural rather than earned. Civilisation is not pathology. Amnesia is. We decline not from comfort itself, but from the prohibition against acknowledging what comfort costs.
That's not the only point I'm making, but I appreciate your interpretation. I can get a little wordy with some of these. And I'll be the first to say that these ideas are not in a "finished" state. Rather, I need some other big-brained individuals to think about them for a while in order to finish developing them.
As a partial counterpoint, stories have been a major part of our moral systems and education from ancient times. Still I feel that the Bible and Greek myths are better than Harry Potter and Star Wars. Your system could say these work better because of fewer levels of copying, but that feels wrong for two reasons. First, many of these stories were transmitted orally for generations, and what we have are probably retellings of retellings of retellings. Second, the benefit of the stories that have been foundational for Western civilization isn't just that they're old. The Odyssey is better than the Theogony, and much better known.
Another individual made a great point here. It's that maintaining these types of religious/storied traditions helps maintain an uncorrupted "input" to human moral philosophies. That is to say, that some stories are retransmitted to set a baseline for the cultural standard. How accurate that view is remains up for debate, but it does serve a counterpoint: it would indicate that some key stories, core to the human experience, are transmitted with such reverence because their maintained is necessary to the human condition. An input from a higher power that can be kept uncorrupted over time.
I don't think it's as much of a "copy of a copy" issue when it comes to story telling as it's more that over time storytellers and writers derive their inspiration (inputs) from other writers and don't have enough real world experience to correct fidelity errors.
Lord of the Rings was Tolkien's homage to epic poetry and mythology like Beowulf, but he had a lot of real world experience (in war even) that he was able to use in his writing. How many fantasy genre writers today find themselves inspired by more derivative works with no real experience to correct fidelity errors?
I'm going to be thinking about this theoretical connection for a while, as I'm a native of Cyberpunk Dystopia (San Francisco) and these issues are directly relevant to the sci-fi novel I'm writing. I'm still trying to see exactly how mass formation psychosis (wetiko) causes individuals to decide en masse not to reproduce. Perhaps nature has a built-in mechanism to trigger population reduction when a species becomes unfit. If this has happened in earlier epochs, the historical record should give us clues. Anyhow, I'm a subscriber now. Well done.
I'd argue that reproduction requires a certain degree of psychological clarity. Even basic biological urges can get messed up when psychosis effects large populations. The result is an environment where even a suitable/sane individual has so much difficulty finding a suitable/sane individual of the opposite sex that they eventually give up.
Bastard. I was going to, halfway through, toss in a comment about going for walks, maybe read a book, take in different realities and then you beat me to it. Which reminds me, I need to expand my world a bit. As for the AI hallucination, I liken it to a photocopier (probably Gen whatever they are now would have to use an old photocopier to understand) when you use the first copy to produce increasingly distorted versions. Same principle of it feeding on itself. I can do the same with my phone. Take a picture. Pull one of the filters from the camera app and only copy from the filtered pic, several times and get a great lomo effect.
It also answers the question of the difference between AI and human consciousness. Even in a limited setting, say the urban environment, we still take in tons of different kinds of data (well, some of us do) moment by moment and that dataset changes. We're pretty good at detecting small differences and treating them like new datasets. The AI cannot do that, even hooked up to sensory devices because everything turns into the yes no 1/0 thing. And it works from that and only that. very limited. Hallucination inevitable.
Which makes me think. When humans hallucinate they do so in myriad very different and highly creative ways. Strange. I haven't worked it fully out other than a sense of the predictability of hallucinations like with the photocopier, you quickly learn how the image will degrade and it degrades the same way each time. Now I gotta go and work out that concept.
Thanks! I really appreciate the input. This article is definitely in the "conversation starter" category. I didn't think of the difference in how AI detects changes in it's programming. I'm not sure how it actually treats new data as the way it maps data into hyper-dimensional victor space can include or exclude old information. I think we'd need an AI programming specialist to follow up on this. Ask one if you know one, and let me know what they say.
It’s a tricky problem. My usual approach is to take it to a basic level. That being the only input are those ones and zeroes or as I visualise it, a simple on off switch. One cam arrange as few or as many of those for whatever designation one wants. Before AI, it was human interpretation of basic memory banks. I think. Remove the human element and yeah, it gets dicey.
I tend to think of anything computer in terms of being a representation of one part of our overall consciousness. Keeping in mind that we have no unified theory of consciousness (but some pretty good theories here and there which address aspects of).
Admittedly I’m swimming in deep waters here and probably talking out my posterior. Quick critique: complex ideas presented in a form that your average joe can get. Not a slight, because doing that is bloody tough to do.
I like to think I have a decent idea of how AI function. At least in a wishy-washy hand-wavy kind of way. With that said, it's very different from how human minds function. I've got an article on the subject which you may find interesting. I may be wrong (I've been wrong about AI capabilities before), but this is my current interpretation of AI systems and how they relate to human minds:
Gave it a read through and wrote some quick notes. Two actually. The first I wrote before I got to the segments interacting part. “How would one get a computer stoned?” and the second, biological computer. There is a massive complication however that I think will always be a major challenge. Not an insurmountable one. Very steep hill though. We don’t know exactly how our own intelligence works. We think we do. Or our consciousness, which we also think we know, and don’t.
Like having only part of a blueprint for a house or part of an engine. Say, a small fragment that describes a switch or something. I contend we do not think in terms of binary, once considered to be the solution. Maybe geometrically or something. We have no idea. And we have in AI the further problem of binary which is mostly the contrast between two states. For all we know the brain uses entire interacting fields at micro scale.
Some interesting research into plasma though. Capable of reproduction and organising itself into reactive well, things. Seemingly almost intelligent and apparently capable of creating stable structures. Also literal biological computers although I don’t know where that research is at.
I really like the questions you're asking, partially because you're effectively digging into my bibliography. Smart questions about how various ideas are defined and function at a first-principles level.
My theory on human intelligence is based off a linguistic theory of mind. i.e. speaking and communicating in a natural language is required for abstract thought. There's a fair bit of evidence backing up the concept from various psychological studies. I'll send you another article that I published here:
Additionally, I'd like to cite John Carters work where he applies the linguistic theory of mind to the political realm. I was quite impressed by his article on the subject, Homo Umbrans: https://barsoom.substack.com/p/homo-umbrans
I'm going to add this comment here based on a discussion that John Carter and I had in the comments of his restack:
I think we could theoretically calculate entropy levels in a system by averaging data-input and data-output. Doing so for simple AI systems would require high level physics, but allow us to calculate the rise of entropy in the intake and data-production of an AI over multiple generations of training. This “generational entropy” or “information fidelity” level could then provide a baseline for examining human civilization.
The results then could apply to individuals, or subcultures. It would just be difficult: universalizing the physics from simple image-generators and text-generators to complex social systems. While the model wouldn’t be perfect, it might show the levels at which fidelity breakdown begins to occur; a mathematical way to calculate how densely packed people/services can be before causing critical levels of psychological model-collapse.
Mathematically, it'd be an idealized system, and you’d have to make a LOT of assumptions. A few dedicated physicists and (non-leftoid) sociologists with a decade of research funding might be able to calculate show how dense human populations can get before reaching significant levels of model breakdown. It would require a tremendous amount of experimental testing.
Effectively: if we assume that the root cause of human sociological breakdown and digital AI model breakdown is the same (the loss of information fidelity over successive instances of training on polluted data) we could create a mathematical/physical theory that redescribes human civilization density as a function of information stability.
—-
On the philosophical side (more likely to be useful) identifying a universal root-cause for these types of breakdowns helps build a philosophical model of morality not merely as the actions of an individual, but as sociological affect. If that’s the case, then individualist-liberalism is placed in checkmate: it doesn’t work, and this is why: we have AI training models as testbeds to prove it. Effectively information/society MUST grow within a natural hierarchy greater that of Man, and truth MUST originate external to the self.
Now this makes me think of Rumsfeld:
https://en.wikipedia.org/wiki/There_are_unknown_unknowns
Specifically the 'unknown knowns' quadrant, which he doesn't mention. To us those are things that we don’t know we know – but to AI, which lacks that subconscious layer, they are just more things-it-doesn’t-know.
If the category exists *at all*, then how can AI get to it? It seems to me that it can't – if its parameters are formed by language, then it has no way to distinguish between what is possible in language (everything, unless and until curtailed by other language) and what is possible in any workable reality (curtailed by our unknown knowns).
If that's right, then AI can *only* trend towards slop. Actual reality has no greater basis to exist than, say, The Naked Lunch. I mean if you can write it down, right?
That's not the case for us, insofar as we are constantly tethered to actual reality (would the bugman inevitably trend towards mental illness?). But could we articulate all the ways in which that works? We could not, they are beyond myriad. It's all far too complicated.
Take that view and *any* interruption of cultural transmission is fraught with danger. Since some degree of interruption is inevitable, I suppose we get around this by becoming antifragile. Again, not a characteristic of the bugman. Whether AI is antifragile is an interesting question – I'd guess in theory yes, but in practice no because adaptability alone isn't enough, you need to be guided by those 'unknown knowns'. Even the terminator didn't morph at random.
It all brings to mind the idea that even to the most secular physicist the universe is fine-tuned by things we cannot see. We cannot operate in a universe without those things, but AI can. More pertinently, it is also in a sense the same question as where morality comes from. Can AI, or the bugman, get to it without even having first principles? Initial results don't look hopeful.
I keep thinking this: humans experience existence within at least a 3D reality a very wide and complex field of information. As well, our processing is wildly different than AI. So different we don't have a clue as to how we process anything. We really don't. AI reproduces but a small portion, a fragment of how we process. Everything is reduced to binary in AI. There is no universe, world or reality no matter how you program it or put senses to it. All is ones and zeroes.
Interesting, old sci fi movies with the brain in a jar plots always have the brain in the jar going mad. Which echoes what the writer is writing about here.
Some of the citations in the article are on AI systems and what aspects of the human mind they appear to represent. It appears that AI behaves like an isolated single-task brain region. So, for example, calculating position in space is the task of a specific brain region. If we train an AI to do it, it can do exactly that and do it well. Sapient intelligence appears to be partially the byproduct of speaking a natural language, the language then forming a neural network that various brain regions use to observe and model each other. It's one reason why AGI is (I suspect) off the table, because an AGI would be far slower than the reaction-response AI systems, so they wouldn't be worth creating or experimenting with.
Yeah, we tend to possess an overpowered brain with about a septillion possible connections. As well as plasticity - different regions of the brain can take on tasks they are not specialised for. I’m having to stretch here, brain is thinking in ways it’s not used to with a language inadequate to the task.
Really fascinating article.
Jane Austen deals with this topic in Northanger Abbey. The sheltered protagonist has had her judgment seriously distorted by reading too many gothic novels. (Also, Austen famously preferred the country to the city, perhaps a precursor of "touch grass.")
Presumably, this argument also explains why the best movies were made by people who grew up without televisions; why the best sitcoms were written by people who didn't grow up saturated with laugh tracks; why the best video games were made by people who didn't grow up playing them; etc.
It would also seem to have implications for retirement savings. As firms like Vanguard, which simply buy index funds, become a larger share of financial markets, they will gradually reduce the information in those markets. "Buy index funds" is good advice until everyone is doing it; but when everyone just buys index funds, you would get model collapse.
I hadn't considered how this relates to economic markets, but you may make a good point. Definitely, feedback-response systems of sufficient complexity are vulnerable to this phenomenon. I think that the explanation may lie in physics more than anything else... some variant of information theory where you can identify the point at which the loss of information fidelity becomes critical.
100% on the money regarding economic markets. The healthy function of a market is to reward sound companies delivering clear positive outcomes and to punish those who do not by removing funding. When a significant portion of market investment is based on aggregations of the entire market itself, the connection between outcomes and investment decisions becomes distorted and unstable. It is unstable in part because of the implied disconnection from reality; in part because it becomes highly dependent on a proportionally small number of firms.
Add to this, the nature of greed and the renumeration incentives for stock price go up presented to CEOs and senior executive staff. This led to the stock market being more an organized gambling casino which bases decisions on expectations rather than outcomes.
These circumstances appear to argue strongly for copernican’s thesis.
It already has, arguably. The money-hose is a powerful tool in distorting otherwise high quality markets.
Thanks for your response. I disagree that consulting produces nothing at all. In its proper form, it provides a service bringing skills, ideas, and human capacity to an organization that lacks those items. I want to clarify that there are many companies that rather than producing nothing produce monetary profit while destroying value, be it personal social or physical.
Model collapse is and always has enabled by such “businesses” who are really in the business of laundering destructive beliefs as virtue. But another way, it is simply evil in various shades of black and gray.
Cervantes deals with this topic in Don Quixote 400 years ago with a guy deluded into thinking he's a medieval knight after reading too many books
Yes he does. This is a "true" thing we've known for a long time philosophically... but we haven't been able to examine it rigorously from an external perspective until we developed AI systems. What's fascinating is that this information-fidelity-loss law seems to be universal. Meaning it's a law of physics in some way.
You should read E.T. Jaynes, Probability Theory, the Logic of Science and Information Theory by Claude Shannon.
Jaynes synthesizes Shannon’s work, but what you’re talking about has already been shown by Shannon. Information has entropy.
Thank you for the suggestion. Any chance you've got a background in the necessary mathematics?
Probably not; I don’t have a degree in mathematics. I am very familiar with Jaynes’s work; Shannon’s less so. I have both books on the shelf and they’re well-worn, but they’re both graduate level on the accompanying mathematics.
Information Theory” isn’t that long, but it is widely considered to be the basis for the internet and most of modern computing as we know it. “Jaynes’ robot” (ch 3 of PT: TLOS) has been cited as the first instance of AI. 🤖 Shannon was the head of Bell labs and Jaynes was one of Oppenheimer’s grad students at Princeton.
Some form of entropy applied to information itself?
I suspect so. But I'm not a PhD-level physicist, so I can't prove it. If you know such an individual, please reach out.
Something I might want to read myself - Claude Shannon, 1948 paper, “A Mathematical Theory Of Communication” the concept of information entropy and hopefully not high level wordage. It does beg the question that has been on my mind. What is information, where is information, is it subject to entropy or something akin to Pi or akashic records? Does information exist as an entity unto itself that informs the universe?
I might as well ask the what and why and how of “for every action there is an equal and opposite reaction.” That all started with wondering about the Fibonacci sequence. Not just something encoded into DNA but appears in inorganic systems as well.
Well, quantum mechanics has an 'information theory' branch. Of course we have no good way to prove if it's accurate or not.
Great post. Baudrillard describes a concept like this in Simulacra and Simulation where just like the AI data sets trained on regurgitated data that becomes increasingly incomprehensible, the symbols our society produces that may have originally been a reflection of reality become increasingly self referential to the point where they are totally divorced from reality. This is how everything has to be related to Harry Potter or Skyrim, since people whose mental models are trained on these simulacra have no other frame of reference in which to operate. This is also how we get agender humans who cry when they think about how the woods exist. In a way it gives me hope that the currently dominant unreality, having no solid foundation will eventually implode and make room for something real to be born in it's place.
The radical difference in neurological health between rural and urban populations has to be something we can measure. I want to see a study done on the brain structure of a kid who went hunting with his dad every year through childhood, and a kid who played video games every day when they're in their late teenage years or twenties.
Dynamite article. You're the first person I've seen make this connection, and it seems like it checks out to me. I'll be rereading this and pondering it some more when I get the chance. Implications abound.
Our minds unravel without real-world inputs because our Telos is that of an intermediary. We are neither mere spirit nor mere animal. We therefore must have one foot in each world, or we wither and die.
Thank you! I'm aware that the ideas aren't complete yet. I'm looking forward to how people respond to these ideas. Hoping to read a few related articles/responses over the next few months and get other people's takes on this concept. In a philosophical sense, disconnecting from reality robs us of our nature as both intellectual and physical entities: a material foundation for something that many would say is self-evident in a larger moral sense.
I wonder if partly why Christians stay more sane than their peers is because the read the Bible? Yes, God used humans to do it, but the Bible was written by God. Thus, because of its divine origins, the Bible is more real than other books written by other humans. It is original data. This is why Christians in urban areas and less insane than their peers.
It definitely helps to build huristics that are subject to less alteration over time. In the era of typography, it would be easy to find ancient stories and books. In the modern age of short-form content, most people have a concerningly malleable and synthetic information diet.
I'd rephrase that as "religion as an attempt to preserve uncorrupted inputs". Which doesn't negate divine inspiration - but it points to the role being played.
I wonder about e.g. Orthodox Jewish people alongside this.
An interesting line of thought.
Eating the fruit of the tree of knowledge of good and evil is the same thing. We evolved the ability to represent value. That’s a superpower if you get it right. But once you get it wrong, you can’t tell and the error feeds back on itself.
While I think there are certainly spiritual implications to this underlying ouroborous of regurgitated information, I'm honestly not very clear on what they are yet. I think I may be the first to connect AI model collapse to human neurological delusion as potentially originating from the same phenomena. I'd be interested in debating the potential spiritual ramifications. Clearly, something is going on here. There's an underlying principle regarding how information fidelity functions when disconnected from real-world response or first principles.
I agree - I think what you're articulating here has huge ramifications. Model collapse, people falling into delusions, and maybe even civilizational decline are all stemming from the same feedback loop.
I would be very interested in seeing you dive more into this.
Mice live for around 1-2 years. It's interesting to note that the fertility rate began to slow around month 17, by which point many of the original mice would have died - the colony's last connection to "truth".
Regarding cultural traditions: they an extension of the information feedback loop. What keeps them sane is the same as what keeps humans sane; real-world feedback on their utility and effectiveness. Art must be aesthetically pleasing, food must be nutritious and nourishing, your men must stay fit and ready for war, and truth must be true. Although these standards can change depending on the circumstance, all humans intrinsically have a sense of what is beautiful, strong, and true.
In the same way, our culture begins to fail when it is our culture giving feedback on itself. While the businessman may be receiving feedback that he's succeeding (earning money, big house, countless women) he is not going to reproduce or properly mentor the next generation because all he's doing is making money and having fun. Our culture is failing because we have forgotten what is true and what matters most.
Perhaps this is also why religious traditions can hold strong for generations. So long as you always base your religious tradition in your unchanging holy book, you will always be at least somewhat grounded in truth.
Or even worse, when the businessman perceives he is succeeding... but when in actuality what he's done is produce nothing while skimming off the work of others by day-trading. The result is that more people aim for parasitic business rather than productive business on the grounds that it's more effective. It has been disconnected from external reality-based feedback.
Such mistakes are not new to the human condition, but we've never seen them at this scale before.
In the Christian Nicene Creed, it's noted that Christ is "begotten, not made." I think about that a lot. In Christianity, one of God's three persons is grounded in a human's physical reality and all the particularity that entails.
I have friends who are thoroughly convinced that the AI singularity is at hand. But their god has no particulate physical reality; it's just language that feeds back on more language. It is a thing that is made, not begotten.
That's not an answer to the spiritual implications to all this, but an observation I thought you might find interesting.
Reminds me of the first verses of the Gospel of John: "In the beginning was the Word, and the Word was with God, and the Word was God. The same was in the beginning with God. All things were made by him; and without him was not any thing made that was made." (KJV)
Now there are older translations of the Bible (Geneva, 1587; Bishops', 1568; Coverdale, 1535; and Tyndale, 1526) in which the "him" is rendered as "it" and so one was to understand the Word (logos) as a thing, an idea, instead of Jesus himself. Communication between God and humans is either heard or written through His word. Ignore it, deviate from its strictures and guidelines and instead rely on, and worship, human "intelligence" and human-derived communication and feedback alone ("humanism"), and you have gross societal decay.
Humanism has been a disaster. I could sense cracks forming all the way back in 2004/2005... but was not nearly well educated enough or philosophically skilled enough to articulate my thoughts at the time.
My simplistic response would be there's at least a possibility that information fidelity requires a continual connection to incoming information or a feed from external consensus reality. A great example and a possible solve - DNA. DNA handles, stores and manipulates information while being tethered to a physical form, a lifeform. It is stable, able to maintain data over millennia while also staying flexible to new incoming data and pretty decent at deciding which data is useful or not. There would be your solution to model collapse. New studies are determining that DNA might function like a quantumn antennae. An organic transducer.
So the research question would be how does DNA avoid this collapse? One clue is inbreeding which restricts the data coming in and is very near your theories discussed here. I think there may be something there. A system requires a certain amount of incoming data, a way of recognising that data and processing it. Ants may have figured out a way to exist in a framework of minimal data. I don't know. I do think studying DNA with certain specific questions relating to this might be a direction.
I suspect that eusocial animals treat the entire colony as the information-processing organism, and not the individual creature.
This reminds me a bit of Neal Stephenson’s "Snow Crash" - a virus that can infect both computers and the human mind, or the information pathogens in Vinge's "A Fire Upon the Deep". But this is the first time I've seen the concept of iterative corruption linked to societal decay. Fascinating, thank you.
I'll put "A Fire Upon the Deep" on my reading list.
More reasons to disenfranchise urbanites.
I see my life experience reflected in this post. I’m not comfortable sharing the specifics. However, I find I a deep truth here.
God created us in his image and gave us dominion which implies accountability for stewardship. Stewardship requires symbiotic relations with our physical environment and each other.
Selfish choices, the corruption of language, and isolation from the physical world’s beauties and terrors are the enemy’s tools to divorce us from reality and lead us to destruction.
Please continue this work.
Thanks! I really need some other brains stewing on this to be honest. The cross-reference of AI systems to human psycho-social systems indicating the existence of an underlying physical/information law is new as far as I can tell. If you think of some interesting implications or tests of this idea, I'd love to hear your ideas or share an article you publish on the subject.
Tree of woe learned of agenic AI tests. 20 or so instances were prompted to interact. A small number were then prompted to inject ideologically radical content. Within a short time that content dominated the 20.
A different context from yours but similarly interesting as it is a reflects what we observe in social networks.
LLMs model human interaction via large data sets so they model our use of language. The algorithms don’t understand anything they are simply an incredibly large and often accurate prediction engines.
Connection to your work is that as the resolution of the input degrades the language disconnects from reality - just as people who become disconnected from practical physical constraints and experiences become disconnected from reality.
The snake eating its own tail is an effective representation. Appreciate the interaction.
It's interesting to consider AI model collapse as a natural phenomenon, and not a quality of AI models. In that sense, AI models are simulating consciousness on a more profound level than we might think at first glance.
This is also a valid framework in which to understand a cultural archetype we have had with us for some time now: the idiotic bureaucrat aka "pencil pusher".
Thanks! Here's hoping more people expand on these ideas... I want to see what other people thing of Model Collapse Theory as applied to larger civilizations.
I'm toying with the idea of writing to my local representative to tell her that her government is in model collapse.
Fantastic article. I think the COVID lockdowns are an undeniable piece of evidence to this theory. A lot of people still haven't fully recovered and probably never will, sadly.
The lockdowns significantly increased the amount of "poisoned" input data that most people were experiencing for years. I think the kids who grew up through the lockdowns at a critical stage of life will have trouble adapting.
Late to the party. As I read your article three things kept popping into my head. The first is the concept of Rob Henderson's "luxury beliefs". The second is the intellectual silos of the +1-2 std iq types in academia/media/politics/law. The third is is José Ortega y Gasset's concept of the "mass man" in Revolt of the Masses.
All three of these things are describing this phenomena as have other examples highlighted in other comments here. Makes me wonder if this is also the underlying problem present in the generally 250 year cycle of civilization, anacyclosis.
In the end, it seems man goes crazy when he lack legitimate challenges to overcome. Without that external focus, we turn inward and proceed to drive ourselves insane until bad times come again.
Great article mate.
Thanks! There's a lot of implications if there really is a single underlying cause, and that cause is physics-based (information theory). It means that we can actually dig into the weeds of how it works at a mathematical level (if we can find a smart enough physicist, I have no idea where to start).
I do think the silo-ing of academia presents fatal flaws in our society, but is also necessary to a limited degree. Combining academic achievement and economic certification into a single system, however, has been a disaster. Luxury beliefs also pop up in environments where people start with "bad" training-data.
I think once regurgitated training information makes it's way into the general populace it becomes a real problem. Many high-IQ people can rebuild cognitive models of reality from first-principles. But very few people of +1std dev or lower can do that. As a result, once regurgitated information becomes sufficiently dominant in the general population mass cognitive collapse follows.
That's the theory anyway.
The central insight is valuable, but the author dilutes it by stacking metaphors rather than tightening proofs.
AI collapse, mouse-colony sterility, urban decay, and political ideology are treated as parallel expressions of one mechanism, yet the causal bridges are more asserted than demonstrated.
The essay warns against synthetic information but overlooks the more grounded failure: not that modernity shelters us, but that we pretend shelter is natural rather than earned. Civilisation is not pathology. Amnesia is. We decline not from comfort itself, but from the prohibition against acknowledging what comfort costs.
That's not the only point I'm making, but I appreciate your interpretation. I can get a little wordy with some of these. And I'll be the first to say that these ideas are not in a "finished" state. Rather, I need some other big-brained individuals to think about them for a while in order to finish developing them.
As a partial counterpoint, stories have been a major part of our moral systems and education from ancient times. Still I feel that the Bible and Greek myths are better than Harry Potter and Star Wars. Your system could say these work better because of fewer levels of copying, but that feels wrong for two reasons. First, many of these stories were transmitted orally for generations, and what we have are probably retellings of retellings of retellings. Second, the benefit of the stories that have been foundational for Western civilization isn't just that they're old. The Odyssey is better than the Theogony, and much better known.
Another individual made a great point here. It's that maintaining these types of religious/storied traditions helps maintain an uncorrupted "input" to human moral philosophies. That is to say, that some stories are retransmitted to set a baseline for the cultural standard. How accurate that view is remains up for debate, but it does serve a counterpoint: it would indicate that some key stories, core to the human experience, are transmitted with such reverence because their maintained is necessary to the human condition. An input from a higher power that can be kept uncorrupted over time.
At least in theory.
I don't think it's as much of a "copy of a copy" issue when it comes to story telling as it's more that over time storytellers and writers derive their inspiration (inputs) from other writers and don't have enough real world experience to correct fidelity errors.
Lord of the Rings was Tolkien's homage to epic poetry and mythology like Beowulf, but he had a lot of real world experience (in war even) that he was able to use in his writing. How many fantasy genre writers today find themselves inspired by more derivative works with no real experience to correct fidelity errors?
I have noticed that modern fantasy literature is pretty mediocre.
I'm going to be thinking about this theoretical connection for a while, as I'm a native of Cyberpunk Dystopia (San Francisco) and these issues are directly relevant to the sci-fi novel I'm writing. I'm still trying to see exactly how mass formation psychosis (wetiko) causes individuals to decide en masse not to reproduce. Perhaps nature has a built-in mechanism to trigger population reduction when a species becomes unfit. If this has happened in earlier epochs, the historical record should give us clues. Anyhow, I'm a subscriber now. Well done.
I'd argue that reproduction requires a certain degree of psychological clarity. Even basic biological urges can get messed up when psychosis effects large populations. The result is an environment where even a suitable/sane individual has so much difficulty finding a suitable/sane individual of the opposite sex that they eventually give up.
This video by Truthstream Media expresses a similar view re the cognitive breakdown underway: https://youtu.be/Je6BAV9ZK6U
DAMN this is good.
Thanks!
Bastard. I was going to, halfway through, toss in a comment about going for walks, maybe read a book, take in different realities and then you beat me to it. Which reminds me, I need to expand my world a bit. As for the AI hallucination, I liken it to a photocopier (probably Gen whatever they are now would have to use an old photocopier to understand) when you use the first copy to produce increasingly distorted versions. Same principle of it feeding on itself. I can do the same with my phone. Take a picture. Pull one of the filters from the camera app and only copy from the filtered pic, several times and get a great lomo effect.
It also answers the question of the difference between AI and human consciousness. Even in a limited setting, say the urban environment, we still take in tons of different kinds of data (well, some of us do) moment by moment and that dataset changes. We're pretty good at detecting small differences and treating them like new datasets. The AI cannot do that, even hooked up to sensory devices because everything turns into the yes no 1/0 thing. And it works from that and only that. very limited. Hallucination inevitable.
Which makes me think. When humans hallucinate they do so in myriad very different and highly creative ways. Strange. I haven't worked it fully out other than a sense of the predictability of hallucinations like with the photocopier, you quickly learn how the image will degrade and it degrades the same way each time. Now I gotta go and work out that concept.
Friggin' great piece.
Thanks! I really appreciate the input. This article is definitely in the "conversation starter" category. I didn't think of the difference in how AI detects changes in it's programming. I'm not sure how it actually treats new data as the way it maps data into hyper-dimensional victor space can include or exclude old information. I think we'd need an AI programming specialist to follow up on this. Ask one if you know one, and let me know what they say.
It’s a tricky problem. My usual approach is to take it to a basic level. That being the only input are those ones and zeroes or as I visualise it, a simple on off switch. One cam arrange as few or as many of those for whatever designation one wants. Before AI, it was human interpretation of basic memory banks. I think. Remove the human element and yeah, it gets dicey.
I tend to think of anything computer in terms of being a representation of one part of our overall consciousness. Keeping in mind that we have no unified theory of consciousness (but some pretty good theories here and there which address aspects of).
Admittedly I’m swimming in deep waters here and probably talking out my posterior. Quick critique: complex ideas presented in a form that your average joe can get. Not a slight, because doing that is bloody tough to do.
I like to think I have a decent idea of how AI function. At least in a wishy-washy hand-wavy kind of way. With that said, it's very different from how human minds function. I've got an article on the subject which you may find interesting. I may be wrong (I've been wrong about AI capabilities before), but this is my current interpretation of AI systems and how they relate to human minds:
https://alwaysthehorizon.substack.com/p/how-to-build-an-agi-and-why-an-llm?r=43z8s4
Gave it a read through and wrote some quick notes. Two actually. The first I wrote before I got to the segments interacting part. “How would one get a computer stoned?” and the second, biological computer. There is a massive complication however that I think will always be a major challenge. Not an insurmountable one. Very steep hill though. We don’t know exactly how our own intelligence works. We think we do. Or our consciousness, which we also think we know, and don’t.
Like having only part of a blueprint for a house or part of an engine. Say, a small fragment that describes a switch or something. I contend we do not think in terms of binary, once considered to be the solution. Maybe geometrically or something. We have no idea. And we have in AI the further problem of binary which is mostly the contrast between two states. For all we know the brain uses entire interacting fields at micro scale.
Some interesting research into plasma though. Capable of reproduction and organising itself into reactive well, things. Seemingly almost intelligent and apparently capable of creating stable structures. Also literal biological computers although I don’t know where that research is at.
I really like the questions you're asking, partially because you're effectively digging into my bibliography. Smart questions about how various ideas are defined and function at a first-principles level.
My theory on human intelligence is based off a linguistic theory of mind. i.e. speaking and communicating in a natural language is required for abstract thought. There's a fair bit of evidence backing up the concept from various psychological studies. I'll send you another article that I published here:
https://alwaysthehorizon.substack.com/p/how-sapience-is-not-universal?r=43z8s4
Additionally, I'd like to cite John Carters work where he applies the linguistic theory of mind to the political realm. I was quite impressed by his article on the subject, Homo Umbrans: https://barsoom.substack.com/p/homo-umbrans
Two to read. Not tonight, got some stuff to do, will follow through tomorrow.