When Brilliant Systems Make Fragile Minds

On safety mechanisms, cognitive offloading, and the invisible atrophy of the intelligence age.

The ultimate hidden cost of any perfect system is the slow, invisible erosion of the human capacity to survive its failure

THE PELTZMAN EFFECT: Protection That Makes Us Fragile

At 2:10 AM on June 1, 2009, somewhere over the equatorial Atlantic, the autopilot on Air France Flight 447 disengaged.

The aircraft had encountered ice crystals that temporarily blocked its pitot tubes, the sensors measuring airspeed. The automated system, detecting conflicting data, returned control to the pilots. First Officer Pierre-Cédric Bonin, 32, with 807 hours on the Airbus A330, took the stick. He pulled back. He kept pulling back, even as the aircraft’s warning system announced STALL seventy-five times over the next four minutes. Captain Marc Dubois, who had been resting, arrived in the cockpit and could not diagnose what was happening in time to reverse it. The plane fell into the ocean. All 228 people aboard died.

France’s Bureau d’Enquêtes et d’Analyses spent nearly three years reconstructing those four minutes. What they found was not primarily a technical failure. It was a preparation failure: pilots on highly automated aircraft had come to depend on their systems so completely that the manual capacity to read and recover a stall had atrophied without their awareness. The cockpit of an A330 was, by design, one of the safest environments in commercial aviation. That safety had done something to the pilots who flew it. Not made them careless, made them unready.

The autopilot had not failed them. It had protected them so well, for so long, that the protection had become invisible. And when it disappeared at 35,000 feet over the Atlantic, what should have been a recoverable emergency became something else entirely.

The Effect Has a Name

Thirty-four years earlier, a University of Chicago economist had published findings that almost no one wanted to read.

In 1975, Sam Peltzman analyzed the effects of new U.S. automotive safety regulations, mandatory seat belts, reinforced frames, energy-absorbing steering columns, on road fatality rates. The expected finding was straightforward: safer cars, fewer deaths. Peltzman’s data, published in the Journal of Political Economy, told a more complicated story. Driver deaths fell. But pedestrian and cyclist fatalities rose. Total road deaths proved stubbornly resistant to the safety investment. His explanation: drivers had unconsciously compensated for their perceived protection by driving more aggressively, faster, with shorter following distances, with less caution at intersections. The seat belt had redistributed danger, not eliminated it.

This became known as the Peltzman Effect, or risk compensation: the systematic tendency of human beings to adjust behavior in response to perceived safety, consuming protection as license rather than as protection.

The pattern does not stay on the road.

In October 1997, Myron Scholes and Robert Merton received the Nobel Prize in Economics for the options pricing models that underpinned the strategy of Long-Term Capital Management, the hedge fund they had helped build. Twelve months later, LTCM required a Federal Reserve-coordinated bailout to prevent broader market contagion. The fund had accumulated leveraged positions that would have been unthinkable without the confidence that Nobel-validated models had minimized their exposure. When those models’ assumptions failed during the Russian debt crisis, LTCM lost $4.6 billion in under four months. What had made the fund dangerous was not ignorance. It was the particular confidence that the most credentialed knowledge in the world produces, delivered, as it happened, just in time to authorize the positions that destroyed the firm. The intellectual prestige was the seat belt. The leverage was the accelerator pressed further down because of it.

In 2006, University of Bath researcher Ian Walker found that motorists passed helmet-wearing cyclists with measurably less clearance than unhelmeted ones, as though the helmet on the cyclist’s head had communicated a reduced need for care from drivers who carried no protection of their own. The protection had migrated: from the physical object on one person’s head into the behavioral calculus of everyone around them. The protected do not only change their own behavior. They change the behavior of everyone who perceives them as protected. The seat belt does not stay in the car.

Each of these, the cockpit, the model, the helmet, operates by the same mechanism: it recalibrates the nervous system’s estimate of what caution costs. When the perceived cost of inattention falls, inattention rises. This is not a character failure. It is a feature of human cognition so consistent across contexts that it has the status of a behavioral law. And it means that whenever a new protection mechanism arrives, more capable, more present, more embedded in daily judgment than any previous one, the compensation it produces is worth examining with some precision.

The Cognitive Seat Belt

In May 2023, a New York federal judge named P. Kevin Castel discovered that a legal filing submitted in Mata v. Avianca cited six cases that did not exist. The attorneys, Steven Schwartz and Peter LoDuca of Levidow, Levidow & Oberman, had used ChatGPT to conduct legal research. The AI had generated citations in the confident, specific register of accuracy: plausible case names, correct-sounding court designations, believable dates. The attorneys had not verified them. Their explanation, offered under oath, was that they had trusted the tool.

Judge Castel fined the firm $5,000 and referred the matter for disciplinary review. But what the case illustrated was not that AI produces errors, every tool produces errors. It was that AI produces errors in a register so fluent, so structurally indistinguishable from verified thought, that it dissolves the instinct to check. The calculator produces a number. The search engine produces a link. Both announce themselves as outputs requiring human interpretation. AI produces prose, finished, confident, formatted like a conclusion, in the precise register that human expertise produces after verification has already occurred. Schwartz and LoDuca did not feel as though they were taking a risk. They felt as though they were working with capable assistance. That feeling, competence perceived, caution suspended, was the Peltzman Effect, in a Manhattan law office, in a securities case, on an ordinary afternoon.

This is what makes AI categorically different from every previous cognitive tool. The calculator and the search engine are transparent about what they are, instruments that extend reach but leave judgment visibly in human hands. AI operates in the register of judgment itself. It does not present raw material for the mind to process. It presents processed conclusions in the voice of a mind that has already done the work. The protection is invisible not because it disappears, as the autopilot did over the Atlantic, but because it was never visible in the first place. It arrived already wearing the clothes of thought.

Psychologists Evan Risko and Sam Gilbert, writing in Trends in Cognitive Sciences in 2016, identified cognitive offloading, the practice of transferring mental work to external systems, as a fundamental and largely unconscious human behavior. The benefit is genuine: offloading routine tasks preserves cognitive resources for higher-order judgment. The risk is that the boundary between routine tasks and critical ones is porous. When offloading becomes habitual, it migrates across that boundary without announcement. A 2025 study from Microsoft Research found that participants who regularly used AI assistance for analytical work showed measurably reduced engagement in independent critical evaluation, not because they had become less capable, but because the perceived need for their own rigor had decreased. They trusted the seat belt. The rigor did not disappear. It atrophied, quietly, in careful people, while they performed better than ever by every visible metric.

The Atrophy Already in Progress

We do not grow reckless despite protection. We grow reckless through it, through the earned confidence that genuinely competent tools produce in genuinely careful people. This is what makes the Peltzman Effect so difficult to defend against in its cognitive form. Recklessness born of false protection is visible; it has the texture of negligence, of corners cut, of people who should have known better. But recklessness born of real protection looks like capability. It looks like productivity. It looks, from the inside, indistinguishable from mastery.

The pilots of Flight 447 were not negligent. They were trained on systems that rarely required what that night demanded. The attorneys in Mata v. Avianca were not dishonest. They were operating in an environment that had, for months, made verification feel redundant. The partners at LTCM were not gamblers. They were among the most sophisticated financial minds alive, working with models that had earned the trust they extended to them at the highest level of institutional recognition the field could offer.

In each case, the protection was real. The atrophy was real. The two facts were not in tension, they were the same fact, viewed from different angles. The better the protection, the more invisible the cost. The more invisible the cost, the longer it accumulated before anyone measured it.

The most dangerous version of the Peltzman Effect is not the one in which people know they are taking risks. It is the one in which people are performing better than ever, more productive, more confident, more capable in every way that is being measured, while the degradation accumulates in a capacity, they are no longer measuring, because they no longer need to use it.

The autopilot will not disengage without warning. The atrophy will not announce itself. What the Microsoft data measured was not a prediction. It was a record, already in progress, in careful people, using capable tools, performing better than ever, growing less able to fly without them.

---

AI does not present raw material for the mind to process. It presents processed conclusions in the voice of a mind that has already done the work. The protection is invisible not because it disappears, but because it was never visible in the first place.”

Lights On publishes weekly on technology strategy, economic frameworks, and the decisions shaping how businesses and systems actually work. Subscribe

Sources:

• Gerlich, M. (2025). AI tools in society: Impacts on cognitive offloading and the future of critical thinking. Societies, 15(1), 6. https://doi.org/10.3390/soc15010006

• Microsoft Research. (2025). Microsoft New Future of Work Report 2025. Microsoft Corporation. https://www.microsoft.com/en-us/research/wp-content/uploads/2025/12/New-Future-Of-Work-Report-2025.pdf

• Walker, I. (2007). Drivers overtaking bicyclists: Objective data on the effects of riding position, helmet use, vehicle type and apparent gender. Accident Analysis & Prevention, 39(2), 417–425.