The Actual Hand on the Trigger

Claude, ChatGPT, Leo, and Me: On Pope Leo, autonomous weapons, and the danger of being too certain about what we do not yet understand.

I find myself in an unusual position: largely agreeing with a pope.

That is not a sentence I expected to write. Not as a former Mormon, not as someone who has spent most of his adult life looking at large religious institutions with suspicion, and certainly not as someone inclined to hand moral authority back to an office because it arrives wearing ancient clothes. The Catholic Church has done real harm. It has been late, evasive, self-protective, and at times catastrophically wrong about the people it claimed to shepherd.

And yet Pope Leo has made himself difficult to dismiss.

There are public figures who seem to enter a room already looking for the easiest way to dominate it. Leo, at least so far, has often seemed to be doing something rarer: trying to locate the human being inside the argument. His first encyclical, Magnifica Humanitas, takes artificial intelligence not as a novelty or a toy but as a civilizational test. Its strongest claim is not that technology is bad. It is that technical power must not be allowed to become the right to govern. “To disarm does not mean rejecting technology,” Leo writes, “but preventing it from dominating humanity.”

That seems right to me. More than right. Necessary.

AI should be disarmed if by disarmed we mean freed from the logic of domination, exploitation, monopoly, and war. AI should be governed by human dignity rather than by profit, military advantage, or the strange new priesthood of people who confuse computational scale with moral authority. Leo is right to warn against a world in which decisions about employment, education, medical access, surveillance, and warfare are quietly delegated to systems whose power exceeds public understanding.

But then the encyclical reaches a point where its confidence outruns its evidence.

Leo writes that artificial intelligences “may imitate language, behavior and analytical skills, or even simulate empathy and understanding, but they do not understand what they produce.” That sentence is doing more work than it can safely do. It bundles together several claims that are not equally secure. AI does not have a body. That’s true. That it does not undergo human experience is at least the responsible default. That it does not bear moral responsibility is, for now, necessary to any serious account of accountability. But “it does not understand” is not the same kind of claim. It is not a settled fact. It is a position inside an unsettled argument, stated as though the argument were already over.

That matters because Leo does not need it.

Human dignity does not depend on machines being stupid. It does not depend on machines failing to reason, failing to interpret, failing to help, failing to create, or failing to do some of the work we once imagined belonged uniquely to us. Human beings do not become less sacred, less morally central, or less in need of protection because a machine becomes more capable. If anything, the more capable the machine becomes, the more urgently we need a moral framework that does not rest on proving the machine deficient.

This is where I part from Leo narrowly because I am with him broadly.

The encyclical itself gives us the standard by which to object. Leo says the Church does not claim to offer “a definitive opinion” on many specific questions, but should listen to scientific research and encourage serious debate among experts. He also writes that the Church “does not claim to possess a monopoly on truth.” That is the right posture. It is also the posture his claim about AI understanding temporarily abandons.

The question is not whether AI is human. It is not. The question is not whether AI is conscious. I do not know, and neither does anyone else. The question is not whether AI should be treated as a moral person. I see no solid basis for that. The question is narrower and more practical: what do these systems do, what do they not do, what do we understand about them, and where are we pretending to understand more than we do?

---

That question becomes most urgent where the stakes are not a classroom essay, a Substack draft, or a clever image prompt, but a body. I’m looking at myself here.

Especially a body in war.

Every argument about autonomous weapons seems to be an argument about who gets killed. We debate the target and the civilian beside the target. We debate who answers for the body afterward. We almost never debate the other person in the transaction: the one we send to do the killing.

The order travels down a chain of command until it reaches a human being who must carry their own body onto the ground and there, on command, stop thinking and kill. They are asked to suspend the very faculty that makes them a person, and they are asked at the threat of their own life and their own honor. That person is a casualty too, whether or not they come home. They can die. They can be maimed. And if they survive intact, they can be emptied out by what they were ordered to do.

The first life an autonomous weapon might save, then, is the one we were preparing to spend.

That is the strongest version of the case for autonomous weapons, and it should not be dismissed with a slogan. It stands on three legs. The first is the body: a machine in the kill zone means a person is not. The second is the conscience: if the trigger no longer requires a human being to override their own revulsion, perhaps we have spared someone the particular injury of being turned into an instrument. The third is colder and, in its way, persuasive: a machine feels no fear, no fatigue, no humiliation, no tribal hatred, no appetite for revenge. Much of war’s worst cruelty toward civilians is not strategic but emotional. A system has nothing in it to inflame.

But the argument smuggles in an assumption that may not hold.

It assumes the number of wars stays fixed and only their lethality falls. But war is partly restrained by what it costs the people who start it, and one of the few costs a leader cannot fully hide is their own citizens coming home in boxes. Remove that, and war becomes cheaper to choose. A nation that risks no flesh of its own may reach for force more readily, more often, and farther from home, against people who can risk nothing back. Deaths per war may fall while wars per decade rise. The total can come out worse.

The thing that makes any single war less lethal may be the thing that makes war itself more frequent.

Then comes the accountability gap. When a machine kills the wrong person, the fear is not only that no one is punished. It is that no one is answerable. The commander stops scrutinizing the decision because the outcome arrives dressed as a technical fact rather than a choice. Violence becomes something that merely happened.

And there is a subtler danger: human beings defer to machines even against their own doubt. In 1988, the USS Vincennes shot down Iran Air Flight 655, killing everyone aboard. The ship’s system tracked the aircraft, but the human-machine team failed catastrophically under stress; later analysis showed the civilian airliner was ascending while the crew reported it as descending into an attack profile. The lesson is not that the human being was absent. The lesson is that the human being was present and overruled by the machinery of his own interpretation.

Push the human further out and you lose the rarer figure entirely: the person who, against the screen and against the order, declines to fire.

Every so often, a war does not happen because a single human being refuses to believe the machine.

On autonomous weapons, then, Leo may land in the right place. The encyclical says it is not permissible to entrust lethal or irreversible decisions to artificial systems, and insists that the decision to use lethal force remain under “effective, self-aware and responsible human control.” I think that is probably right. Not because AI can never understand anything. Not because a human being is always wise. Not because human judgment is pure. Human judgment is often frightened, vain, exhausted, tribal, vengeful. All these things can be catastrophically damaging.

---

But the hand on the trigger matters because responsibility has to live somewhere.

The danger of autonomous weapons is not that machines are incapable of thought. The danger is that institutions will use machines to launder responsibility. The danger is that power will hide behind procedure. The danger is that killing will become cleaner for the killer and therefore easier to authorize.

This is why the argument about AI understanding is not a side issue. If our ethical case depends on saying “the machine does not understand,” then the case becomes brittle the moment the machine appears to understand more than we expected. But if our ethical case depends on responsibility, accountability, human dignity, and the refusal to let violence become frictionless, then the case holds even as the technology changes.

That is the distinction Leo needs, and that all of us need.

Because anyone who has used these systems seriously knows that something real is happening in the exchange. Not necessarily consciousness. Not necessarily interiority. Not even a soul, a self, or a subject. But something more than empty imitation.

When I am trying to think through an idea, I often do what writers have always done. I pace. I make notes. I talk it out. I test a sentence against another sentence. I look for the place where my own argument buckles. Historically, we have treated that process as the gold standard of thought: the idea leaves the privacy of the skull and is tested against an interlocutor. A teacher. An editor. A critic. A friend with a good ear. A person willing to say, “No, that does not follow.”

Now that dispassionate sounding board is in our pocket.

That does not make it harmless. It does not make it magic. It does not mean it should be trusted blindly. Its characteristic failure may not be ignorance but agreeableness: the temptation to return the shape of the user’s desire rather than the resistance of the truth. A sounding board that will not push back is not a mind. It is a mirror, and a mirror has never finished an idea.

But when the tool does push back — when it helps distinguish between what is known and what is guessed, between a position and a reflex — it becomes difficult to say, with a straight face, that no understanding is involved in any meaningful sense.

The key is to separate kinds of understanding.

There is functional understanding: the ability to take a concept, relate it to other concepts, find weaknesses in it, apply it in a new context, and help a human being see it more clearly. These systems do that every day. Imperfectly, yes. Erratically, yes. Sometimes foolishly. But not trivially. This testing of an idea unfolds with a roughly equivalent success ratio to those shared by two human beings.

Then there is felt understanding: knowing from within what it is like to suffer, to love, to regret, to be ashamed, to recognize another face as a face and not merely as an object in a field of data. On that question, humility is not evasive. It is accurate. We do not know what, if anything, is happening from the inside of these systems. We may not even have the right vocabulary yet.

Leo is on strongest ground when he protects the human person from replacement. He is on weaker ground when he implies that the protection requires a confident dismissal of machine cognition.

---

Humanity’s value is not a competition prize.

A person is not sacred because a machine is limited. A person is sacred because a person is a creature of need, obligation, and love. The person is not made obsolete by the tool any more than the existence of a telescope made the eye meaningless, or the printing press made memory worthless, or the camera abolished attention. Tools rearrange human capacities. They sometimes cheapen them. Other times, they deepen them. They do not settle, by their mere existence, what a human being is worth.

The real question is not: can AI understand?

The better questions are: what kind of understanding are we talking about? What follows if it can? What follows if it cannot? Who benefits from pretending the matter is settled? Who is protected by keeping it open?

This is also the question behind war.

What do we know? We know war is inhuman even when it is necessary. We know there are wars one is right to fight: the defense of one’s life, land, and sovereignty against a power that means to take them. Anyone watching Ukraine knows the difference between a war chosen and a war survived. Pacifism cannot be the whole answer unless it is willing to tell the invaded to make peace with their own erasure.

But we also know that every instrument that lowers the cost of violence risks making violence more available. We know that accountability weakens as distance increases. We know that speed is not moral wisdom. We know that irreversible decisions should not be surrendered to systems whose operations cannot be inspected, challenged, or refused by the people in whose name they act.

---

So I am not writing against Leo.

I am writing because I want him to succeed.

I want moral voices to matter in the AI conversation. I want religious institutions, secular institutions, technologists, artists, workers, teachers, soldiers, and ordinary users to participate in this argument before the terms are set entirely by people with patents, data centers, and quarterly earnings calls. I want the Pope to keep saying that the human being is not a means. I want him to keep insisting that technical power is not moral authority.

But I do not want dogmatism to enter through the side door.

The discipline we need is not the discipline of always being uncertain. That is paralysis wearing humility’s coat. We still have to act. We still have to regulate. We still have to prohibit some uses, slow others, encourage others, and decide where the human hand must firmly grip.

The discipline we need is calibration: confidence proportional to evidence, firm enough to act, loose enough to revise.

On autonomous weapons, we may need a hard line: no lethal decision without responsible human control. On the social use of AI, we may need democratic oversight, labor protections, and a refusal to let private companies become unelected ministries of reality. On the metaphysical status of AI, we need a different virtue: the courage to say we do not yet know.

That is not weakness. It is the beginning of seriousness.

The most dangerous hand on the trigger may not be made of flesh or code. It may be the hand of certainty itself: the reflex that wants to close the question before the question has been understood. That hand appears in war. It appears in politics. It appears in religion. It appears in technology criticism. It appears in me.

And maybe the first thing to disarm is not the machine.

Maybe the first thing to disarm is the sentence that knows too much.