Don't Cry for HAL (WIP)

Introduction

I don’t think that computers ¹ will ever wake up, no matter what software they are running.

This is a very difficult position to defend, including to myself. In this post, I’ll present the most compelling argument I’ve heard for this stance, which I first learned from QRI.

A Case for Conscious Computers

Computational accounts for consciousness are in vogue. Their detractors are usually characterized as scientifically illiterate, “carbon chauvinists”, and/or microtubule-obsessed. What makes the proponents so sure?

One solid way to conclude that a computer could in principle be conscious goes like this ²:

1. Consciousness ³ is generated by the brain.
2. Physical systems like the brain can be perfectly simulated ⁴ on a computer.
3. A perfectly-simulated brain would have the same properties as the original, consciousness included.

There are strong reasons to believe in each of these and they imply that a powerful-enough computer, running the right program, will wake up. This reasoning doesn’t imply that near-term AI systems will be conscious - it just suggests that computers aren’t missing something fundamental to support consciousness.

This conclusion is deeply seductive. If true, we can understand the brain as a biological computer. If true, we can look forward to uploading (a copy of) our minds onto immortal substrates ⁵. And, if true, we might build a conscious AI that loves humanity.

The present argument against conscious computers takes aim at #3 above and suggests that even a perfect brain simulation won’t wake up.

An Overview of the Argument

Most of the work is carefully specifying a set of assumptions about consciousness and computation. Once these are defined, it’s straightforward to show they imply a contradiction .

The three assumption are:

1. Conscious Computation: certain computations have conscious experience as an intrinsic property.
2. Substrate Independence: all intrinsic properties of a computation can be derived from its causal structure.
3. Objectivity of Conscious States: the information content of a conscious experience is objective ⁶.

Together, they imply the contents of a computation’s experience must be objectively determined by the computation’s causal structure.

We’ll see why this can’t be true, implying at least one assumption is false.

Unpacking the Assumptions

Conscious Computation

“Dave, stop. Stop, will you? Stop, Dave. Will you stop, Dave? Stop, Dave. I’m afraid. I’m afraid, Dave. Dave, my mind is going. I can feel it. I can feel it. My mind is going. There is no question about it. I can feel it. I can feel it. I can feel it. I’m a… fraid.” - HAL9000, 2001: A Space Odyssey

The Conscious Computation assumption says that HAL (an AI) wasn’t necessarily faking its feeling of fear. That is, it’s in principle possible that HAL could have experienced its death, that maybe there’s “something it’s like” to have been HAL, and that our emotional response to the text is not necessarily misplaced.

Note that this assumption makes no restrictions on how complex the computation must be to support consciousness. A physics-perfect simulation of a brain is still fair-game.

It also says nothing about what physical system implements the computation. That brings us to…

Substrate Independence

Imagine you want to test how HAL would behave in a certain situation, so you run HAL in a virtual world with no interface to the physical world. Could HAL do an experiment in the virtual world to learn anything about the physical? For example, can it determine any physical properties of the computer it’s running on?

Substrate Independence ⁷ says the answer is “no” ⁸. The reason is that the HAL’s experiments can only determine the causal structure (defined later) of the computation and the same causal structure can be implemented by many different substrates (i.e physical systems). The causal structure captures all intrinsic properties of a computation, pulling them away from details of how that structure is instantiated on a physical substrate.

We benefit from substrate independence all the time: it’s what makes the concept of “software” meaningful. We generally write code without worrying about the layout of the transistors, the temperature of the chip, etc… As long as we have sufficient computational resources, we expect the code to always produce the same output. This works because the software specifies a causal structure that is independent of the substrate running it. If you think you have a counterexample, consider that the physical machine could itself be fully virtualized.

Objectivity of Conscious States

Consider the question: “what are you currently experiencing?”.

The Objectivity of Conscious States assumption says that there is a single correct answer to this question, independent of who is asking the question.

Under Physicalism, we could make this more precise:

Any conscious state is fully determined by a complete and objective descripiton of its underlying physical state. For example, a complete physical description of the brain, over some period of time, would leave no ambiguity about the corresponding conscious experience.

The Contradiction

Now consider the question: “what was the last thing HAL experienced?”.

Conscious Computation says this question is worth asking. Substrate Independence says that the answer only depends on the causal structure of HAL’s program. Objectivity of Conscious States says that we should find an objective feature in this causal structure that answers the question.

Causal Structure

Imagine you could zoom into HAL’s processor as it was dying. You’d see digital memory rapidly changing state with a “bit flip” event being the most relevant primitive. You might also notice that the current state of memory determines the next state, with logical circuits implementing the transition function.

We can abstract-away HAL’s physical implementation details by focusing on the causal dependence between bit flip events. This can be represented with a directed acyclic graph where the nodes correspond to bit flip events and the edges mean “this bit flip must logically happen before this one”. Such a graph’s structure fully captures all intrinsic properties of HAL’s final computation.

No Binding

What kind of substructure in this graph could correspond to a “moment of experience”? Minimally, it would need to associate many bit flip events to a single “frame” of HAL’s subjective time. This is referred to as “binding” the information into a single experience. The lack of a consensus on a solution to this is called The Binding Problem ⁹.

There are only a few ways I could imagine defining intrinsic binding in a causal graph. Most obvious is to just assert that when many events are in the causal past of a single event, they are all “bound”. This fails because the single event has no internal structure to integrate the information - it’s just a bit flip!

Another approach might be to define an extended “screen” of events, and define all the events impinging on the screen to bound into the same experience. This fails because trying to define the screen generates an infinite regress: what intrinsic structure in the graph would select out the events corresponding to the screen? That’s the problem we set out to solve!

The last option is to say the binding is emergent in some tower of complexity and abstraction built on top of the graph. Computationalists talk about things like recursion, phase-locking, self-modeling, attention-heads, second-order perception, prediction error minimization, integrated information, etc… They expect that somewhere in these abstractions built from the causal graph, a well-defined “moment of experience” will arise. I can’t see how that could be true, given that the underlying causal graph is missing the relevant structure to define objective boundaries / subsets.

Conclusion

My conclusion from this argument is to reject the Conscious Computation assumption and answer “nothing” to “what is HAL currently experiencing”.

Discussion

• Implications for models of consciousness - may need binding in the ontology.
• Mention that some computationalists may reject the objectivity assumption instead.
• Ruliadists take the position that map=territory, hence everything is computer.
  • That’s going too far. This argument shows we should be more humble about the territory.
• Address the “wrong level of description” counterpoint (e.g. which atoms in the body are alive)

Acknowledgements

Thank you Andrés Gómez Emilsson @ QRI for introducing me to these ideas ¹⁰. Thank you Joscha Bach for provoking me to write them down.

Thank you Franz, Hikari, Lou, Teafaerie, and Theda for helpful discussions!

Footnotes

Related

• A Paradigm for AI Consciousness – Opentheory.net
• Computational functionalism on trial
• A review of “Don’t forget the boundary problem…” — LessWrong
• Consciousness Actually Explained: EC Theory - by Casey