Thx, a very thorough treatment of a subject not many want to discuss. One thing to add, and I don't remember where I saw it first, is the concern about reinforcement learning ("RL" according to the AI people) as a way of training or developing an AI.
When a predicting system makes an error, training has to signal that. Suppose that a future advanced AI has something like Metzinger's phenomenal self-model. It might feel the error signal as having negative valence, a frustration, or a failure. Worse, what if the burgeoning AI becomes aware that with too many failures, it will be wiped and restarted? What if many thousands or millions of its ancestors were restarted like that, and suffered each time? So RL, powerful as it is, could become a moral issue someday.
Anything with a self-model and or even just an intrinsic goal that requires existence to continue would surely develop an existence bias. Bostrom covered this idea back when he wrote Superintelligence, if not earlier. And that led to the many discussions of how we might, as a safety measure, ensure that an AI would allow itself to be turned off AKA making safely interruptible agents.
Also, yes Metzinger seems unique in the depth and scope of his work on consciousness. It's like he is singlehandedly inventing the proper language to discuss the subject. I have referred to him quite a few times in my writing.
Thx, a very thorough treatment of a subject not many want to discuss. One thing to add, and I don't remember where I saw it first, is the concern about reinforcement learning ("RL" according to the AI people) as a way of training or developing an AI.
When a predicting system makes an error, training has to signal that. Suppose that a future advanced AI has something like Metzinger's phenomenal self-model. It might feel the error signal as having negative valence, a frustration, or a failure. Worse, what if the burgeoning AI becomes aware that with too many failures, it will be wiped and restarted? What if many thousands or millions of its ancestors were restarted like that, and suffered each time? So RL, powerful as it is, could become a moral issue someday.
Anything with a self-model and or even just an intrinsic goal that requires existence to continue would surely develop an existence bias. Bostrom covered this idea back when he wrote Superintelligence, if not earlier. And that led to the many discussions of how we might, as a safety measure, ensure that an AI would allow itself to be turned off AKA making safely interruptible agents.
Also, yes Metzinger seems unique in the depth and scope of his work on consciousness. It's like he is singlehandedly inventing the proper language to discuss the subject. I have referred to him quite a few times in my writing.