“‘The Solomonoff Prior is Malign’ is a special case of a simpler argument” by David Matolcsi

LessWrong (Curated & Popular)

[Warning: This post is probably only worth reading if you already have opinions on the Solomonoff induction being malign, or at least heard of the concept and want to understand it better.]
Introduction
I recently reread the classic argument from Paul Christiano about the Solomonoff prior being malign, and Mark Xu's write-up on it. I believe that the part of the argument about the Solomonoff induction is not particularly load-bearing, and can be replaced by a more general argument that I think is easier to understand. So I will present the general argument first, and only explain in the last section how the Solomonoff prior can come into the picture.
I don't claim that anything I write here is particularly new, I think you can piece together this picture from various scattered comments on the topic, but I think it's good to have it written up in one place.
[...]
---
Outline:
(00:17) Introduction
(00:56) How an Oracle gets manipulated
(05:25) What went wrong?
(05:28) The AI had different probability estimates than the humans for anthropic reasons
(07:01) The AI was thinking in terms of probabilities and not expected values
(08:40) Probabilities are cursed in general, only expected values are real
(09:19) What about me?
(13:00) Should this change any of my actions?
(16:25) How does the Solomonoff prior come into the picture?
(20:10) Conclusion
The original text contained 14 footnotes which were omitted from this narration.
---
First published:
November 17th, 2024
Source:
https://www.lesswrong.com/posts/KSdqxrrEootGSpKKE/the-solomonoff-prior-is-malign-is-a-special-case-of-a
---
Narrated by TYPE III AUDIO.