Lê Nguyên Hoang: /* Motivation and justification */

2020-03-04T08:58:28Z

Motivation and justification

Lê Nguyên Hoang: Created page with "The ABCDE roadmap refers to a decomposition proposed by [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19a] to better highlight..."

2020-01-20T21:32:27Z

Created page with "The ABCDE roadmap refers to a decomposition proposed by [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19a] to better highlight..."

New page

The ABCDE roadmap refers to a decomposition proposed by [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19a] to better highlight the key challenges of AI ethics and safety. It is also discussed at greater length in [https://arxiv.org/pdf/1809.01036 Hoang][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Roadmap+for+Robust+End-to-End+Alignment&btnG= 19b] and [https://laboutique.edpsciences.fr/produit/1107/9782759824304/Le%20fabuleux%20chantier HoangElmhamdi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Le+fabuleux+chantier%3A+Rendre+l%27intelligence+artificielle+robustement+b%C3%A9n%C3%A9fique&btnG= 19<sup>FR</sup>].

== The decomposition ==

The decomposition consists of 5 steps: Alice, Bob, Charlie, Dave and Erin.

Erin's goal is quality data collection and certification, by relying on all sorts of sensors and user inputs, as well as cryptography. Techniques related to [[Blockchain]] may also be useful to guarantee the traceability of data.

Dave is in charge of world model inference from Erin's data. In particular, Dave should correct for sampling biases and account for uncertainty due to data incompleteness. To do so, it may rely on heuristic forms of [[Bayesianism]], like [[representational learning]] constructed by [[GAN]]-like architectures.

Charlie must compute human preferences. In particular, she should probably implement some [[social choice]] mechanism to combine incompatible preferences. Also, she should probably distinguish [[volition]] from instinctive preference. Combining techniques like [[inverse reinforcement learning]] and [[active learning]] is probably critical to design Charlie.

Bob would design incentive-compatible rewards to be given to Alice. By combining Erin, Dave and Charlie's computations, Dave could send to Alice humans' preferences for different states of the world, including the (likely) current state of the world and the probable future states of the world. But to avoid [[Goodhart's law]] and [[wireheading]], it would likely be dangerous to do so directly. Instead, Bob could enable and incentivize the [[corrigibility]] of Erin, Dave and Charlie's computations, by feeding Alice with larger rewards when Erin, Dave and Charlie perform more accurate computations. Designing Bob may be called the [[programmed corrigibility]] problem.

Finally, Alice is going to perform [[reinforcement learning]] using Bob's rewards.

== Motivation and justification ==

The fundamental assumption of the ABCDE roadmap is that tomorrow's most powerful algorithms will be performing reinforcement learning.

From a theoretical perspective, this assumption is strongly backed by the [[AIXI]] framework and, for instance, [[Solomonoff's completeness]] theorem.

From an empirical perspective, it is also backed by the numerous recent successes of reinforcement learning, for instance in Go, Chess and Shogi [https://arxiv.org/pdf/1712.01815 SHSAL+][https://dblp.org/rec/bibtex/journals/corr/abs-1712-01815 17] [https://arxiv.org/pdf/1911.08265 SAHSS+][https://dblp.org/rec/bibtex/journals/corr/abs-1911-08265 19], in video games like Atari games [https://arxiv.org/pdf/1312.5602 MKSGA+][https://dblp.org/rec/bibtex/journals/corr/MnihKSGAWR13 13] or StarCraft [https://www.nature.com/articles/s41586-019-1724-z.epdf VBCMD+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Grandmaster+level+in+StarCraft+II+using+multi-agent+reinforcement+learning&btnG= 19], in combinatorial problems like protein folding [https://kstatic.googleusercontent.com/files/b4d715e8f8b6514cbfdc28a9ad83e14b6a8f86c34ea3b3cc844af8e76767d21ac3df5b0a9177d5e3f6a40b74caf7281a386af0fab8ca62f687599abaf8c8810f EJKSG+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=De+novo+structure+prediction+with+deep%C2%ADlearning+based+scoring&btnG= 18] [https://www.nature.com/articles/s41586-019-1923-7.epdf?author_access_token=Z_KaZKDqtKzbE7Wd5HtwI9RgN0jAjWel9jnR3ZoTv0MCcgAwHMgRx9mvLjNQdB2TlQQaa7l420UCtGo8vYQ39gg8lFWR9mAZtvsN_1PrccXfIbc6e-tGSgazNL_XdtQzn1PHfy21qdcxV7Pw-k3htw%3D%3D SEJKS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Improved+protein+structure+prediction+using+potentials+from+deep+learning&btnG= 20], or in arguably today's most influential algorithm, namely YouTube's recommendation system [https://www.ijcai.org/proceedings/2019/0360.pdf IJWNA+][https://dblp.org/rec/bibtex/conf/ijcai/IeJWNAWCCB19 19].

Reinforcement learning are probably going to keep improving. The only way to make sure that reinforcement learning algorithms will be robustly beneficial is arguably to make sure that these algorithms optimize a desirable goal. This is known as the [[alignment]] problem.

In the case of reinforcement learning, the goal is the sum of discounted future rewards. As a result, the reward function is critical to AI ethics and safety. The ABCDE roadmap highlights it, by further decomposing the components of the reward function.

It is argued that the ABCDE roadmap is likely to be useful, because it allows to decompose the alignment problem into numerous subproblems, which are both (hopefully) independent enough to be tackled separately, and complementary enough so that solutions of subproblems can be easily combined into a solution to the global alignment problem.

@@ Line 17: / Line 17: @@
 == Motivation and justification ==
-The fundamental assumption of the ABCDE roadmap is that tomorrow's most powerful algorithms will be performing reinforcement learning.
+The fundamental assumption of the ABCDE roadmap is that tomorrow's most powerful algorithms will be performing within the [[reinforcement learning]] framework. From a theoretical perspective, this assumption is strongly backed by the [[AIXI]] framework and, for instance, [[Solomonoff's completeness]] theorem.
 From an empirical perspective, it is also backed by the numerous recent successes of reinforcement learning, for instance in Go, Chess and Shogi [https://arxiv.org/pdf/1712.01815 SHSAL+][https://dblp.org/rec/bibtex/journals/corr/abs-1712-01815 17] [https://arxiv.org/pdf/1911.08265 SAHSS+][https://dblp.org/rec/bibtex/journals/corr/abs-1911-08265 19], in video games like Atari games [https://arxiv.org/pdf/1312.5602 MKSGA+][https://dblp.org/rec/bibtex/journals/corr/MnihKSGAWR13 13] or StarCraft [https://www.nature.com/articles/s41586-019-1724-z.epdf VBCMD+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Grandmaster+level+in+StarCraft+II+using+multi-agent+reinforcement+learning&btnG= 19], in combinatorial problems like protein folding [https://kstatic.googleusercontent.com/files/b4d715e8f8b6514cbfdc28a9ad83e14b6a8f86c34ea3b3cc844af8e76767d21ac3df5b0a9177d5e3f6a40b74caf7281a386af0fab8ca62f687599abaf8c8810f EJKSG+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=De+novo+structure+prediction+with+deep%C2%ADlearning+based+scoring&btnG= 18] [https://www.nature.com/articles/s41586-019-1923-7.epdf?author_access_token=Z_KaZKDqtKzbE7Wd5HtwI9RgN0jAjWel9jnR3ZoTv0MCcgAwHMgRx9mvLjNQdB2TlQQaa7l420UCtGo8vYQ39gg8lFWR9mAZtvsN_1PrccXfIbc6e-tGSgazNL_XdtQzn1PHfy21qdcxV7Pw-k3htw%3D%3D SEJKS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Improved+protein+structure+prediction+using+potentials+from+deep+learning&btnG= 20], or in arguably today's most influential algorithm, namely YouTube's recommendation system [https://www.ijcai.org/proceedings/2019/0360.pdf IJWNA+][https://dblp.org/rec/bibtex/conf/ijcai/IeJWNAWCCB19 19].

ABCDE roadmap - Revision history

Lê Nguyên Hoang: /* Motivation and justification */

Lê Nguyên Hoang: Created page with "The ABCDE roadmap refers to a decomposition proposed by [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19a] to better highlight..."