RB Wiki - User contributions [en]

Robustly Beneficial group

2020-03-10T18:52:53Z

Lê Nguyên Hoang: /* Past papers */

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* A Roadmap for Robust End-to-End Alignment. [https://arxiv.org/abs/1809.01036 Hoang][https://dblp.org/rec/bibtex/journals/corr/abs-1809-01036 18] [https://www.youtube.com/watch?v=uR05T8YklvE RB8].
* Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. [https://arxiv.org/abs/1911.08265 SAHSS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Mastering+Atari%2C+Go%2C+Chess+and+Shogi+by+Planning+with+a+Learned+Model&btnG= 19] [https://www.youtube.com/watch?v=NBprWAfCcFY RB7].
* Intelligent Autonomous Things on the Battlefield. AI for the Internet of Everything. [https://arxiv.org/ftp/arxiv/papers/1902/1902.10086.pdf KottStump][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Intelligent+Autonomous+Things+on+the+Battlefield&btnG= 19] [https://youtu.be/gxwGkZeSg30 RB6].
* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18] [https://www.youtube.com/watch?v=bmD-myeu19Q RB5].
* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] [https://www.youtube.com/watch?v=_RuyXyekx6g RB4].
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* An fMRI Investigation of Emotional Engagement in Moral Judgment [https://science.sciencemag.org/content/sci/293/5537/2105.full.pdf?casa_token=7YSThrIwxB0AAAAA:cQJCIjltjkF3GT2V6Op-WBEExmGrwuOsvK6a93ejFZNi6pGRbrWRmoIEOlekacpUbRk04V06Jy9wC4k GSNDC][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=an+fmri+investigation+of+emotional+engagement+in+moral+judgment&btnG= 01]
* Reflections on Trusting Trust. Turing Award Lecture. [https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf Thompson][https://dblp.org/img/download.dark.hollow.16x16.png 84]

Robustly Beneficial group

2020-03-10T18:46:45Z

Lê Nguyên Hoang:

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. [https://arxiv.org/abs/1911.08265 SAHSS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Mastering+Atari%2C+Go%2C+Chess+and+Shogi+by+Planning+with+a+Learned+Model&btnG= 19].
* Intelligent Autonomous Things on the Battlefield. AI for the Internet of Everything. [https://arxiv.org/ftp/arxiv/papers/1902/1902.10086.pdf KottStump][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Intelligent+Autonomous+Things+on+the+Battlefield&btnG= 19] [https://youtu.be/gxwGkZeSg30 RB6].
* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18] [https://www.youtube.com/watch?v=bmD-myeu19Q RB5].
* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] [https://www.youtube.com/watch?v=_RuyXyekx6g RB4].
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* An fMRI Investigation of Emotional Engagement in Moral Judgment [https://science.sciencemag.org/content/sci/293/5537/2105.full.pdf?casa_token=7YSThrIwxB0AAAAA:cQJCIjltjkF3GT2V6Op-WBEExmGrwuOsvK6a93ejFZNi6pGRbrWRmoIEOlekacpUbRk04V06Jy9wC4k GSNDC][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=an+fmri+investigation+of+emotional+engagement+in+moral+judgment&btnG= 01]
* Reflections on Trusting Trust. Turing Award Lecture. [https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf Thompson][https://dblp.org/img/download.dark.hollow.16x16.png 84]

ABCDE roadmap

2020-03-04T08:58:28Z

Lê Nguyên Hoang: /* Motivation and justification */

The ABCDE roadmap refers to a decomposition proposed by [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19a] to better highlight the key challenges of AI ethics and safety. It is also discussed at greater length in [https://arxiv.org/pdf/1809.01036 Hoang][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Roadmap+for+Robust+End-to-End+Alignment&btnG= 19b] and [https://laboutique.edpsciences.fr/produit/1107/9782759824304/Le%20fabuleux%20chantier HoangElmhamdi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Le+fabuleux+chantier%3A+Rendre+l%27intelligence+artificielle+robustement+b%C3%A9n%C3%A9fique&btnG= 19FR].

== The decomposition ==

The decomposition consists of 5 steps: Alice, Bob, Charlie, Dave and Erin.

Erin's goal is quality data collection and certification, by relying on all sorts of sensors and user inputs, as well as cryptography. Techniques related to [[Blockchain]] may also be useful to guarantee the traceability of data.

Dave is in charge of world model inference from Erin's data. In particular, Dave should correct for sampling biases and account for uncertainty due to data incompleteness. To do so, it may rely on heuristic forms of [[Bayesianism]], like [[representational learning]] constructed by [[GAN]]-like architectures.

Charlie must compute human preferences. In particular, she should probably implement some [[social choice]] mechanism to combine incompatible preferences. Also, she should probably distinguish [[volition]] from instinctive preference. Combining techniques like [[inverse reinforcement learning]] and [[active learning]] is probably critical to design Charlie.

Bob would design incentive-compatible rewards to be given to Alice. By combining Erin, Dave and Charlie's computations, Dave could send to Alice humans' preferences for different states of the world, including the (likely) current state of the world and the probable future states of the world. But to avoid [[Goodhart's law]] and [[wireheading]], it would likely be dangerous to do so directly. Instead, Bob could enable and incentivize the [[corrigibility]] of Erin, Dave and Charlie's computations, by feeding Alice with larger rewards when Erin, Dave and Charlie perform more accurate computations. Designing Bob may be called the [[programmed corrigibility]] problem.

Finally, Alice is going to perform [[reinforcement learning]] using Bob's rewards.

== Motivation and justification ==

The fundamental assumption of the ABCDE roadmap is that tomorrow's most powerful algorithms will be performing within the [[reinforcement learning]] framework. From a theoretical perspective, this assumption is strongly backed by the [[AIXI]] framework and, for instance, [[Solomonoff's completeness]] theorem.

From an empirical perspective, it is also backed by the numerous recent successes of reinforcement learning, for instance in Go, Chess and Shogi [https://arxiv.org/pdf/1712.01815 SHSAL+][https://dblp.org/rec/bibtex/journals/corr/abs-1712-01815 17] [https://arxiv.org/pdf/1911.08265 SAHSS+][https://dblp.org/rec/bibtex/journals/corr/abs-1911-08265 19], in video games like Atari games [https://arxiv.org/pdf/1312.5602 MKSGA+][https://dblp.org/rec/bibtex/journals/corr/MnihKSGAWR13 13] or StarCraft [https://www.nature.com/articles/s41586-019-1724-z.epdf VBCMD+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Grandmaster+level+in+StarCraft+II+using+multi-agent+reinforcement+learning&btnG= 19], in combinatorial problems like protein folding [https://kstatic.googleusercontent.com/files/b4d715e8f8b6514cbfdc28a9ad83e14b6a8f86c34ea3b3cc844af8e76767d21ac3df5b0a9177d5e3f6a40b74caf7281a386af0fab8ca62f687599abaf8c8810f EJKSG+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=De+novo+structure+prediction+with+deep%C2%ADlearning+based+scoring&btnG= 18] [https://www.nature.com/articles/s41586-019-1923-7.epdf?author_access_token=Z_KaZKDqtKzbE7Wd5HtwI9RgN0jAjWel9jnR3ZoTv0MCcgAwHMgRx9mvLjNQdB2TlQQaa7l420UCtGo8vYQ39gg8lFWR9mAZtvsN_1PrccXfIbc6e-tGSgazNL_XdtQzn1PHfy21qdcxV7Pw-k3htw%3D%3D SEJKS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Improved+protein+structure+prediction+using+potentials+from+deep+learning&btnG= 20], or in arguably today's most influential algorithm, namely YouTube's recommendation system [https://www.ijcai.org/proceedings/2019/0360.pdf IJWNA+][https://dblp.org/rec/bibtex/conf/ijcai/IeJWNAWCCB19 19].

Reinforcement learning are probably going to keep improving. The only way to make sure that reinforcement learning algorithms will be robustly beneficial is arguably to make sure that these algorithms optimize a desirable goal. This is known as the [[alignment]] problem.

In the case of reinforcement learning, the goal is the sum of discounted future rewards. As a result, the reward function is critical to AI ethics and safety. The ABCDE roadmap highlights it, by further decomposing the components of the reward function.

It is argued that the ABCDE roadmap is likely to be useful, because it allows to decompose the alignment problem into numerous subproblems, which are both (hopefully) independent enough to be tackled separately, and complementary enough so that solutions of subproblems can be easily combined into a solution to the global alignment problem.

Transformer

2020-03-02T09:44:58Z

Lê Nguyên Hoang:

Transformers are learning models that exhibit impressive performances at natural language processing [http://papers.nips.cc/paper/7181-attention-is-all-you-need.pdf VSPUJ+][https://dblp.org/rec/bibtex/conf/nips/VaswaniSPUJGKP17 17] [https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf AWCLAS][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Language+Models+are+Unsupervised+Multitask+Learners&btnG= 19] [https://www.youtube.com/watch?v=89A4jGvaaKk Computerphile19].

Transformer

2020-03-02T09:41:33Z

Lê Nguyên Hoang: Created page with "Transformers are learning models that exhibit impressive performances at natural language processing."

Transformers are learning models that exhibit impressive performances at natural language processing.

Knowledge representation

2020-03-02T09:40:53Z

Lê Nguyên Hoang: /* Transformers */

Knowledge representation is the problem of encoding algorithmically global and common-sense knowledge about the world.

== Knowledge graph ==

== Transformers ==

[https://www.aclweb.org/anthology/P19-1470.pdf BRSMCC][https://dblp.org/rec/bibtex/conf/acl/BosselutRSMCC19 19] designed COMET, an algorithm to combine knowledge graph with [[transformer|transformers]].

Knowledge representation

2020-03-02T09:40:22Z

Lê Nguyên Hoang: Created page with "Knowledge representation is the problem of encoding algorithmically global and common-sense knowledge about the world. == Knowledge graph == == Transformers == [https://www..."

Welcome to the Robustly Beneficial Wiki

2020-03-02T09:37:25Z

Lê Nguyên Hoang: /* How to solve AI ethics (hopefully) */

Welcome to the [[Robustly beneficial|Robustly Beneficial]] wiki!!

This wiki aims to better grasp the scope and the limits of current AI ethics research. It lists references, key ideas and relevant open questions to make algorithms robustly beneficial. Please check also our [https://www.youtube.com/watch?v=WWbw4cla2jw&list=PLgqL_7nXb23FKk_rUfs7vnvyrPshYPfA8 Robustly Beneficial Podcast] ([https://podcasts.apple.com/fr/podcast/robustly-beneficial-podcast/id1496159681 iTunes], [https://playlists.podmytube.com/UCgl_MmjatQif8juz3Lt6iPw/PLgqL_7nXb23FKk_rUfs7vnvyrPshYPfA8.xml RSS]), our [https://www.youtube.com/playlist?list=PLgqL_7nXb23HvhToBb9FwFxj83navY6oq&playnext=1&index=1 Robustly Beneficial Talks] and our [https://twitter.com/robustlyb Twitter account].

The wiki has just been launched, so most pages are still being written. But they will never be finished — this is the whole point of a wiki!

== The structure of the wiki ==

The wiki can be roughly divided into 4 main categories.

=== Why AI ethics is becoming critical ===

If you are new to AI ethics, you should probably start with the [[AI risks]] page. You could then go into arguably today's most important case of AI ethics, namely [[YouTube]]. Note that algorithms also offer formidable [[AI opportunities]] that are definitely worth considering. Find out more by reading about [[online polarization]], [[misinformation]], [[addiction]], [[mental health]] or [[hate]]. And as an example of an urgent AI ethics dilemma, check [https://twitter.com/le_science4all/status/1227690739104174080 this Twitter thread] on responses to a "is climate change a hoax?" search.

And if you know little about the current state of algorithmic research, you might want to check the latest [[impressive advances in AI]]. Or you could check some [[funny applications of AI]]. You can also read Lê's [https://www.lesswrong.com/posts/bwqDrSZvhEDKxRf6z/a-rant-against-robots rant against robots].

=== How today's (and probably tomorrow's) AIs work ===

The most important principle of today's AI is surely [[machine learning]]. Today, it mostly relies on [[stochastic gradient descent]] for (deep) [[neural networks]], which allow [[representational learning]] (see [[convolutional neural network]], [[residual network]], [[transformer]]). See also [[Turing 1950]], [[convexity]], [[generative adversarial network]], [[specialized hardware]] and [[linear systems]].

[[Bayesianism]] has been argued to be the ideal form of supervised and unsupervised learning, if we had infinite computational power (see [[Solomonoff's demon]], [[Laplace 1814]]). It has numerous desirable properties, like [[statistical admissibility]], [[Bayesian agreement]] or the [[Bayesian brain]] hypothesis. See also [[Bayesian examination]] and [[conjugate priors]].

A branch of learning called [[reinforcement learning]], which relies on [[Q-learning]] or [[policy learning]], seems likely to become the core framework of today's and tomorrow's AIs. [[AIXI]] achieves the upper-bound for [[Legg-Hutter intelligence]], which aims to measure [[artificial general intelligence|general intelligence]].

To understand the gap between Bayesianism/AIXI and practical machine learning, we need to understand the constraints of computational [[complexity]] theory. By building upon the [[Church-Turing thesis]], the [[Kolmogorov-Solomonoff complexity]] and knowledge from [[human brain computations]], this allows some insights into [[human-level AI]], in addition to [[experts' AI predictions]]. See also [[entropy]] and [[sophistication]].

AIs are already doing [[distributed learning]], which raises numerous challenges, like [[Byzantine fault tolerance]] and [[model drift]].

=== Why AI safety and ethics is harder than meets the eye ===

We want to get algorithms to do what we would really want them to do. But this turns out to raise numerous highly nontrivial problems, like [[Goodhart's law]], [[overfitting]], [[robust statistics]], [[confounding variables]], [[adversarial attacks]], [[algorithmic bias]], [[cognitive bias]], [[backfire effect]], [[distributional shift]], [[privacy]], [[human liabilities]], [[interpretability]], [[reward hacking]], [[wireheading]] and [[instrumental convergence]]. Because of all such problems, it seems crucial that algorithms be able to reason about their ignorance, using [[Bayesianism|Bayesian]] principles, [[moral uncertainty]] and [[second opinion querying]]. Algorithms must be [[robustly beneficial]].

AI ethics also demands that we solve thorny philosophical dilemmas, like the [[repugnant conclusion]], [[Newcomb's paradox]] and [[moral realism]]. Unfortunately, we have numerous [[cognitive bias|cognitive biases]], which seem critical to understand to solve AI ethics. Results about [[counterfactual]], [[von Neumann-Morgenstern preferences]] and [[Dutch book]] also seem useful to consider.

=== How to solve AI ethics (hopefully) ===

To solve AI ethics, [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19a] proposed the [[ABCDE roadmap]], which decomposes the [[alignment]] problem into numerous (hopefully) orthogonal and complementary subproblems. Such subproblems include [[data certification]], perhaps through [[Blockchain]], [[world model inference]] of [[knowledge representation]] through [[Bayesianism]] and/or [[representational learning]], [[volition]] learning perhaps from [[Preference learning from comparisons|comparisons]] and [[social choice]] solutions, [[corrigibility]] and safe [[reinforcement learning]].

The fabulous endeavor to make AIs robustly beneficial can seem overwhelming, given how extraordinarily interdisciplinary it is. While it is worthwhile to have an overview of the problem, we believe it is also useful for aspiring contributors to identify more precise problems they can contribute to. In this wiki, we propose targeted research directions for different expertises and research interests. Please check the following pages that may be of interest to you.
<ul>
<li>[[how newcomers can contribute]]</li>
<li>[[how mathematicians can contribute]]</li>
<li>[[how computer scientists can contribute]]</li>
<li>[[how software engineers can contribute]]</li>
<li>[[how neuroscientists can contribute]]</li>
<li>[[how philosophers can contribute]]</li>
<li>[[how social scientists can contribute]]</li>
<li>[[how economists can contribute]]</li>
<li>[[how policy-makers can contribute]]</li>
<li>[[how journalists can contribute]]</li>
<li>[[how medical doctors can contribute]]</li>
<li>[[how physicists can contribute]]</li>
<li>[[how educators can contribute]]</li>
<li>[[how science communicators can contribute]]</li>
<li>[[how community builders can contribute]]</li>
</ul>

== About the authors ==

This wiki is written and edited mostly by members of the [[Robustly Beneficial group]], which regularly meets at EPFL, in Lausanne, Switzerland. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join]. So far, the main authors are [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Louis_Faucon|Louis Faucon]].

Lê and Mahdi recently co-wrote the book The Fabulous Endeavor: Make Artificial Intelligence Robustly Beneficial [https://laboutique.edpsciences.fr/produit/1107/9782759824304/Le%20fabuleux%20chantier HoangElmhamdi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Le+fabuleux+chantier%3A+Rendre+l%27intelligence+artificielle+robustement+b%C3%A9n%C3%A9fique&btnG= 19FR] (the English version is pending).

Overfitting

2020-02-26T17:09:11Z

Lê Nguyên Hoang: /* Details */

Overfitting occurs where fitting training data too closely is counter-productive to out-of-sample predictions.

== Bias-variance tradeoff ==

[http://web.mit.edu/6.435/www/Geman92.pdf GBD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=neural+networks+and+the+bias%2Fvariance+dilemma+geman+bienenstock+doursat&btnG= 92] identified the bias-variance tradeoff, which quantifies out-of-sample errors as a sum of inductive bias and model variance, for random samples drawn from the true distribution.

Formally, let <math>f(x,S)</math> the <math>y</math>-prediction of the algorithm trained with sample <math>S</math> for feature <math>x</math>. Then <math>\mathbb E_{x,y,S}[(f(x,S)-y)^2] = bias^2 + variance + \sigma^2</math>, where the expectation is over <math>x</math>, <math>S</math> and <math>y</math>, where <math>bias = \mathbb E_{x,S}[f(x,S)-f^*(x)]</math> is the bias with respect to the optimal prediction <math>f^*</math>, where <math>variance = \mathbb E_{x,S}[f(x,S)^2] - \mathbb E_{x,S}[f(x,S)]^2</math> is how much the prediction varies from one sample to the other and <math>\sigma^2 = \mathbb E_{x,y}[(f^*(x)-y)^2]</math> is the unpredictable components of <math>y</math> given <math>x</math>.

It has been argued that overfitting is caused by increased variance when we consider a learning algorithm that is too sensitive to the randomness of sampling <math>S</math>. This sometimes occurs when the number of parameters of the learning algorithm is too large (but not necessarily!).

== PAC-learning ==

To prove theoretical guarantees of non-overfitting, [http://web.mit.edu/6.435/www/Valiant84.pdf Valiant][https://dblp.org/rec/bibtex/journals/cacm/Valiant84 84] introduced the concept of probably approximately correct (PAC) learning. More explanations here: [https://www.youtube.com/watch?v=uB2X2OuD4Rg&list=PLie7a1OUTSagZB9mFZnVBgsNfBtcUGJWB&index=8 Wandida16a].

In particular, the fundamental theorem of statistical learning [https://www.amazon.com/Understanding-Machine-Learning-Theory-Algorithms/dp/1107057132/ ShalevshwartzBendavidBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Understanding+Machine+Learning+Theory+Algorithms+shalev-shwartz+ben-david&btnG= 14] [https://www.youtube.com/watch?v=RkWuLtFPBKU&list=PLie7a1OUTSagZB9mFZnVBgsNfBtcUGJWB&index=14 Wandida16b] provides guarantees of PAC learning, when the number of data points sufficiently exceed the VC dimension of the set of learnable algorithms. Since this VC dimension is often essentially the number of parameters (assuming finite representation of the parameters as float or double), then this means that PAC learning is guaranteed when <math>\#data \ll \#parameters</math>.

This has become a conventional wisdom for a while.

== Test set overfitting ==

There has been concerns about overfitting of test sets, as these are used more and more to measure the performance of machine learning algorithms. But [https://arxiv.org/pdf/1902.10811.pdf RRSS][https://dblp.org/rec/bibtex/conf/icml/RechtRSS19 19] analyze statistical patterns on reported test set performances, and argue that there still is actual progress.

== Double descent ==

However, the conventional wisdom is in sharp contradiction with today's success of deep neural networks [https://openreview.net/pdf?id=Sy8gdB9xx ZBHRV][https://dblp.org/rec/bibtex/conf/iclr/ZhangBHRV17 17], [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] [https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19], but also kernel methods [http://proceedings.mlr.press/v80/belkin18a/belkin18a.pdf BMM][https://dblp.org/rec/bibtex/conf/icml/BelkinMM18 18] [http://proceedings.mlr.press/v89/belkin19a/belkin19a.pdf BRT][https://dblp.org/rec/bibtex/conf/aistats/BelkinRT19 19], ridgeless (random feature) linear regression [https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8849614 MVS][https://dblp.org/rec/bibtex/conf/isit/MuthukumarVS19 19] [https://arxiv.org/pdf/1906.11300 BLLT][https://dblp.org/rec/bibtex/journals/corr/abs-1906-11300 19] [https://arxiv.org/pdf/1908.05355 MeiMontanari][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+generalization+error+of+random+features+regression%3A+Precise+asymptotics+and+double+descent+curve&btnG= 19] [https://arxiv.org/pdf/1903.08560 HMRT][https://dblp.org/rec/bibtex/journals/corr/abs-1903-08560 19] [https://arxiv.org/pdf/1903.07571 BHX][https://dblp.org/rec/bibtex/journals/corr/abs-1903-07571 19] [https://arxiv.org/pdf/2002.08404.pdf JSSHG][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Implicit+Regularization+of+Random+Feature+Models&btnG= 20] and even ensembles [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19]. Learning algorithms seem to often achieve their best out-of-sample performance when they are massively overparameterized and perfectly fit the training data (called interpolation).

Intriguingly, a double descent phenomenon often occurs [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] [https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19], where the performance at the test set first behaves as predicted by the bias-variance dilemma, but then improves and outperforms what would be advised by classical statistical learning.

All such results suggest that overfitting eventually disappears, which contradicts conventional wisdom.

== Details ==

[https://openreview.net/pdf?id=Sy8gdB9xx ZBHRV][https://dblp.org/rec/bibtex/conf/iclr/ZhangBHRV17 17] showed that large interpolating neural networks generalize well, even for large noise in the data. Also, they showed that inductive bias likely plays a limited role, as neural networks still manage to learn quite efficiently data whose labels are completely shuffled. They also proved that a neural network with <math>2n+d</math> parameters can interpolate <math>n</math> data points of dimension <math>d</math>.

[https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] observed double descent for random Fourier features (which [http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf RahimiRecht][https://dblp.org/rec/bibtex/conf/nips/RahimiR07 07] proved to be intimately connected to kernel methods), neural networks, decision tree and ensemble methods.

[https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19] show that a very wide variety of deep neural networks exhibit a wide variety of double descent phenomenons. Not only is there double descent with respect to the number of parameters, but there also seems to be double descent with respect to the width of the neural networks, and weirdly also with respect to epochs of learning steps. They conjecture that "effective model complexity" (the number of data points for which the model is able to achieve small training loss) is a critical point where overfitting occurs. Before and beyond this, overfitting appears to vanish.

[http://proceedings.mlr.press/v80/belkin18a/belkin18a.pdf BMM][https://dblp.org/rec/bibtex/conf/icml/BelkinMM18 18] present experiments that show that interpolating kernel methods also generalize well and are able to fit random labels (though in this paper, they do not exhibit double descent). They also show that, because norms of interpolaters grow superpolynomially in Hilbert space (in <math>exp(\theta(n^{1/d}))</math>), usual bounds controlling overfitting are actually trivial for large datasets. This indicates the need for radically different approach to understand overfitting.

[http://proceedings.mlr.press/v89/belkin19a/belkin19a.pdf BRT][https://dblp.org/rec/bibtex/conf/aistats/BelkinRT19 19] show examples of singular kernel interpolators (<math>K(x,y)\sim||x-y||^{-a}</math>) that achieve optimal rates, even for improper learning (meaning that the true function to learn does not belong to the set of hypotheses).

Note that the connection between kernel methods and neural networks has been made, for instance by [http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf RahimiRecht][https://dblp.org/rec/bibtex/conf/nips/RahimiR07 07] and [http://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf JHG][https://dblp.org/rec/bibtex/conf/nips/JacotHG18 18]. Essentially, random features implement approximate kernel methods. And the first layers of neural networks with random (or even trained) weights can be regarded as computations of random features.

[https://arxiv.org/pdf/1903.08560 HMRT][https://dblp.org/rec/bibtex/journals/corr/abs-1903-08560 19] use random matrix theory (to estimate eigenvalues of <math>X^TX</math> when <math>X</math> are random samples) to show that ridgless regression (regression with minimum <math>\ell_2</math>-norm) features "infinite double descent" as the size <math>n</math> of the training data sets grows to infinity, along with the number of parameters <math>p</math> (they assume <math>p/n \rightarrow gamma</math>, and show infinite overfitting for <math>gamma\sim 1</math>). This is shown for both a linear model where <math>X = \Sigma^{1/2} Z</math>, for some fixed <math>\Sigma</math> and some well-behaved <math>Z</math> of mean 0 and variance 1, and a component-wise linearity <math>X = \sigma(WZ)</math>. In both cases, it is assumed that <math>y=x^T\beta+\varepsilon</math>, where <math>\mathbb E[\varepsilon]=0</math>. There are also assumptions of finite fixed variance for <math>\varepsilon</math>, and finite fourth moment for <math>Z</math> (needed for random matrix theory).

[https://arxiv.org/pdf/1903.07571 BHX][https://dblp.org/rec/bibtex/journals/corr/abs-1903-07571 19] analyze two other data models. The former is a classical Gaussian linear regression with a huge space of features. But the regression is only made within a (random) subset <math>T</math> of features, in which case double descent is observed, and errors can be derived from the norms of the true regression for <math>T</math>-coordinates and non-<math>T</math>-coordinates. A similar analysis is then provided for a random Fourier feature model.

[https://arxiv.org/pdf/1908.05355 MeiMontanari][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+generalization+error+of+random+features+regression%3A+Precise+asymptotics+and+double+descent+curve&btnG= 19] consider still another data model, where <math>z</math> is drawn uniformly randomly on a <math>d</math>-sphere, and <math>y = f(z)+\varepsilon</math>, such that <math>\mathbb E[\varepsilon]=0</math> and <math>\varepsilon</math> has finite fourth moment. The ridgeless linear regression is then over some random features <math>x_i = \sigma(Wz_i)</math>, where <math>\sigma</math> applies component-wisely and <math>W</math> has random rows of <math>\ell_2</math>-norm equal to 1. They prove that this yields a double descent.

[https://arxiv.org/pdf/2002.08404.pdf JSSHG][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Implicit+Regularization+of+Random+Feature+Models&btnG= 20] study a random feature model where the true function and the features are drawn from a Gaussian process. They prove an upper-bound between a regularized Bayesian posterior prediction at a point <math>x</math> and the expectation of the (slightly differently) regularized random feature prediction at this point, which can be argued to go to zero under reasonable assumptions, as the number of parameters goes to infinity. Moreover, as the regularization goes to zero, the random feature linear regression comes closer to the Bayesian posterior prediction. They also prove a bound between the expected loss of the regularized random feature model and that of the regularized kernel model. They also show that the "effective ridge" <math>\tilde \lambda</math> (i.e. ridge of the kernel method equivalent to the ridge <math>\lambda</math> of the random feature regression) is key to understanding the variance explosion. In particular, they relate it to the explosion of <math>\partial_\lambda \tilde \lambda</math>. An interesting insight is that the adequate linear regression regularization should thus be smaller than (and can be exactly computed from) the kernel method ridge.

Overfitting

2020-02-26T17:07:35Z

Lê Nguyên Hoang: /* Details */

Overfitting

2020-02-26T17:03:39Z

Lê Nguyên Hoang: /* Details */

Overfitting

2020-02-26T16:58:40Z

Lê Nguyên Hoang: /* Details */

Overfitting

2020-02-26T16:56:04Z

Lê Nguyên Hoang: /* Double descent */

Overfitting

2020-02-26T16:55:02Z

Lê Nguyên Hoang: /* Double descent */

Overfitting occurs where fitting training data too closely is counter-productive to out-of-sample predictions.

== Bias-variance tradeoff ==

[http://web.mit.edu/6.435/www/Geman92.pdf GBD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=neural+networks+and+the+bias%2Fvariance+dilemma+geman+bienenstock+doursat&btnG= 92] identified the bias-variance tradeoff, which quantifies out-of-sample errors as a sum of inductive bias and model variance, for random samples drawn from the true distribution.

Formally, let <math>f(x,S)</math> the <math>y</math>-prediction of the algorithm trained with sample <math>S</math> for feature <math>x</math>. Then <math>\mathbb E_{x,y,S}[(f(x,S)-y)^2] = bias^2 + variance + \sigma^2</math>, where the expectation is over <math>x</math>, <math>S</math> and <math>y</math>, where <math>bias = \mathbb E_{x,S}[f(x,S)-f^*(x)]</math> is the bias with respect to the optimal prediction <math>f^*</math>, where <math>variance = \mathbb E_{x,S}[f(x,S)^2] - \mathbb E_{x,S}[f(x,S)]^2</math> is how much the prediction varies from one sample to the other and <math>\sigma^2 = \mathbb E_{x,y}[(f^*(x)-y)^2]</math> is the unpredictable components of <math>y</math> given <math>x</math>.

It has been argued that overfitting is caused by increased variance when we consider a learning algorithm that is too sensitive to the randomness of sampling <math>S</math>. This sometimes occurs when the number of parameters of the learning algorithm is too large (but not necessarily!).

== PAC-learning ==

To prove theoretical guarantees of non-overfitting, [http://web.mit.edu/6.435/www/Valiant84.pdf Valiant][https://dblp.org/rec/bibtex/journals/cacm/Valiant84 84] introduced the concept of probably approximately correct (PAC) learning. More explanations here: [https://www.youtube.com/watch?v=uB2X2OuD4Rg&list=PLie7a1OUTSagZB9mFZnVBgsNfBtcUGJWB&index=8 Wandida16a].

In particular, the fundamental theorem of statistical learning [https://www.amazon.com/Understanding-Machine-Learning-Theory-Algorithms/dp/1107057132/ ShalevshwartzBendavidBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Understanding+Machine+Learning+Theory+Algorithms+shalev-shwartz+ben-david&btnG= 14] [https://www.youtube.com/watch?v=RkWuLtFPBKU&list=PLie7a1OUTSagZB9mFZnVBgsNfBtcUGJWB&index=14 Wandida16b] provides guarantees of PAC learning, when the number of data points sufficiently exceed the VC dimension of the set of learnable algorithms. Since this VC dimension is often essentially the number of parameters (assuming finite representation of the parameters as float or double), then this means that PAC learning is guaranteed when <math>\#data \ll \#parameters</math>.

This has become a conventional wisdom for a while.

== Test set overfitting ==

There has been concerns about overfitting of test sets, as these are used more and more to measure the performance of machine learning algorithms. But [https://arxiv.org/pdf/1902.10811.pdf RRSS][https://dblp.org/rec/bibtex/conf/icml/RechtRSS19 19] analyze statistical patterns on reported test set performances, and argue that there still is actual progress.

== Double descent ==

However, the conventional wisdom is in sharp contradiction with today's success of deep neural networks [https://openreview.net/pdf?id=Sy8gdB9xx ZBHRV][https://dblp.org/rec/bibtex/conf/iclr/ZhangBHRV17 17], [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] [https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19], but also kernel methods [http://proceedings.mlr.press/v80/belkin18a/belkin18a.pdf BMM][https://dblp.org/rec/bibtex/conf/icml/BelkinMM18 18] [http://proceedings.mlr.press/v89/belkin19a/belkin19a.pdf BRT][https://dblp.org/rec/bibtex/conf/aistats/BelkinRT19 19], ridgeless (random feature) linear regression [https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8849614 MVS][https://dblp.org/rec/bibtex/conf/isit/MuthukumarVS19 19] [https://arxiv.org/pdf/1906.11300 BLLT][https://dblp.org/rec/bibtex/journals/corr/abs-1906-11300 19] [https://arxiv.org/pdf/1908.05355 MeiMontanari][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+generalization+error+of+random+features+regression%3A+Precise+asymptotics+and+double+descent+curve&btnG= 19] [https://arxiv.org/pdf/1903.08560 HMRT][https://dblp.org/rec/bibtex/journals/corr/abs-1903-08560 19] [https://arxiv.org/pdf/1903.07571 BHX][https://dblp.org/rec/bibtex/journals/corr/abs-1903-07571 19] [https://arxiv.org/pdf/2002.08404.pdf JSSHG][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Implicit+Regularization+of+Random+Feature+Models&btnG= 20] and even ensembles [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19]. Learning algorithms seem to often achieve their best out-of-sample performance when they are massively overparameterized and perfectly fit the training data (called interpolation).

Intriguingly, a double descent phenomenon often occurs [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] [https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19], where the performance at the test set first behaves as predicted by the bias-variance dilemma, but then improves and outperforms what would be advised by classical statistical learning.

[https://arxiv.org/pdf/2002.08404.pdf JSSHG][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Implicit+Regularization+of+Random+Feature+Models&btnG= 20] study a random feature model where the true function and the features are drawn from a Gaussian process. They prove an upper-bound between a regularized Bayesian posterior prediction and the expectation of the (slightly differently) regularized random feature prediction, which can be argued to go to zero under reasonable assumptions, as the number of parameters goes to infinity. Moreover, as the regularization goes to zero, the random feature linear regression comes closer to the Bayesian posterior prediction.

All such results suggest that overfitting eventually disappears, which contradicts conventional wisdom.

== Details ==

[https://openreview.net/pdf?id=Sy8gdB9xx ZBHRV][https://dblp.org/rec/bibtex/conf/iclr/ZhangBHRV17 17] showed that large interpolating neural networks generalize well, even for large noise in the data. Also, they showed that inductive bias likely plays a limited role, as neural networks still manage to learn quite efficiently data whose labels are completely shuffled. They also proved that a neural network with <math>2n+d</math> parameters can interpolate <math>n</math> data points of dimension <math>d</math>.

[https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] observed double descent for random Fourier features (which [http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf RahimiRecht][https://dblp.org/rec/bibtex/conf/nips/RahimiR07 07] proved to be intimately connected to kernel methods), neural networks, decision tree and ensemble methods.

[https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19] show that a very wide variety of deep neural networks exhibit a wide variety of double descent phenomenons. Not only is there double descent with respect to the number of parameters, but there also seems to be double descent with respect to the width of the neural networks, and weirdly also with respect to epochs of learning steps. They conjecture that "effective model complexity" (the number of data points for which the model is able to achieve small training loss) is a critical point where overfitting occurs. Before and beyond this, overfitting appears to vanish.

[http://proceedings.mlr.press/v80/belkin18a/belkin18a.pdf BMM][https://dblp.org/rec/bibtex/conf/icml/BelkinMM18 18] present experiments that show that interpolating kernel methods also generalize well and are able to fit random labels (though in this paper, they do not exhibit double descent). They also show that, because norms of interpolaters grow superpolynomially in Hilbert space (in <math>exp(\theta(n^{1/d}))</math>), usual bounds controlling overfitting are actually trivial for large datasets. This indicates the need for radically different approach to understand overfitting.

[http://proceedings.mlr.press/v89/belkin19a/belkin19a.pdf BRT][https://dblp.org/rec/bibtex/conf/aistats/BelkinRT19 19] show examples of singular kernel interpolators (<math>K(x,y)\sim||x-y||^{-a}</math>) that achieve optimal rates, even for improper learning (meaning that the true function to learn does not belong to the set of hypotheses).

Note that the connection between kernel methods and neural networks has been made, for instance by [http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf RahimiRecht][https://dblp.org/rec/bibtex/conf/nips/RahimiR07 07] and [http://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf JHG][https://dblp.org/rec/bibtex/conf/nips/JacotHG18 18]. Essentially, random features implement approximate kernel methods. And the first layers of neural networks with random (or even trained) weights can be regarded as computations of random features.

[https://arxiv.org/pdf/1903.08560 HMRT][https://dblp.org/rec/bibtex/journals/corr/abs-1903-08560 19] use random matrix theory (to estimate eigenvalues of <math>X^TX</math> when <math>X</math> are random samples) to show that ridgless regression (regression with minimum <math>\ell_2</math>-norm) features "infinite double descent" as the size <math>n</math> of the training data sets grows to infinity, along with the number of parameters <math>p</math> (they assume <math>p/n \rightarrow gamma</math>, and show infinite overfitting for <math>gamma\sim 1</math>). This is shown for both a linear model where <math>X = \Sigma^{1/2} Z</math>, for some fixed <math>\Sigma</math> and some well-behaved <math>Z</math> of mean 0 and variance 1, and a component-wise linearity <math>X = \sigma(WZ)</math>. In both cases, it is assumed that <math>y=x^T\beta+\varepsilon</math>, where <math>\mathbb E[\varepsilon]=0</math>. There are also assumptions of finite fixed variance for <math>\varepsilon</math>, and finite fourth moment for <math>Z</math> (needed for random matrix theory).

[https://arxiv.org/pdf/1903.07571 BHX][https://dblp.org/rec/bibtex/journals/corr/abs-1903-07571 19] analyze two other data models. The former is a classical Gaussian linear regression with a huge space of features. But the regression is only made within a (random) subset <math>T</math> of features, in which case double descent is observed, and errors can be derived from the norms of the true regression for <math>T</math>-coordinates and non-<math>T</math>-coordinates. A similar analysis is then provided for a random Fourier feature model.

[https://arxiv.org/pdf/1908.05355 MeiMontanari][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+generalization+error+of+random+features+regression%3A+Precise+asymptotics+and+double+descent+curve&btnG= 19] consider still another data model, where <math>z</math> is drawn uniformly randomly on a <math>d</math>-sphere, and <math>y = f(z)+\varepsilon</math>, such that <math>\mathbb E[\varepsilon]=0</math> and <math>\varepsilon</math> has finite fourth moment. The ridgeless linear regression is then over some random features <math>x_i = \sigma(Wz_i)</math>, where <math>\sigma</math> applies component-wisely and <math>W</math> has random rows of <math>\ell_2</math>-norm equal to 1. They prove that this yields a double descent.

Robust statistics

2020-02-26T07:00:25Z

Lê Nguyên Hoang: /* Robustness to additive poisoning */

Robust statistics is the problem of estimating parameters from unreliable empirical data. Typically, suppose that a fraction of the training dataset is compromised. Can we design algorithms that nevertheless succeed in learning adequately from such a partially compromised dataset?

This question has arguably become crucial, as large-scale algorithms perform learning from users' data. Yet, clearly, if the algorithm is used by thousands, millions or billions of users, many of the data will likely be corrupted, because of bugs [https://www.youtube.com/watch?v=yb2zkxHDfUE standupmaths20], or because some users will maliciously want to exploit or attack the algorithm. This latter case is known as a poisoning attack.

Over the last three years, there have been fascinating recent advances, both for classical statistical tasks [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] and [[modern machine learning]] [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] especially in very high dimensional settings such as [[training neural networks]] [http://proceedings.mlr.press/v80/mhamdi18a EGR]. We discussed robust statistics in [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].

== Example of the median ==

Suppose the data are real numbers, and we want to estimate the mean of the true data (which we shall call inliers). Note that the naive empirical mean estimate would be a bad idea here, as a single malicious user could completely upset the empirical mean estimate. In fact, by choosing its input data adequately (called outliers), the malicious user can make the empirical mean estimate equal whatever the malicious user wants it to be.

It turns out that using the median of the dataset is a robust way to do so. Indeed, even if 45% of the data are outliers, the median will still be a quantile of the inliers, which should not be too far from the actual mean. The median is said to have a 0.5 statistical breakdown point [https://www.researchgate.net/profile/Peter_Rousseeuw/publication/303193534_Robust_Regression_Outlier_Detection_John_Wiley_Sons/links/5d7a06b64585151ee4af67da/Robust-Regression-Outlier-Detection-John-Wiley-Sons.pdf RousseeuwLeroy][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+regression+and+outlier+detection+rousseeuw+leroy&btnG= 05]. No statistical method can achieve a better breakdown point, but other methods also achieve 0.5 statistical breakdown, like trimmed mean (we remove sufficiently many extreme values on both sides and take the mean of the rest).

Another way to quantify robustness is to compute a high-probability upper bound between the empirical median and the mean μ of the true distribution of inliers. Call ε the fraction of outliers. It turns out that, assuming the true distribution is a normal distribution <math>\mathcal N(\mu,1)</math>, given <math>n=\Omega\left( \frac{d+\log(1/\tau)}{\varepsilon^2}\right)</math> data, we can guarantee <math>|median-\mu| = O(\varepsilon)</math> with probability <math>1-\tau</math>. This asymptotic bound is also best possible [https://projecteuclid.org/download/pdf_1/euclid.aoms/1177703732 Huber][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+estimation+of+a+location+parameter+huber&btnG= 92].

== Poisoning models ==

The above model holds for arguably the strongest poisoning model. This is one where an adversary gets to read the full dataset before we can, and is able to erase a fraction <math>\varepsilon</math> of the data, and to replace them by any imaginable input. The dataset is then analyzed by our (robust) statistics algorithm.

A weaker, but still widespread, model is one where a fraction <math>1-\varepsilon</math> comes from the true distribution, while the remaining <math>\varepsilon</math> is chosen by the adversary [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BMGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17].

Other models include an adversary with only erasing power, or an adversary that must choose its outliers without knowledge of the values of the inliers. Evidently, any guarantee for such weaker poisoning models will also hold for stronger poisoning models [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19].

Perhaps the most general form of poisoning attack is the following. Consider a "true dataset" <math>D</math>. However, the attacker gets to distort the dataset using a distort function <math>f \in \mathcal F</math>, thereby yielding <math>f(D)</math>. Suppose we now have a best-possible machine learning algorithm <math>ML</math> that learns from data. It would ideally compute <math>ML(D)</math>. But we can only exploit <math>f(D)</math>, by some hopefully robust machine learning algorithm <math>RML(f(D))</math>. What we would like is to guarantee that <math>d(ML(D),RML(f(D))) < bound</math>, for a suitable distance <math>d</math> and any <math>f \in \mathcal F</math>, where <math>RML</math> is tractable.

A further generalization of this could consist in assuming a prior probabilistic belief on the set <math>\mathcal F</math> of attack models that we need to defend against. This would correspond to the study of Byzantine Bayesian learning, which we may model as <math>\max_{RML} \mathbb E_{\mathcal F} \left[ \min_{f \in \mathcal F} \mathbb E[u|D,RML(f(D))] \right]</math>.

It is noteworthy that robustness to such attacks are useful, even if there are no adversary. Indeed, data may still get corrupted, because of bugs, crashes or misuse by a human operator (see for instance the case of gene mutations caused by Excel that plagued genetic research [https://www.sciencemag.org/news/2016/08/one-five-genetics-papers-contains-errors-thanks-microsoft-excel Boddy16]). Our algorithms need to remain performant despite such issues. An algorithm robust to strong attacks will be robust to such weaker flaws.

== Robustness to additive poisoning ==

Unfortunately, results that hold for small dimensions generalize poorly to high dimensions, either because of weak robustness guarantees or computational slowness. Typically, the coordinate-wise median and the [[geometric median]] both yield <math>\Omega(\varepsilon \sqrt{d})</math>-error, even in the limit of infinite-size datasets and assuming normality for inliers. This is very bad, as today's [[neural networks]] often have <math>d\sim 10^6</math>, if not <math>10^9</math> or <math>10^{12}</math> parameters.

On the other hand, assuming the true distribution is a spherical normal distribution <math>\mathcal N(0,I)</math>, Tukey proposed another approach based on identifying the directions of largest variances, since these are likely to be the "attack line" of the adversary [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Mathematics+and+the+picturing+of+data+tukey&btnG= Tukey75]. Tukey's median yields <math>O(\varepsilon)</math>-error with high probability <math>1-\tau</math> for <math>n=\Omega\left( \frac{d+\log(1/\tau)}{\varepsilon^2}\right)</math> data points. Unfortunately, Tukey's median is NP-hard to compute, and is typically exponential in d.

[https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] proved that such a bound can be achieved in quasi-linear-time, even for heavy-tailed distribution with bounded but unknown variance. [https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] proved that in the strong poisoning model, their quasi-linear-time algorithm achieves <math>O(||\Sigma||_{op} \sqrt{\varepsilon})</math> error, which is asymptotically optimal in terms of performance and computation time. On the other hand, [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] used an approach akin to median-of-means, for <math>K = \Omega(\varepsilon n)</math>, they designed an algorithm that achieves an error <math>O\left( \sqrt{\frac{Tr(\Sigma)}{n}} + \sqrt{\frac{||\Sigma||_{op} K}{n}} \right)</math> in time <math>\tilde O(nd + uKd)</math>. Here, <math>u</math> is an integer parameter, which is needed to guarantee a subgaussian decay rate of errors, which will be in <math>1-\exp(-\Theta(K+u))</math>. Note that <math>Tr(\Sigma)</math> is essentially the "effective dimension" of the data points.

Their technique relies on partitioning data into <math>K</math> buckets, computing the means for each bucket, and replacing the computation of median of the means by a covering SDP that fits all configuration of bucket poisoning. It turns out that approximations of such an covering SDP can be founded in quasi-linear-time [https://arxiv.org/pdf/1201.5135 PTZ][https://dblp.org/rec/bibtex/conf/spaa/PengT12 12]. It turns out that, rather than being used to directly compute a mean estimator, this is actually used to perform gradient descent, starting from the coordinate-wise median, and then descending along the direction provided by the covering SDP. [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] proved that only <math>O(\log d)</math> such steps were needed to guarantee their bound.

== Robustness to strong poisoning ==

Note that all the papers of this section seem to strongly rely on the knowledge of the covariance matrix of inliers by the algorithm.

For robustness to strong poisoning, Tukey's ideas can be turned into an polynomial-time algorithm for robust statistics mean estimate. The trick is to identify worst-case "attack line" by computing the largest eigenvalue of the empirical covariance matrix, and to remove extremal points along such lines to reduce variance. [https://arxiv.org/pdf/1604.06443.pdf DKKLMS][https://dblp.org/rec/bibtex/conf/focs/DiakonikolasKK016 16] [http://proceedings.mlr.press/v70/diakonikolas17a/diakonikolas17a.pdf DKKLMS][https://dblp.org/rec/bibtex/conf/icml/DiakonikolasKK017 17] show that, for <math>n=\Omega(d/\varepsilon^2)</math>, this yields <math>O(\varepsilon \sqrt{\log(1/\varepsilon)})</math>-error with high probability for sub-Gaussian inliers, and <math>O(\sigma \sqrt{\varepsilon})</math> for inliners whose true covariance matrix <math>\Sigma</math> is such that <math>\sigma^2 I - \Sigma</math> is semidefinite positive.

The asymptotical optimal bound <math>O(\varepsilon)</math> has been achieved a more sophisticated filtering polynomial-time algorithm by [https://epubs.siam.org/doi/pdf/10.1137/1.9781611975031.171 DKKLM+][https://dblp.org/rec/bibtex/conf/soda/DiakonikolasKK018 18] for Gaussian distribution in the additive poisoning model, while [https://ieeexplore.ieee.org/document/8104048 DKS][https://dblp.org/rec/bibtex/conf/focs/DiakonikolasKS17 17] showed that no polynomial-time can achieve better than <math>O(\varepsilon \sqrt{\log(1/\varepsilon)})</math> in the Statistical Query Model with strong poisoning.

Quasi-linear-time robust mean estimators have been designed by [https://epubs.siam.org/doi/pdf/10.1137/1.9781611975482.171 CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19], i.e. with <math>\tilde O(nd)</math> up to logarithmic factors, based on filtering methods of extremal points on variance-maximizing directions.

Note that all such results can be applied to robust linear regression, by applying robust mean estimator to gradient descent estimator (with the mean taken over data points), assuming that the covariance matrix of the distribution of features <math>x</math> is known, and that the noise is of known mean and variance. Robust covariance matrix estimation can also be addressed by the framework, as it too can be regarded as a robust mean estimation problem (the fourth moment then needs to be assumed to be upper-bounded).

== What if there are more outliers than inliers? ==

Another puzzling question concerns the setting where outliers may be more numerous than inliers. A simple argument shows that methods cited above just won't work. What can be done for such a setting? Can we still learn something from the data, or do we have to throw away the dataset altogether? Let's cite three approaches that may be fruitful.

First, [https://dl.acm.org/doi/pdf/10.1145/1374376.1374474?casa_token=CY0VwFuDnskAAAAA:Q_eNEhY5XxJ8_rll_t1TeRYQiTB6fJeTXQG_OJGwVeghBAcpD6rwFCExUqrmwX5SP-N9iKd8n2CxKQ BBV][https://dblp.org/rec/bibtex/conf/stoc/BalcanBV08 08] introduced list-decodable learning, which consists in returning several hypothesis, and [https://arxiv.org/pdf/1711.07211 DKS][https://dblp.org/rec/bibtex/conf/stoc/DiakonikolasKS18 18] provided polynomial-time for robust list-decodable mean estimation.

Second, one might try to a apply a robust Bayesian inference to the data, which would yield a set of posterior beliefs. This framework has yet to be defined.

Third, we may assume that the data are more or less [[data certification|certified]]. This is a natural setting, as we humans often judge the reliability of raw data depending on its source, and we usually consider a continuum of reliability, rather than a clear cut binary classification. Algorithms should probably do the same at some point, but there does not yet seem to be an algorithmic framework to pose this problem. In particular, it could be interesting to analyze threat models where different degrees or sorts of certification have different levels of liability.

== Robust statistics for neural networks ==

The main application of robust statistics (at least relevant to AI ethics) seems to be the aggregation of [[stochastic gradient descent|stochastic gradients]] for [[neural networks]]. In this setting, even a linear-time algorithm in <math>\Omega(nd)</math> is impractical if we demand <math>n \geq d</math> (which is necessary to have dimension-independent guarantees). In practice, this setting is often carried out with batches whose size <math>n</math> is significantly smaller than <math>d</math>. In fact, despite conventional wisdom and PAC-learning theory, it seems that <math>n \ll d</math> may be a desirable setting to do neural network learning (see [[overfitting]] where we discuss double descent). For <math>n \ll d</math>, is there a gain in using algorithms more complex than coordinate-wise median?

[http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] proposed Krum and multi-Krum, aggregation algorithms for this setting that have weaker robustness guarantees but are more efficient. Is it possible to improve upon them?

== Robust statistics for agreement and multi-agent settings ==

Another venue where robust statistics are needed is the increasingly multi-agent setting that modern AI is built on. [https://dl.acm.org/doi/10.1145/2488608.2488657 MH2013]

Robust statistics

2020-02-26T06:59:30Z

Lê Nguyên Hoang: /* Robustness to additive poisoning */

Robust statistics is the problem of estimating parameters from unreliable empirical data. Typically, suppose that a fraction of the training dataset is compromised. Can we design algorithms that nevertheless succeed in learning adequately from such a partially compromised dataset?

This question has arguably become crucial, as large-scale algorithms perform learning from users' data. Yet, clearly, if the algorithm is used by thousands, millions or billions of users, many of the data will likely be corrupted, because of bugs [https://www.youtube.com/watch?v=yb2zkxHDfUE standupmaths20], or because some users will maliciously want to exploit or attack the algorithm. This latter case is known as a poisoning attack.

Over the last three years, there have been fascinating recent advances, both for classical statistical tasks [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] and [[modern machine learning]] [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] especially in very high dimensional settings such as [[training neural networks]] [http://proceedings.mlr.press/v80/mhamdi18a EGR]. We discussed robust statistics in [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].

== Example of the median ==

Suppose the data are real numbers, and we want to estimate the mean of the true data (which we shall call inliers). Note that the naive empirical mean estimate would be a bad idea here, as a single malicious user could completely upset the empirical mean estimate. In fact, by choosing its input data adequately (called outliers), the malicious user can make the empirical mean estimate equal whatever the malicious user wants it to be.

It turns out that using the median of the dataset is a robust way to do so. Indeed, even if 45% of the data are outliers, the median will still be a quantile of the inliers, which should not be too far from the actual mean. The median is said to have a 0.5 statistical breakdown point [https://www.researchgate.net/profile/Peter_Rousseeuw/publication/303193534_Robust_Regression_Outlier_Detection_John_Wiley_Sons/links/5d7a06b64585151ee4af67da/Robust-Regression-Outlier-Detection-John-Wiley-Sons.pdf RousseeuwLeroy][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+regression+and+outlier+detection+rousseeuw+leroy&btnG= 05]. No statistical method can achieve a better breakdown point, but other methods also achieve 0.5 statistical breakdown, like trimmed mean (we remove sufficiently many extreme values on both sides and take the mean of the rest).

Another way to quantify robustness is to compute a high-probability upper bound between the empirical median and the mean μ of the true distribution of inliers. Call ε the fraction of outliers. It turns out that, assuming the true distribution is a normal distribution <math>\mathcal N(\mu,1)</math>, given <math>n=\Omega\left( \frac{d+\log(1/\tau)}{\varepsilon^2}\right)</math> data, we can guarantee <math>|median-\mu| = O(\varepsilon)</math> with probability <math>1-\tau</math>. This asymptotic bound is also best possible [https://projecteuclid.org/download/pdf_1/euclid.aoms/1177703732 Huber][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+estimation+of+a+location+parameter+huber&btnG= 92].

== Poisoning models ==

The above model holds for arguably the strongest poisoning model. This is one where an adversary gets to read the full dataset before we can, and is able to erase a fraction <math>\varepsilon</math> of the data, and to replace them by any imaginable input. The dataset is then analyzed by our (robust) statistics algorithm.

A weaker, but still widespread, model is one where a fraction <math>1-\varepsilon</math> comes from the true distribution, while the remaining <math>\varepsilon</math> is chosen by the adversary [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BMGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17].

Other models include an adversary with only erasing power, or an adversary that must choose its outliers without knowledge of the values of the inliers. Evidently, any guarantee for such weaker poisoning models will also hold for stronger poisoning models [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19].

Perhaps the most general form of poisoning attack is the following. Consider a "true dataset" <math>D</math>. However, the attacker gets to distort the dataset using a distort function <math>f \in \mathcal F</math>, thereby yielding <math>f(D)</math>. Suppose we now have a best-possible machine learning algorithm <math>ML</math> that learns from data. It would ideally compute <math>ML(D)</math>. But we can only exploit <math>f(D)</math>, by some hopefully robust machine learning algorithm <math>RML(f(D))</math>. What we would like is to guarantee that <math>d(ML(D),RML(f(D))) < bound</math>, for a suitable distance <math>d</math> and any <math>f \in \mathcal F</math>, where <math>RML</math> is tractable.

A further generalization of this could consist in assuming a prior probabilistic belief on the set <math>\mathcal F</math> of attack models that we need to defend against. This would correspond to the study of Byzantine Bayesian learning, which we may model as <math>\max_{RML} \mathbb E_{\mathcal F} \left[ \min_{f \in \mathcal F} \mathbb E[u|D,RML(f(D))] \right]</math>.

It is noteworthy that robustness to such attacks are useful, even if there are no adversary. Indeed, data may still get corrupted, because of bugs, crashes or misuse by a human operator (see for instance the case of gene mutations caused by Excel that plagued genetic research [https://www.sciencemag.org/news/2016/08/one-five-genetics-papers-contains-errors-thanks-microsoft-excel Boddy16]). Our algorithms need to remain performant despite such issues. An algorithm robust to strong attacks will be robust to such weaker flaws.

== Robustness to additive poisoning ==

Unfortunately, results that hold for small dimensions generalize poorly to high dimensions, either because of weak robustness guarantees or computational slowness. Typically, the coordinate-wise median and the [[geometric median]] both yield <math>\Omega(\varepsilon \sqrt{d})</math>-error, even in the limit of infinite-size datasets and assuming normality for inliers. This is very bad, as today's [[neural networks]] often have <math>d\sim 10^6</math>, if not <math>10^9</math> or <math>10^{12}</math> parameters.

On the other hand, assuming the true distribution is a spherical normal distribution <math>\mathcal N(0,I)</math>, Tukey proposed another approach based on identifying the directions of largest variances, since these are likely to be the "attack line" of the adversary [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Mathematics+and+the+picturing+of+data+tukey&btnG= Tukey75]. Tukey's median yields <math>O(\varepsilon)</math>-error with high probability <math>1-\tau</math> for <math>n=\Omega\left( \frac{d+\log(1/\tau)}{\varepsilon^2}\right)</math> data points. Unfortunately, Tukey's median is NP-hard to compute, and is typically exponential in d.

[https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] proved that such a bound can be achieved in quasi-linear-time, even for heavy-tailed distribution with bounded but unknown variance. [https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] proved that in the strong poisoning model, their quasi-linear-time algorithm achieves <math>O(\sqrt{\varepsilon})</math> error, which is asymptotically optimal in terms of performance and computation time. On the other hand, [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] used an approach akin to median-of-means, for <math>K = \Omega(\varepsilon n)</math>, they designed an algorithm that achieves an error <math>O\left( \sqrt{\frac{Tr(\Sigma)}{n}} + \sqrt{\frac{||\Sigma||_{op} K}{n}} \right)</math> in time <math>\tilde O(nd + uKd)</math>. Here, <math>u</math> is an integer parameter, which is needed to guarantee a subgaussian decay rate of errors, which will be in <math>1-\exp(-\Theta(K+u))</math>. Note that <math>Tr(\Sigma)</math> is essentially the "effective dimension" of the data points.

Their technique relies on partitioning data into <math>K</math> buckets, computing the means for each bucket, and replacing the computation of median of the means by a covering SDP that fits all configuration of bucket poisoning. It turns out that approximations of such an covering SDP can be founded in quasi-linear-time [https://arxiv.org/pdf/1201.5135 PTZ][https://dblp.org/rec/bibtex/conf/spaa/PengT12 12]. It turns out that, rather than being used to directly compute a mean estimator, this is actually used to perform gradient descent, starting from the coordinate-wise median, and then descending along the direction provided by the covering SDP. [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] proved that only <math>O(\log d)</math> such steps were needed to guarantee their bound.

== Robustness to strong poisoning ==

Note that all the papers of this section seem to strongly rely on the knowledge of the covariance matrix of inliers by the algorithm.

For robustness to strong poisoning, Tukey's ideas can be turned into an polynomial-time algorithm for robust statistics mean estimate. The trick is to identify worst-case "attack line" by computing the largest eigenvalue of the empirical covariance matrix, and to remove extremal points along such lines to reduce variance. [https://arxiv.org/pdf/1604.06443.pdf DKKLMS][https://dblp.org/rec/bibtex/conf/focs/DiakonikolasKK016 16] [http://proceedings.mlr.press/v70/diakonikolas17a/diakonikolas17a.pdf DKKLMS][https://dblp.org/rec/bibtex/conf/icml/DiakonikolasKK017 17] show that, for <math>n=\Omega(d/\varepsilon^2)</math>, this yields <math>O(\varepsilon \sqrt{\log(1/\varepsilon)})</math>-error with high probability for sub-Gaussian inliers, and <math>O(\sigma \sqrt{\varepsilon})</math> for inliners whose true covariance matrix <math>\Sigma</math> is such that <math>\sigma^2 I - \Sigma</math> is semidefinite positive.

The asymptotical optimal bound <math>O(\varepsilon)</math> has been achieved a more sophisticated filtering polynomial-time algorithm by [https://epubs.siam.org/doi/pdf/10.1137/1.9781611975031.171 DKKLM+][https://dblp.org/rec/bibtex/conf/soda/DiakonikolasKK018 18] for Gaussian distribution in the additive poisoning model, while [https://ieeexplore.ieee.org/document/8104048 DKS][https://dblp.org/rec/bibtex/conf/focs/DiakonikolasKS17 17] showed that no polynomial-time can achieve better than <math>O(\varepsilon \sqrt{\log(1/\varepsilon)})</math> in the Statistical Query Model with strong poisoning.

Quasi-linear-time robust mean estimators have been designed by [https://epubs.siam.org/doi/pdf/10.1137/1.9781611975482.171 CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19], i.e. with <math>\tilde O(nd)</math> up to logarithmic factors, based on filtering methods of extremal points on variance-maximizing directions.

Note that all such results can be applied to robust linear regression, by applying robust mean estimator to gradient descent estimator (with the mean taken over data points), assuming that the covariance matrix of the distribution of features <math>x</math> is known, and that the noise is of known mean and variance. Robust covariance matrix estimation can also be addressed by the framework, as it too can be regarded as a robust mean estimation problem (the fourth moment then needs to be assumed to be upper-bounded).

== What if there are more outliers than inliers? ==

Another puzzling question concerns the setting where outliers may be more numerous than inliers. A simple argument shows that methods cited above just won't work. What can be done for such a setting? Can we still learn something from the data, or do we have to throw away the dataset altogether? Let's cite three approaches that may be fruitful.

First, [https://dl.acm.org/doi/pdf/10.1145/1374376.1374474?casa_token=CY0VwFuDnskAAAAA:Q_eNEhY5XxJ8_rll_t1TeRYQiTB6fJeTXQG_OJGwVeghBAcpD6rwFCExUqrmwX5SP-N9iKd8n2CxKQ BBV][https://dblp.org/rec/bibtex/conf/stoc/BalcanBV08 08] introduced list-decodable learning, which consists in returning several hypothesis, and [https://arxiv.org/pdf/1711.07211 DKS][https://dblp.org/rec/bibtex/conf/stoc/DiakonikolasKS18 18] provided polynomial-time for robust list-decodable mean estimation.

Second, one might try to a apply a robust Bayesian inference to the data, which would yield a set of posterior beliefs. This framework has yet to be defined.

Third, we may assume that the data are more or less [[data certification|certified]]. This is a natural setting, as we humans often judge the reliability of raw data depending on its source, and we usually consider a continuum of reliability, rather than a clear cut binary classification. Algorithms should probably do the same at some point, but there does not yet seem to be an algorithmic framework to pose this problem. In particular, it could be interesting to analyze threat models where different degrees or sorts of certification have different levels of liability.

== Robust statistics for neural networks ==

The main application of robust statistics (at least relevant to AI ethics) seems to be the aggregation of [[stochastic gradient descent|stochastic gradients]] for [[neural networks]]. In this setting, even a linear-time algorithm in <math>\Omega(nd)</math> is impractical if we demand <math>n \geq d</math> (which is necessary to have dimension-independent guarantees). In practice, this setting is often carried out with batches whose size <math>n</math> is significantly smaller than <math>d</math>. In fact, despite conventional wisdom and PAC-learning theory, it seems that <math>n \ll d</math> may be a desirable setting to do neural network learning (see [[overfitting]] where we discuss double descent). For <math>n \ll d</math>, is there a gain in using algorithms more complex than coordinate-wise median?

[http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] proposed Krum and multi-Krum, aggregation algorithms for this setting that have weaker robustness guarantees but are more efficient. Is it possible to improve upon them?

== Robust statistics for agreement and multi-agent settings ==

Another venue where robust statistics are needed is the increasingly multi-agent setting that modern AI is built on. [https://dl.acm.org/doi/10.1145/2488608.2488657 MH2013]

Robust statistics

2020-02-26T06:58:43Z

Lê Nguyên Hoang:

Robust statistics is the problem of estimating parameters from unreliable empirical data. Typically, suppose that a fraction of the training dataset is compromised. Can we design algorithms that nevertheless succeed in learning adequately from such a partially compromised dataset?

This question has arguably become crucial, as large-scale algorithms perform learning from users' data. Yet, clearly, if the algorithm is used by thousands, millions or billions of users, many of the data will likely be corrupted, because of bugs [https://www.youtube.com/watch?v=yb2zkxHDfUE standupmaths20], or because some users will maliciously want to exploit or attack the algorithm. This latter case is known as a poisoning attack.

Over the last three years, there have been fascinating recent advances, both for classical statistical tasks [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] and [[modern machine learning]] [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] especially in very high dimensional settings such as [[training neural networks]] [http://proceedings.mlr.press/v80/mhamdi18a EGR]. We discussed robust statistics in [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].

== Example of the median ==

Suppose the data are real numbers, and we want to estimate the mean of the true data (which we shall call inliers). Note that the naive empirical mean estimate would be a bad idea here, as a single malicious user could completely upset the empirical mean estimate. In fact, by choosing its input data adequately (called outliers), the malicious user can make the empirical mean estimate equal whatever the malicious user wants it to be.

It turns out that using the median of the dataset is a robust way to do so. Indeed, even if 45% of the data are outliers, the median will still be a quantile of the inliers, which should not be too far from the actual mean. The median is said to have a 0.5 statistical breakdown point [https://www.researchgate.net/profile/Peter_Rousseeuw/publication/303193534_Robust_Regression_Outlier_Detection_John_Wiley_Sons/links/5d7a06b64585151ee4af67da/Robust-Regression-Outlier-Detection-John-Wiley-Sons.pdf RousseeuwLeroy][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+regression+and+outlier+detection+rousseeuw+leroy&btnG= 05]. No statistical method can achieve a better breakdown point, but other methods also achieve 0.5 statistical breakdown, like trimmed mean (we remove sufficiently many extreme values on both sides and take the mean of the rest).

Another way to quantify robustness is to compute a high-probability upper bound between the empirical median and the mean μ of the true distribution of inliers. Call ε the fraction of outliers. It turns out that, assuming the true distribution is a normal distribution <math>\mathcal N(\mu,1)</math>, given <math>n=\Omega\left( \frac{d+\log(1/\tau)}{\varepsilon^2}\right)</math> data, we can guarantee <math>|median-\mu| = O(\varepsilon)</math> with probability <math>1-\tau</math>. This asymptotic bound is also best possible [https://projecteuclid.org/download/pdf_1/euclid.aoms/1177703732 Huber][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+estimation+of+a+location+parameter+huber&btnG= 92].

== Poisoning models ==

The above model holds for arguably the strongest poisoning model. This is one where an adversary gets to read the full dataset before we can, and is able to erase a fraction <math>\varepsilon</math> of the data, and to replace them by any imaginable input. The dataset is then analyzed by our (robust) statistics algorithm.

A weaker, but still widespread, model is one where a fraction <math>1-\varepsilon</math> comes from the true distribution, while the remaining <math>\varepsilon</math> is chosen by the adversary [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BMGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17].

Other models include an adversary with only erasing power, or an adversary that must choose its outliers without knowledge of the values of the inliers. Evidently, any guarantee for such weaker poisoning models will also hold for stronger poisoning models [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19].

Perhaps the most general form of poisoning attack is the following. Consider a "true dataset" <math>D</math>. However, the attacker gets to distort the dataset using a distort function <math>f \in \mathcal F</math>, thereby yielding <math>f(D)</math>. Suppose we now have a best-possible machine learning algorithm <math>ML</math> that learns from data. It would ideally compute <math>ML(D)</math>. But we can only exploit <math>f(D)</math>, by some hopefully robust machine learning algorithm <math>RML(f(D))</math>. What we would like is to guarantee that <math>d(ML(D),RML(f(D))) < bound</math>, for a suitable distance <math>d</math> and any <math>f \in \mathcal F</math>, where <math>RML</math> is tractable.

A further generalization of this could consist in assuming a prior probabilistic belief on the set <math>\mathcal F</math> of attack models that we need to defend against. This would correspond to the study of Byzantine Bayesian learning, which we may model as <math>\max_{RML} \mathbb E_{\mathcal F} \left[ \min_{f \in \mathcal F} \mathbb E[u|D,RML(f(D))] \right]</math>.

It is noteworthy that robustness to such attacks are useful, even if there are no adversary. Indeed, data may still get corrupted, because of bugs, crashes or misuse by a human operator (see for instance the case of gene mutations caused by Excel that plagued genetic research [https://www.sciencemag.org/news/2016/08/one-five-genetics-papers-contains-errors-thanks-microsoft-excel Boddy16]). Our algorithms need to remain performant despite such issues. An algorithm robust to strong attacks will be robust to such weaker flaws.

== Robustness to additive poisoning ==

Unfortunately, results that hold for small dimensions generalize poorly to high dimensions, either because of weak robustness guarantees or computational slowness. Typically, the coordinate-wise median and the [[geometric median]] both yield <math>\Omega(\varepsilon \sqrt{d})</math>-error, even in the limit of infinite-size datasets and assuming normality for inliers. This is very bad, as today's [[neural networks]] often have <math>d\sim 10^6</math>, if not <math>10^9</math> or <math>10^{12}</math> parameters.

On the other hand, assuming the true distribution is a spherical normal distribution <math>\mathcal N(0,I)</math>, Tukey proposed another approach based on identifying the directions of largest variances, since these are likely to be the "attack line" of the adversary [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Mathematics+and+the+picturing+of+data+tukey&btnG= Tukey75]. Tukey's median yields <math>O(\varepsilon)</math>-error with high probability <math>1-\tau</math> for <math>n=\Omega\left( \frac{d+\log(1/\tau)}{\varepsilon^2}\right)</math> data points. Unfortunately, Tukey's median is NP-hard to compute, and is typically exponential in d.

[https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] proved that such a bound can be achieved in quasi-linear-time, even for heavy-tailed distribution with bounded but unknown variance. [https://arxiv.org/pdf/1811.09380.pdf CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19] proved that in the strong poisoning model, their quasi-linear-time algorithm achieves <math>O(\sqrt{\varepsilon}</math> error, which is asymptotically optimal in terms of performance and computation time. On the other hand, [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] used an approach akin to median-of-means, for <math>K = \Omega(\varepsilon n)</math>, they designed an algorithm that achieves an error <math>O\left( \sqrt{\frac{Tr(\Sigma)}{n}} + \sqrt{\frac{||\Sigma||_{op} K}{n}} \right)</math> in time <math>\tilde O(nd + uKd)</math>. Here, <math>u</math> is an integer parameter, which is needed to guarantee a subgaussian decay rate of errors, which will be in <math>1-\exp(-\Theta(K+u))</math>. Note that <math>Tr(\Sigma)</math> is essentially the "effective dimension" of the data points.

Their technique relies on partitioning data into <math>K</math> buckets, computing the means for each bucket, and replacing the computation of median of the means by a covering SDP that fits all configuration of bucket poisoning. It turns out that approximations of such an covering SDP can be founded in quasi-linear-time [https://arxiv.org/pdf/1201.5135 PTZ][https://dblp.org/rec/bibtex/conf/spaa/PengT12 12]. It turns out that, rather than being used to directly compute a mean estimator, this is actually used to perform gradient descent, starting from the coordinate-wise median, and then descending along the direction provided by the covering SDP. [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] proved that only <math>O(\log d)</math> such steps were needed to guarantee their bound.

== Robustness to strong poisoning ==

Note that all the papers of this section seem to strongly rely on the knowledge of the covariance matrix of inliers by the algorithm.

For robustness to strong poisoning, Tukey's ideas can be turned into an polynomial-time algorithm for robust statistics mean estimate. The trick is to identify worst-case "attack line" by computing the largest eigenvalue of the empirical covariance matrix, and to remove extremal points along such lines to reduce variance. [https://arxiv.org/pdf/1604.06443.pdf DKKLMS][https://dblp.org/rec/bibtex/conf/focs/DiakonikolasKK016 16] [http://proceedings.mlr.press/v70/diakonikolas17a/diakonikolas17a.pdf DKKLMS][https://dblp.org/rec/bibtex/conf/icml/DiakonikolasKK017 17] show that, for <math>n=\Omega(d/\varepsilon^2)</math>, this yields <math>O(\varepsilon \sqrt{\log(1/\varepsilon)})</math>-error with high probability for sub-Gaussian inliers, and <math>O(\sigma \sqrt{\varepsilon})</math> for inliners whose true covariance matrix <math>\Sigma</math> is such that <math>\sigma^2 I - \Sigma</math> is semidefinite positive.

The asymptotical optimal bound <math>O(\varepsilon)</math> has been achieved a more sophisticated filtering polynomial-time algorithm by [https://epubs.siam.org/doi/pdf/10.1137/1.9781611975031.171 DKKLM+][https://dblp.org/rec/bibtex/conf/soda/DiakonikolasKK018 18] for Gaussian distribution in the additive poisoning model, while [https://ieeexplore.ieee.org/document/8104048 DKS][https://dblp.org/rec/bibtex/conf/focs/DiakonikolasKS17 17] showed that no polynomial-time can achieve better than <math>O(\varepsilon \sqrt{\log(1/\varepsilon)})</math> in the Statistical Query Model with strong poisoning.

Quasi-linear-time robust mean estimators have been designed by [https://epubs.siam.org/doi/pdf/10.1137/1.9781611975482.171 CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19], i.e. with <math>\tilde O(nd)</math> up to logarithmic factors, based on filtering methods of extremal points on variance-maximizing directions.

Note that all such results can be applied to robust linear regression, by applying robust mean estimator to gradient descent estimator (with the mean taken over data points), assuming that the covariance matrix of the distribution of features <math>x</math> is known, and that the noise is of known mean and variance. Robust covariance matrix estimation can also be addressed by the framework, as it too can be regarded as a robust mean estimation problem (the fourth moment then needs to be assumed to be upper-bounded).

== What if there are more outliers than inliers? ==

Another puzzling question concerns the setting where outliers may be more numerous than inliers. A simple argument shows that methods cited above just won't work. What can be done for such a setting? Can we still learn something from the data, or do we have to throw away the dataset altogether? Let's cite three approaches that may be fruitful.

First, [https://dl.acm.org/doi/pdf/10.1145/1374376.1374474?casa_token=CY0VwFuDnskAAAAA:Q_eNEhY5XxJ8_rll_t1TeRYQiTB6fJeTXQG_OJGwVeghBAcpD6rwFCExUqrmwX5SP-N9iKd8n2CxKQ BBV][https://dblp.org/rec/bibtex/conf/stoc/BalcanBV08 08] introduced list-decodable learning, which consists in returning several hypothesis, and [https://arxiv.org/pdf/1711.07211 DKS][https://dblp.org/rec/bibtex/conf/stoc/DiakonikolasKS18 18] provided polynomial-time for robust list-decodable mean estimation.

Second, one might try to a apply a robust Bayesian inference to the data, which would yield a set of posterior beliefs. This framework has yet to be defined.

Third, we may assume that the data are more or less [[data certification|certified]]. This is a natural setting, as we humans often judge the reliability of raw data depending on its source, and we usually consider a continuum of reliability, rather than a clear cut binary classification. Algorithms should probably do the same at some point, but there does not yet seem to be an algorithmic framework to pose this problem. In particular, it could be interesting to analyze threat models where different degrees or sorts of certification have different levels of liability.

== Robust statistics for neural networks ==

The main application of robust statistics (at least relevant to AI ethics) seems to be the aggregation of [[stochastic gradient descent|stochastic gradients]] for [[neural networks]]. In this setting, even a linear-time algorithm in <math>\Omega(nd)</math> is impractical if we demand <math>n \geq d</math> (which is necessary to have dimension-independent guarantees). In practice, this setting is often carried out with batches whose size <math>n</math> is significantly smaller than <math>d</math>. In fact, despite conventional wisdom and PAC-learning theory, it seems that <math>n \ll d</math> may be a desirable setting to do neural network learning (see [[overfitting]] where we discuss double descent). For <math>n \ll d</math>, is there a gain in using algorithms more complex than coordinate-wise median?

[http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] proposed Krum and multi-Krum, aggregation algorithms for this setting that have weaker robustness guarantees but are more efficient. Is it possible to improve upon them?

== Robust statistics for agreement and multi-agent settings ==

Another venue where robust statistics are needed is the increasingly multi-agent setting that modern AI is built on. [https://dl.acm.org/doi/10.1145/2488608.2488657 MH2013]

Reinforcement learning

2020-02-25T18:38:33Z

Lê Nguyên Hoang: Created page with "Reinforcement learning is a general framework for sequential decision-making. == MuZero == [https://arxiv.org/pdf/1911.08265.pdf SAHSS+][https://dblp.org/rec/bibtex/journal..."

Reinforcement learning is a general framework for sequential decision-making.

== MuZero ==

[https://arxiv.org/pdf/1911.08265.pdf SAHSS+][https://dblp.org/rec/bibtex/journals/corr/abs-1911-08265 19] introduced MuZero.

Distributional shift

2020-02-24T09:54:57Z

Lê Nguyên Hoang: Created page with "Distributional shift is the problem of achieving good performances despite a change in the data distribution. Typically if an algorithm learns from a distribution <math>\mathc..."

Distributional shift is the problem of achieving good performances despite a change in the data distribution. Typically if an algorithm learns from a distribution <math>\mathcal D</math>, can we guarantee that it will perform well when tested on distribution <math>\mathcal D'</math>?

== Distributional shift ==

[shankar roelofs mania fang recht imagenet citation needed] argue, however, that there may still be significant generalization errors caused by distributional shift. In particular, they designed their own labeling and showed that, while humans have similar accuracy on ImageNet and this alternative labeling, state-of-the-art algorithms failed to generalize. Distributional shift seems to be a big challenge.

Note: this does not seem to caused by over-optimization.

Overfitting

2020-02-24T09:54:39Z

Lê Nguyên Hoang:

Overfitting occurs where fitting training data too closely is counter-productive to out-of-sample predictions.

== Bias-variance tradeoff ==

[http://web.mit.edu/6.435/www/Geman92.pdf GBD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=neural+networks+and+the+bias%2Fvariance+dilemma+geman+bienenstock+doursat&btnG= 92] identified the bias-variance tradeoff, which quantifies out-of-sample errors as a sum of inductive bias and model variance, for random samples drawn from the true distribution.

Formally, let <math>f(x,S)</math> the <math>y</math>-prediction of the algorithm trained with sample <math>S</math> for feature <math>x</math>. Then <math>\mathbb E_{x,y,S}[(f(x,S)-y)^2] = bias^2 + variance + \sigma^2</math>, where the expectation is over <math>x</math>, <math>S</math> and <math>y</math>, where <math>bias = \mathbb E_{x,S}[f(x,S)-f^*(x)]</math> is the bias with respect to the optimal prediction <math>f^*</math>, where <math>variance = \mathbb E_{x,S}[f(x,S)^2] - \mathbb E_{x,S}[f(x,S)]^2</math> is how much the prediction varies from one sample to the other and <math>\sigma^2 = \mathbb E_{x,y}[(f^*(x)-y)^2]</math> is the unpredictable components of <math>y</math> given <math>x</math>.

It has been argued that overfitting is caused by increased variance when we consider a learning algorithm that is too sensitive to the randomness of sampling <math>S</math>. This sometimes occurs when the number of parameters of the learning algorithm is too large (but not necessarily!).

== PAC-learning ==

To prove theoretical guarantees of non-overfitting, [http://web.mit.edu/6.435/www/Valiant84.pdf Valiant][https://dblp.org/rec/bibtex/journals/cacm/Valiant84 84] introduced the concept of probably approximately correct (PAC) learning. More explanations here: [https://www.youtube.com/watch?v=uB2X2OuD4Rg&list=PLie7a1OUTSagZB9mFZnVBgsNfBtcUGJWB&index=8 Wandida16a].

In particular, the fundamental theorem of statistical learning [https://www.amazon.com/Understanding-Machine-Learning-Theory-Algorithms/dp/1107057132/ ShalevshwartzBendavidBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Understanding+Machine+Learning+Theory+Algorithms+shalev-shwartz+ben-david&btnG= 14] [https://www.youtube.com/watch?v=RkWuLtFPBKU&list=PLie7a1OUTSagZB9mFZnVBgsNfBtcUGJWB&index=14 Wandida16b] provides guarantees of PAC learning, when the number of data points sufficiently exceed the VC dimension of the set of learnable algorithms. Since this VC dimension is often essentially the number of parameters (assuming finite representation of the parameters as float or double), then this means that PAC learning is guaranteed when <math>\#data \ll \#parameters</math>.

This has become a conventional wisdom for a while.

== Test set overfitting ==

There has been concerns about overfitting of test sets, as these are used more and more to measure the performance of machine learning algorithms. But [https://arxiv.org/pdf/1902.10811.pdf RRSS][https://dblp.org/rec/bibtex/conf/icml/RechtRSS19 19] analyze statistical patterns on reported test set performances, and argue that there still is actual progress.

== Double descent ==

However, the conventional wisdom is in sharp contradiction with today's success of deep neural networks [https://openreview.net/pdf?id=Sy8gdB9xx ZBHRV][https://dblp.org/rec/bibtex/conf/iclr/ZhangBHRV17 17], [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] [https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19], but also kernel methods [http://proceedings.mlr.press/v80/belkin18a/belkin18a.pdf BMM][https://dblp.org/rec/bibtex/conf/icml/BelkinMM18 18] [http://proceedings.mlr.press/v89/belkin19a/belkin19a.pdf BRT][https://dblp.org/rec/bibtex/conf/aistats/BelkinRT19 19], ridgeless (random feature) linear regression [https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8849614 MVS][https://dblp.org/rec/bibtex/conf/isit/MuthukumarVS19 19] [https://arxiv.org/pdf/1906.11300 BLLT][https://dblp.org/rec/bibtex/journals/corr/abs-1906-11300 19] [https://arxiv.org/pdf/1908.05355 MeiMontanari][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+generalization+error+of+random+features+regression%3A+Precise+asymptotics+and+double+descent+curve&btnG= 19] [https://arxiv.org/pdf/1903.08560 HMRT][https://dblp.org/rec/bibtex/journals/corr/abs-1903-08560 19] [https://arxiv.org/pdf/1903.07571 BHX][https://dblp.org/rec/bibtex/journals/corr/abs-1903-07571 19] and even ensembles [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19]. Learning algorithms seem to often achieve their best out-of-sample performance when they are massively overparameterized and perfectly fit the training data (called interpolation).

Intriguingly, a double descent phenomenon often occurs [https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] [https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19], where the performance at the test set first behaves as predicted by the bias-variance dilemma, but then improves and outperforms what would be advised by classical statistical learning.

All such results suggest that overfitting eventually disappears, which contradicts conventional wisdom.

== Details ==

[https://openreview.net/pdf?id=Sy8gdB9xx ZBHRV][https://dblp.org/rec/bibtex/conf/iclr/ZhangBHRV17 17] showed that large interpolating neural networks generalize well, even for large noise in the data. Also, they showed that inductive bias likely plays a limited role, as neural networks still manage to learn quite efficiently data whose labels are completely shuffled. They also proved that a neural network with <math>2n+d</math> parameters can interpolate <math>n</math> data points of dimension <math>d</math>.

[https://www.pnas.org/content/pnas/116/32/15849.full.pdf BHMM][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reconciling+modern+machine-learning+practice+and+the+classical+bias%E2%80%93variance+trade-off&btnG= 19] observed double descent for random Fourier features (which [http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf RahimiRecht][https://dblp.org/rec/bibtex/conf/nips/RahimiR07 07] proved to be intimately connected to kernel methods), neural networks, decision tree and ensemble methods.

[https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19] show that a very wide variety of deep neural networks exhibit a wide variety of double descent phenomenons. Not only is there double descent with respect to the number of parameters, but there also seems to be double descent with respect to the width of the neural networks, and weirdly also with respect to epochs of learning steps. They conjecture that "effective model complexity" (the number of data points for which the model is able to achieve small training loss) is a critical point where overfitting occurs. Before and beyond this, overfitting appears to vanish.

[http://proceedings.mlr.press/v80/belkin18a/belkin18a.pdf BMM][https://dblp.org/rec/bibtex/conf/icml/BelkinMM18 18] present experiments that show that interpolating kernel methods also generalize well and are able to fit random labels (though in this paper, they do not exhibit double descent). They also show that, because norms of interpolaters grow superpolynomially in Hilbert space (in <math>exp(\theta(n^{1/d}))</math>), usual bounds controlling overfitting are actually trivial for large datasets. This indicates the need for radically different approach to understand overfitting.

[http://proceedings.mlr.press/v89/belkin19a/belkin19a.pdf BRT][https://dblp.org/rec/bibtex/conf/aistats/BelkinRT19 19] show examples of singular kernel interpolators (<math>K(x,y)\sim||x-y||^{-a}</math>) that achieve optimal rates, even for improper learning (meaning that the true function to learn does not belong to the set of hypotheses).

Note that the connection between kernel methods and neural networks has been made, for instance by [http://papers.nips.cc/paper/3182-random-features-for-large-scale-kernel-machines.pdf RahimiRecht][https://dblp.org/rec/bibtex/conf/nips/RahimiR07 07] and [http://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf JHG][https://dblp.org/rec/bibtex/conf/nips/JacotHG18 18]. Essentially, random features implement approximate kernel methods. And the first layers of neural networks with random (or even trained) weights can be regarded as computations of random features.

[https://arxiv.org/pdf/1903.08560 HMRT][https://dblp.org/rec/bibtex/journals/corr/abs-1903-08560 19] use random matrix theory (to estimate eigenvalues of <math>X^TX</math> when <math>X</math> are random samples) to show that ridgless regression (regression with minimum <math>\ell_2</math>-norm) features "infinite double descent" as the size <math>n</math> of the training data sets grows to infinity, along with the number of parameters <math>p</math> (they assume <math>p/n \rightarrow gamma</math>, and show infinite overfitting for <math>gamma\sim 1</math>). This is shown for both a linear model where <math>X = \Sigma^{1/2} Z</math>, for some fixed <math>\Sigma</math> and some well-behaved <math>Z</math> of mean 0 and variance 1, and a component-wise linearity <math>X = \sigma(WZ)</math>. In both cases, it is assumed that <math>y=x^T\beta+\varepsilon</math>, where <math>\mathbb E[\varepsilon]=0</math>. There are also assumptions of finite fixed variance for <math>\varepsilon</math>, and finite fourth moment for <math>Z</math> (needed for random matrix theory).

[https://arxiv.org/pdf/1903.07571 BHX][https://dblp.org/rec/bibtex/journals/corr/abs-1903-07571 19] analyze two other data models. The former is a classical Gaussian linear regression with a huge space of features. But the regression is only made within a (random) subset <math>T</math> of features, in which case double descent is observed, and errors can be derived from the norms of the true regression for <math>T</math>-coordinates and non-<math>T</math>-coordinates. A similar analysis is then provided for a random Fourier feature model.

[https://arxiv.org/pdf/1908.05355 MeiMontanari][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+generalization+error+of+random+features+regression%3A+Precise+asymptotics+and+double+descent+curve&btnG= 19] consider still another data model, where <math>z</math> is drawn uniformly randomly on a <math>d</math>-sphere, and <math>y = f(z)+\varepsilon</math>, such that <math>\mathbb E[\varepsilon]=0</math> and <math>\varepsilon</math> has finite fourth moment. The ridgeless linear regression is then over some random features <math>x_i = \sigma(Wz_i)</math>, where <math>\sigma</math> applies component-wisely and <math>W</math> has random rows of <math>\ell_2</math>-norm equal to 1. They prove that this yields a double descent.

Social choice

2020-02-23T09:34:50Z

Lê Nguyên Hoang: /* Harsanyi's Utilitarian Theorem */

Social choice is the study of how to elicit, aggregate and explain human preferences for collective decision-making. This is critical to AI ethics, as we will need to decide collectively on which ethics an AI will follow. For instance, what video should be recommended by the YouTube algorithm when a user queries "Trump", "vaccine" or "social justice"?

== Background ==

Social choice theory arguably started with a remarkable memoire by Condorcet [http://classiques.uqac.ca/classiques/condorcet/Essai_application_discours_preliminaire/discours_preliminaire.pdf Condorcet][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Essai+sur+l%27application+de+l%27analyse+%C3%A0+la+probabilit%C3%A9+des+d%C3%A9cisions+rendues+%C3%A0+la+pluralit%C3%A9+des+voix.+1785&btnG= 1785]. He argued that if one alternative is preferred to any other alternative by a majority then it should be selected. This is the [https://en.wikipedia.org/wiki/Condorcet_criterion Condorcet principle] (see [https://www.youtube.com/watch?v=hI89r4LqaCc MrPhi17aFR], [https://www.youtube.com/watch?v=ZZb4TjvupkI MrPhi17bFR]).

Unfortunately, social choice theory is plagued with impossibility results, like the Condorcet paradox ([https://www.youtube.com/watch?v=HoAnYQZrNrQ PBSInfinite17], [https://www.youtube.com/watch?v=v8-2YdUqQqM MicMaths15FR]), Arrow's impossibility theorem ([https://s3.amazonaws.com/academia.edu.documents/40888103/arrow.pdf?response-content-disposition=inline%3B%20filename%3DArrow.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200118T214628Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=283c657c6dc5c2c225f77e14996a77846974b6dd3a2008f4e299131c6255fd75 Arrow][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Difficulty+in+the+Concept+of+Social+Welfare&btnG= 50], [https://www.youtube.com/watch?v=AhVR7gFMKNg PBSInfinite17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]) and the Gibbard-Satterthwaite theorem ([https://www.youtube.com/watch?v=m5crte26fiw Wandida17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]). Different voting systems yield different winners ([https://www.youtube.com/watch?v=vfTJ4vmIsO4 StatChat16FR], [https://www.youtube.com/watch?v=fBYCoPAmpr4&t=371s S4A17aFR]).

Today's most convincing social choice mechanisms are probably the [https://en.wikipedia.org/wiki/Approval_voting approval voting] ([https://dblp.org/rec/bibtex/books/daglib/0017739 BramsFishburnBook07], [https://www.youtube.com/watch?v=orybDrUj4vA CGPGrey]), [https://en.wikipedia.org/wiki/Majority_judgment majority judgment] ([https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=majority+judgment+laraki+balinski&btnG= BalinksiLarakiBook11], [https://www.youtube.com/watch?v=ZoGH7d51bvc ScienceÉtonnante16FR], [https://www.youtube.com/watch?v=_MAo8pUl0U4 S4A17cFR]) and the randomized Condorcet voting system ([https://dspace.mit.edu/bitstream/handle/1721.1/107673/355_2017_1031_ReferencePDF.pdf?sequence=1&isAllowed=y Hoang][https://dblp.org/rec/bibtex/journals/scw/Hoang17 17], [https://www.youtube.com/watch?v=wKimU8jy2a8 S4A17dFR], [https://www.youtube.com/watch?v=vAdGZkXhlNM S4A17eFR]).

== Harsanyi's Utilitarian Theorem ==

[http://darp.lse.ac.uk/papersDB/Harsanyi_(JPolE_55).pdf Harsanyi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=cardinal+welfare%2C+individualistic+ethics%2C+harsanyi&btnG= 55] proved that weighted sums of individuals' utilities are the only social choice mechanisms that aggregate [[Von Neumann-Morgenstern preferences]] to yield Von Neumann-Morgenstern group preferences in such a way that, if every individual of the group is indifferent between probabilistic outcomes <math>p</math> and <math>q</math>, then so is the group. This is a compelling argument for a simple addition of individuals' preferences.

To prove it, consider a finite set of alternatives. Denote <math>u_{ij}</math> is the utility of individual <math>i</math> for alternative <math>j</math>. Individual <math>i</math>'s utility for a probability <math>p</math> equals <math>(up)_i</math>. Denoting <math>v_j</math> the group's utility for alternative <math>j</math>, the indifference property says that <math>up=uq</math> implies <math>v^Tp = v^Tq</math>. Using linear algebra and the fact that this holds for all <math>p, q</math> then implies that <math>v</math> belongs to the vector space spanned by the <math>u_i</math>'s. This results in saying that <math>w = a + \sum w_i u_i</math>, for nonnegative weights <math>w_i</math>'s (and arbitrary constant <math>a</math>).

However, there are major caveats to applying this mechanism in practice. First note that the scaling of different individuals' utility functions (or equivalently of their weights) remains to be settled, which does not seem straightforward to be done. But most importantly, this social choice mechanism is not incentive-compatible. If implemented, individuals will have incentives to exaggerate their preferences (or to tell their representatives to do so). Finally, such expected utility maximization will surely turn into the maximization of some expected proxies, which would then be prone to [[Goodhart's law]].

== The (Huge) Flaw of Classical Social Choice ==

Unfortunately, such approaches are limited because they can only handle a reasonable amount of alternatives. If we are to design AI ethics collectively, we need to choose a code (or, say, guidelines or texts of laws). Yet there are combinatorially many such codes! If we consider 1,000-line codes, this would represent ~210,000 alternatives. Classical voting systems won't do the trick.

Now, there are already lots of results in social choice for structured combinatorial sets of alternatives, mostly derived from auction theory ([https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism VCG mechanism] [https://dblp.org/rec/bibtex/books/cu/NRTV2007 NRTV07] [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR], Myerson's auction [https://dblp.org/rec/bibtex/journals/mor/Myerson81 Myerson81] [https://www.youtube.com/watch?v=FjP5JMUVXxw S4A17gFR], Gale-Shapley [https://dblp.org/rec/bibtex/journals/tamm/GaleS13 GaleShapley62] [https://www.youtube.com/watch?v=Qcv1IqHWAzg Numberphile14] [https://www.youtube.com/watch?v=oHYcOXi06uY S4A17hFR]...). Most impressively, in a series of papers [https://dblp.org/rec/bibtex/journals/sigecom/CaiDW11 CDW11], [https://dblp.org/rec/bibtex/conf/stoc/CaiDW12 CDW12a], [https://dblp.org/rec/bibtex/conf/focs/CaiDW12 CDW12b], [https://dblp.org/rec/bibtex/conf/soda/CaiDW13 CDW13a] and [https://dblp.org/rec/bibtex/conf/focs/CaiDW13 CDW13b], Cai, [https://en.wikipedia.org/wiki/Constantinos_Daskalakis Daskalakis] and Weinberg proved that the polynomial tractability of a Bayesian social choice approximation problem (i.e. with incentive-compatibility constraints) is equivalent to that of the full-information problem with an additional social welfare term to be optimized (see [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR]).

However, we surely have to also tackle the case of unstructured combinatorial sets of alternatives (also, polytime may be too slow in practice).

== Bounds for limited communication complexity ==

A fascinating result by [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19] shows that the worst-case lost of social welfare due to polynomial communication complexity (voters communicate at most log(#alternatives) bits) is unbounded (#alternatives2 for deterministic elicitation+voting, #alternatives for randomized).

There are caveats though. For one thing, this is a worst-case analysis. But human preferences may be more structured. Also, priors can be invoked. Plus, authors assumed that the same elicitation was applied to all voters, which is clearly suboptimal. A lot more research on the communication-complexity versus social-welfare tradeoff is definitely desired. This is exciting!

== Applications to AI Ethics ==

It has been argued to be critical to solve the problem of AI ethics ([https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12457/12204 GRTVW][https://dblp.org/rec/bibtex/conf/aaai/GreeneRTVW16 16],[http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Ethics_Conitzer_etal.pdf CSBDK][https://dblp.org/rec/bibtex/conf/isaim/ConitzerSBD018 17]). In brief, we are unlikely to agree on what ethics to program. However, we might be able to agree on how to agree on some ethics to program even though we disagree. The trick to implement some (virtual) democratic voting on moral preferences.

Interestingly, ideas along these lines have already been developed for the cases of autonomous car dilemmas [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17052/15857 NGADR+][https://dblp.org/rec/bibtex/conf/aaai/NoothigattuGADR18 18] [https://www.youtube.com/watch?v=Y6jfGZXubq0 UpAndAtom18], kidney transplant [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17384/15863 FBSDC][https://dblp.org/rec/bibtex/conf/aaai/FreedmanBSDC18 18] and food donation (called WeBuildAI [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=2ahUKEwj4y5-ggu3mAhXB2qQKHT6GDZ4QFjACegQIARAC&url=https%3A%2F%2Fwww.cs.cmu.edu%2F~akahng%2Fpapers%2Fwebuildai.pdf&usg=AOvVaw2BknquyvgNufy-JlCoPj_G LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19]).

Note that in all such applications, the set of alternatives is combinatorially large. The trick to perform voting with limited elicitation from voters is to collect binary-choice-based preferences, and to then extrapolate preferences for other cases using machine learning (with some inductive bias). Another way to interpret this is to consider that voters get substituted by digital surrogates, whose task is to answer just as the voters would. This is kind of like representative democracy, where voters are replaced by their representatives. But machine learning can allow individual representatives through customized surrogates!

To build trust in the surrogate, WeBuildAI proposes to voters to test their surrogates, and to replace it, if needs be, by some computational model of their owns. They show that such interactions build trust from voters. They also propose some [[interpretability]] framework, where voters are given the implications of the vote of their surrogates.

Now what Lê can't wait for, is for all such frameworks to be applied to problems that really matter, because they influence billions of people. Yes, Lê is (again!) talking about recommender algorithms of social medias like YouTube. How should hate speech be moderated? What should be shown to someone who wants to learn about climate change? Should there be an additional tax on, say, car advertisements? Should angering videos be less viral?

Lê would be thrilled to see social choice theory applied to such critical moral questions.

== Scaled voting ==

One frequent remark that is being made is whether we really can (and should) agree on ethical issues. For instance, [https://www.nature.com/articles/s41586-018-0637-6 ADKSH+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Moral+Machine+experiment+nature+2018&btnG= 18] showed that Japanese prefer to save walkers, while Chinese prefer to save car passengers. Should we really enforce a common ethics worldwide?

Well, we probably don't need to. Cars could be programmed to save Japanese walkers and Chinese car passengers. They could be made to defend freedom in US and baguette in France. While humans usually have preferences for what happens elsewhere in the world, they usually have stronger preferences for what happens near their home. This probably is something that should be considered when designing voting-based ethics.

One proposal to reflect such nuances is [https://en.wikipedia.org/wiki/Quadratic_voting quadratic voting] [https://www.sss.ias.edu/files/pdfs/Rodrik/workshop%2014-15/Weyl-Quadratic_Voting.pdf LalleyWeyl][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=quadratic+voting+lalley+weyl&btnG=&oq=quadratic+voting 18], which can be made secure [https://link.springer.com/article/10.1007/s11127-017-0407-2 ParkRivest][https://dblp.org/rec/bibtex/journals/iacr/ParkR16 16]. In quadratic voting, a voter who wants its vote to weigh n times more must pay n2. This guarantees (asymptotic) efficiency (utilitarian outcome) and incentive-compatibility. But quadratic voting only applies to 2-alternative votes (typically statu quo vs new law) and is manipulable by collusion.

Another interesting point to be made about multidimensional voting is that the (geometric) median is strategy-proof for voters with a peaked preference, and a valuation that decreases with the distance to the peaked preference. The geometric median is particularly suited for, say, determining budget allocation through social choice. Weirdly, we don't know of a neat paper on this, though using sum of distance minimizer is well-known (need citation).

== Preferences versus volitions ==

It's been argued [https://intelligence.org/files/CEV.pdf Yudkowsky][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+yudkowsky&btnG= 04], [https://www.izmemar.com/files/CEV-MachineEthics.pdf Tarleton][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+tarleton&btnG= 10], [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.674.6424&rep=rep1&type=pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=the+value+learning+problem+soares&btnG= 15], [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19] that we surely should not aggregate today's human moral preferences, because of [[cognitive biases]] [https://www.amazon.com/s?k=thinking+fast+and+slow&ref=nb_sb_noss KahnemanBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=thinking+fast+and+slow+kahneman+2011&btnG= 11]. Mostly, our preferences are inconsistent, manipulable via framing, time-dependent, subject to addictions, and so on. We are likely to regret today's claimed preferences in the future, or as soon as we better understand their consequences. Instead, it is argued, we should program human [[volition|volitions]], which corresponds to what we would prefer to prefer, instead of what we simply prefer.

Unfortunately, a lot more research is needed to better formalize and analyze the concept of volition, and how it diverges from preferences. One fruitful path may be to analyze the difference between what's learned through [[inverse reinforcement learning]], as opposed to through (well-framed) elicitation. See [[volition]] for a lot more discussion on this problem.

Social choice

2020-02-23T09:34:03Z

Lê Nguyên Hoang: /* Harsanyi's Utilitarian Theorem */

Social choice is the study of how to elicit, aggregate and explain human preferences for collective decision-making. This is critical to AI ethics, as we will need to decide collectively on which ethics an AI will follow. For instance, what video should be recommended by the YouTube algorithm when a user queries "Trump", "vaccine" or "social justice"?

== Background ==

Social choice theory arguably started with a remarkable memoire by Condorcet [http://classiques.uqac.ca/classiques/condorcet/Essai_application_discours_preliminaire/discours_preliminaire.pdf Condorcet][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Essai+sur+l%27application+de+l%27analyse+%C3%A0+la+probabilit%C3%A9+des+d%C3%A9cisions+rendues+%C3%A0+la+pluralit%C3%A9+des+voix.+1785&btnG= 1785]. He argued that if one alternative is preferred to any other alternative by a majority then it should be selected. This is the [https://en.wikipedia.org/wiki/Condorcet_criterion Condorcet principle] (see [https://www.youtube.com/watch?v=hI89r4LqaCc MrPhi17aFR], [https://www.youtube.com/watch?v=ZZb4TjvupkI MrPhi17bFR]).

Unfortunately, social choice theory is plagued with impossibility results, like the Condorcet paradox ([https://www.youtube.com/watch?v=HoAnYQZrNrQ PBSInfinite17], [https://www.youtube.com/watch?v=v8-2YdUqQqM MicMaths15FR]), Arrow's impossibility theorem ([https://s3.amazonaws.com/academia.edu.documents/40888103/arrow.pdf?response-content-disposition=inline%3B%20filename%3DArrow.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200118T214628Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=283c657c6dc5c2c225f77e14996a77846974b6dd3a2008f4e299131c6255fd75 Arrow][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Difficulty+in+the+Concept+of+Social+Welfare&btnG= 50], [https://www.youtube.com/watch?v=AhVR7gFMKNg PBSInfinite17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]) and the Gibbard-Satterthwaite theorem ([https://www.youtube.com/watch?v=m5crte26fiw Wandida17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]). Different voting systems yield different winners ([https://www.youtube.com/watch?v=vfTJ4vmIsO4 StatChat16FR], [https://www.youtube.com/watch?v=fBYCoPAmpr4&t=371s S4A17aFR]).

Today's most convincing social choice mechanisms are probably the [https://en.wikipedia.org/wiki/Approval_voting approval voting] ([https://dblp.org/rec/bibtex/books/daglib/0017739 BramsFishburnBook07], [https://www.youtube.com/watch?v=orybDrUj4vA CGPGrey]), [https://en.wikipedia.org/wiki/Majority_judgment majority judgment] ([https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=majority+judgment+laraki+balinski&btnG= BalinksiLarakiBook11], [https://www.youtube.com/watch?v=ZoGH7d51bvc ScienceÉtonnante16FR], [https://www.youtube.com/watch?v=_MAo8pUl0U4 S4A17cFR]) and the randomized Condorcet voting system ([https://dspace.mit.edu/bitstream/handle/1721.1/107673/355_2017_1031_ReferencePDF.pdf?sequence=1&isAllowed=y Hoang][https://dblp.org/rec/bibtex/journals/scw/Hoang17 17], [https://www.youtube.com/watch?v=wKimU8jy2a8 S4A17dFR], [https://www.youtube.com/watch?v=vAdGZkXhlNM S4A17eFR]).

== Harsanyi's Utilitarian Theorem ==

[http://darp.lse.ac.uk/papersDB/Harsanyi_(JPolE_55).pdf Harsanyi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=cardinal+welfare%2C+individualistic+ethics%2C+harsanyi&btnG= 55] proved that weighted sums of individuals' utilities are the only social choice mechanisms that aggregate [[Von Neumann-Morgenstern preferences]] to yield Von Neumann-Morgenstern group preferences in such a way that, if every individual of the group is indifferent between probabilistic outcomes <math>p</math> and <math>q</math>, then so is the group. This is a compelling argument for a simple addition of individuals' preferences.

To prove it, consider a finite set of alternatives. Denote <math>u_{ij}</math> is the utility of individual <math>i</math> for alternative <math>j</math>. Individual <math>i</math>'s utility for a probability <math>p</math> equals <math>(up)_i</math>. Denoting <math>v_j</math> the group's utility for alternative <math>j</math>, the indifference property says that <math>u (p-q) = 0</math> implies <math>v^T (p-q) = 0</math>. Using linear algebra and the fact that this holds for all <math>p, q</math> then implies that <math>v</math> belongs to the vector space spanned by the <math>u_i</math>'s. This results in saying that <math>w = a + \sum w_i u_i</math>, for nonnegative weights <math>w_i</math>'s (and arbitrary constant <math>a</math>).

However, there are major caveats to applying this mechanism in practice. First note that the scaling of different individuals' utility functions (or equivalently of their weights) remains to be settled, which does not seem straightforward to be done. But most importantly, this social choice mechanism is not incentive-compatible. If implemented, individuals will have incentives to exaggerate their preferences (or to tell their representatives to do so). Finally, such expected utility maximization will surely turn into the maximization of some expected proxies, which would then be prone to [[Goodhart's law]].

== The (Huge) Flaw of Classical Social Choice ==

Unfortunately, such approaches are limited because they can only handle a reasonable amount of alternatives. If we are to design AI ethics collectively, we need to choose a code (or, say, guidelines or texts of laws). Yet there are combinatorially many such codes! If we consider 1,000-line codes, this would represent ~210,000 alternatives. Classical voting systems won't do the trick.

Now, there are already lots of results in social choice for structured combinatorial sets of alternatives, mostly derived from auction theory ([https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism VCG mechanism] [https://dblp.org/rec/bibtex/books/cu/NRTV2007 NRTV07] [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR], Myerson's auction [https://dblp.org/rec/bibtex/journals/mor/Myerson81 Myerson81] [https://www.youtube.com/watch?v=FjP5JMUVXxw S4A17gFR], Gale-Shapley [https://dblp.org/rec/bibtex/journals/tamm/GaleS13 GaleShapley62] [https://www.youtube.com/watch?v=Qcv1IqHWAzg Numberphile14] [https://www.youtube.com/watch?v=oHYcOXi06uY S4A17hFR]...). Most impressively, in a series of papers [https://dblp.org/rec/bibtex/journals/sigecom/CaiDW11 CDW11], [https://dblp.org/rec/bibtex/conf/stoc/CaiDW12 CDW12a], [https://dblp.org/rec/bibtex/conf/focs/CaiDW12 CDW12b], [https://dblp.org/rec/bibtex/conf/soda/CaiDW13 CDW13a] and [https://dblp.org/rec/bibtex/conf/focs/CaiDW13 CDW13b], Cai, [https://en.wikipedia.org/wiki/Constantinos_Daskalakis Daskalakis] and Weinberg proved that the polynomial tractability of a Bayesian social choice approximation problem (i.e. with incentive-compatibility constraints) is equivalent to that of the full-information problem with an additional social welfare term to be optimized (see [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR]).

However, we surely have to also tackle the case of unstructured combinatorial sets of alternatives (also, polytime may be too slow in practice).

== Bounds for limited communication complexity ==

A fascinating result by [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19] shows that the worst-case lost of social welfare due to polynomial communication complexity (voters communicate at most log(#alternatives) bits) is unbounded (#alternatives2 for deterministic elicitation+voting, #alternatives for randomized).

There are caveats though. For one thing, this is a worst-case analysis. But human preferences may be more structured. Also, priors can be invoked. Plus, authors assumed that the same elicitation was applied to all voters, which is clearly suboptimal. A lot more research on the communication-complexity versus social-welfare tradeoff is definitely desired. This is exciting!

== Applications to AI Ethics ==

It has been argued to be critical to solve the problem of AI ethics ([https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12457/12204 GRTVW][https://dblp.org/rec/bibtex/conf/aaai/GreeneRTVW16 16],[http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Ethics_Conitzer_etal.pdf CSBDK][https://dblp.org/rec/bibtex/conf/isaim/ConitzerSBD018 17]). In brief, we are unlikely to agree on what ethics to program. However, we might be able to agree on how to agree on some ethics to program even though we disagree. The trick to implement some (virtual) democratic voting on moral preferences.

Interestingly, ideas along these lines have already been developed for the cases of autonomous car dilemmas [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17052/15857 NGADR+][https://dblp.org/rec/bibtex/conf/aaai/NoothigattuGADR18 18] [https://www.youtube.com/watch?v=Y6jfGZXubq0 UpAndAtom18], kidney transplant [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17384/15863 FBSDC][https://dblp.org/rec/bibtex/conf/aaai/FreedmanBSDC18 18] and food donation (called WeBuildAI [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=2ahUKEwj4y5-ggu3mAhXB2qQKHT6GDZ4QFjACegQIARAC&url=https%3A%2F%2Fwww.cs.cmu.edu%2F~akahng%2Fpapers%2Fwebuildai.pdf&usg=AOvVaw2BknquyvgNufy-JlCoPj_G LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19]).

Note that in all such applications, the set of alternatives is combinatorially large. The trick to perform voting with limited elicitation from voters is to collect binary-choice-based preferences, and to then extrapolate preferences for other cases using machine learning (with some inductive bias). Another way to interpret this is to consider that voters get substituted by digital surrogates, whose task is to answer just as the voters would. This is kind of like representative democracy, where voters are replaced by their representatives. But machine learning can allow individual representatives through customized surrogates!

To build trust in the surrogate, WeBuildAI proposes to voters to test their surrogates, and to replace it, if needs be, by some computational model of their owns. They show that such interactions build trust from voters. They also propose some [[interpretability]] framework, where voters are given the implications of the vote of their surrogates.

Now what Lê can't wait for, is for all such frameworks to be applied to problems that really matter, because they influence billions of people. Yes, Lê is (again!) talking about recommender algorithms of social medias like YouTube. How should hate speech be moderated? What should be shown to someone who wants to learn about climate change? Should there be an additional tax on, say, car advertisements? Should angering videos be less viral?

Lê would be thrilled to see social choice theory applied to such critical moral questions.

== Scaled voting ==

One frequent remark that is being made is whether we really can (and should) agree on ethical issues. For instance, [https://www.nature.com/articles/s41586-018-0637-6 ADKSH+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Moral+Machine+experiment+nature+2018&btnG= 18] showed that Japanese prefer to save walkers, while Chinese prefer to save car passengers. Should we really enforce a common ethics worldwide?

Well, we probably don't need to. Cars could be programmed to save Japanese walkers and Chinese car passengers. They could be made to defend freedom in US and baguette in France. While humans usually have preferences for what happens elsewhere in the world, they usually have stronger preferences for what happens near their home. This probably is something that should be considered when designing voting-based ethics.

One proposal to reflect such nuances is [https://en.wikipedia.org/wiki/Quadratic_voting quadratic voting] [https://www.sss.ias.edu/files/pdfs/Rodrik/workshop%2014-15/Weyl-Quadratic_Voting.pdf LalleyWeyl][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=quadratic+voting+lalley+weyl&btnG=&oq=quadratic+voting 18], which can be made secure [https://link.springer.com/article/10.1007/s11127-017-0407-2 ParkRivest][https://dblp.org/rec/bibtex/journals/iacr/ParkR16 16]. In quadratic voting, a voter who wants its vote to weigh n times more must pay n2. This guarantees (asymptotic) efficiency (utilitarian outcome) and incentive-compatibility. But quadratic voting only applies to 2-alternative votes (typically statu quo vs new law) and is manipulable by collusion.

Another interesting point to be made about multidimensional voting is that the (geometric) median is strategy-proof for voters with a peaked preference, and a valuation that decreases with the distance to the peaked preference. The geometric median is particularly suited for, say, determining budget allocation through social choice. Weirdly, we don't know of a neat paper on this, though using sum of distance minimizer is well-known (need citation).

== Preferences versus volitions ==

It's been argued [https://intelligence.org/files/CEV.pdf Yudkowsky][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+yudkowsky&btnG= 04], [https://www.izmemar.com/files/CEV-MachineEthics.pdf Tarleton][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+tarleton&btnG= 10], [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.674.6424&rep=rep1&type=pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=the+value+learning+problem+soares&btnG= 15], [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19] that we surely should not aggregate today's human moral preferences, because of [[cognitive biases]] [https://www.amazon.com/s?k=thinking+fast+and+slow&ref=nb_sb_noss KahnemanBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=thinking+fast+and+slow+kahneman+2011&btnG= 11]. Mostly, our preferences are inconsistent, manipulable via framing, time-dependent, subject to addictions, and so on. We are likely to regret today's claimed preferences in the future, or as soon as we better understand their consequences. Instead, it is argued, we should program human [[volition|volitions]], which corresponds to what we would prefer to prefer, instead of what we simply prefer.

Unfortunately, a lot more research is needed to better formalize and analyze the concept of volition, and how it diverges from preferences. One fruitful path may be to analyze the difference between what's learned through [[inverse reinforcement learning]], as opposed to through (well-framed) elicitation. See [[volition]] for a lot more discussion on this problem.

Social choice

2020-02-23T09:32:50Z

Lê Nguyên Hoang: /* Harsanyi's Utilitarian Theorem */

Social choice is the study of how to elicit, aggregate and explain human preferences for collective decision-making. This is critical to AI ethics, as we will need to decide collectively on which ethics an AI will follow. For instance, what video should be recommended by the YouTube algorithm when a user queries "Trump", "vaccine" or "social justice"?

== Background ==

Social choice theory arguably started with a remarkable memoire by Condorcet [http://classiques.uqac.ca/classiques/condorcet/Essai_application_discours_preliminaire/discours_preliminaire.pdf Condorcet][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Essai+sur+l%27application+de+l%27analyse+%C3%A0+la+probabilit%C3%A9+des+d%C3%A9cisions+rendues+%C3%A0+la+pluralit%C3%A9+des+voix.+1785&btnG= 1785]. He argued that if one alternative is preferred to any other alternative by a majority then it should be selected. This is the [https://en.wikipedia.org/wiki/Condorcet_criterion Condorcet principle] (see [https://www.youtube.com/watch?v=hI89r4LqaCc MrPhi17aFR], [https://www.youtube.com/watch?v=ZZb4TjvupkI MrPhi17bFR]).

Unfortunately, social choice theory is plagued with impossibility results, like the Condorcet paradox ([https://www.youtube.com/watch?v=HoAnYQZrNrQ PBSInfinite17], [https://www.youtube.com/watch?v=v8-2YdUqQqM MicMaths15FR]), Arrow's impossibility theorem ([https://s3.amazonaws.com/academia.edu.documents/40888103/arrow.pdf?response-content-disposition=inline%3B%20filename%3DArrow.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200118T214628Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=283c657c6dc5c2c225f77e14996a77846974b6dd3a2008f4e299131c6255fd75 Arrow][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Difficulty+in+the+Concept+of+Social+Welfare&btnG= 50], [https://www.youtube.com/watch?v=AhVR7gFMKNg PBSInfinite17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]) and the Gibbard-Satterthwaite theorem ([https://www.youtube.com/watch?v=m5crte26fiw Wandida17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]). Different voting systems yield different winners ([https://www.youtube.com/watch?v=vfTJ4vmIsO4 StatChat16FR], [https://www.youtube.com/watch?v=fBYCoPAmpr4&t=371s S4A17aFR]).

Today's most convincing social choice mechanisms are probably the [https://en.wikipedia.org/wiki/Approval_voting approval voting] ([https://dblp.org/rec/bibtex/books/daglib/0017739 BramsFishburnBook07], [https://www.youtube.com/watch?v=orybDrUj4vA CGPGrey]), [https://en.wikipedia.org/wiki/Majority_judgment majority judgment] ([https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=majority+judgment+laraki+balinski&btnG= BalinksiLarakiBook11], [https://www.youtube.com/watch?v=ZoGH7d51bvc ScienceÉtonnante16FR], [https://www.youtube.com/watch?v=_MAo8pUl0U4 S4A17cFR]) and the randomized Condorcet voting system ([https://dspace.mit.edu/bitstream/handle/1721.1/107673/355_2017_1031_ReferencePDF.pdf?sequence=1&isAllowed=y Hoang][https://dblp.org/rec/bibtex/journals/scw/Hoang17 17], [https://www.youtube.com/watch?v=wKimU8jy2a8 S4A17dFR], [https://www.youtube.com/watch?v=vAdGZkXhlNM S4A17eFR]).

== Harsanyi's Utilitarian Theorem ==

[http://darp.lse.ac.uk/papersDB/Harsanyi_(JPolE_55).pdf Harsanyi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=cardinal+welfare%2C+individualistic+ethics%2C+harsanyi&btnG= 55] proved that weighted sums of individuals' utilities are the only social choice mechanisms that aggregate [[Von Neumann-Morgenstern preferences]] to yield Von Neumann-Morgenstern group preferences in such a way that, if every individual of the group is indifferent between probabilistic outcomes <math>p</math> and <math>q</math>, then so is the group. This is a compelling argument for a simple addition of individuals' preferences.

To prove it, consider a finite set of alternatives. Denote <math>u_{ij}</math> is the utility of individual <math>i</math> for alternative <math>j</math>. Individual <math>i</math>'s utility for a probability <math>p</math> equals <math>u_i^T p</math>. Denoting <math>v_j</math> the group's utility for alternative <math>j</math>, the indifference property says that <math>u (p-q) = 0</math> implies <math>v^T (p-q) = 0</math>. Using linear algebra and the fact that this holds for all <math>p, q</math> then implies that <math>v</math> belongs to the vector space spanned by the <math>u_i</math>'s. This results in saying that <math>w = a + \sum w_i u_i</math>, for nonnegative weights <math>w_i</math>'s (and arbitrary constant <math>a</math>).

However, there are major caveats to applying this mechanism in practice. First note that the scaling of different individuals' utility functions (or equivalently of their weights) remains to be settled, which does not seem straightforward to be done. But most importantly, this social choice mechanism is not incentive-compatible. If implemented, individuals will have incentives to exaggerate their preferences (or to tell their representatives to do so). Finally, such expected utility maximization will surely turn into the maximization of some expected proxies, which would then be prone to [[Goodhart's law]].

== The (Huge) Flaw of Classical Social Choice ==

Unfortunately, such approaches are limited because they can only handle a reasonable amount of alternatives. If we are to design AI ethics collectively, we need to choose a code (or, say, guidelines or texts of laws). Yet there are combinatorially many such codes! If we consider 1,000-line codes, this would represent ~210,000 alternatives. Classical voting systems won't do the trick.

Now, there are already lots of results in social choice for structured combinatorial sets of alternatives, mostly derived from auction theory ([https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism VCG mechanism] [https://dblp.org/rec/bibtex/books/cu/NRTV2007 NRTV07] [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR], Myerson's auction [https://dblp.org/rec/bibtex/journals/mor/Myerson81 Myerson81] [https://www.youtube.com/watch?v=FjP5JMUVXxw S4A17gFR], Gale-Shapley [https://dblp.org/rec/bibtex/journals/tamm/GaleS13 GaleShapley62] [https://www.youtube.com/watch?v=Qcv1IqHWAzg Numberphile14] [https://www.youtube.com/watch?v=oHYcOXi06uY S4A17hFR]...). Most impressively, in a series of papers [https://dblp.org/rec/bibtex/journals/sigecom/CaiDW11 CDW11], [https://dblp.org/rec/bibtex/conf/stoc/CaiDW12 CDW12a], [https://dblp.org/rec/bibtex/conf/focs/CaiDW12 CDW12b], [https://dblp.org/rec/bibtex/conf/soda/CaiDW13 CDW13a] and [https://dblp.org/rec/bibtex/conf/focs/CaiDW13 CDW13b], Cai, [https://en.wikipedia.org/wiki/Constantinos_Daskalakis Daskalakis] and Weinberg proved that the polynomial tractability of a Bayesian social choice approximation problem (i.e. with incentive-compatibility constraints) is equivalent to that of the full-information problem with an additional social welfare term to be optimized (see [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR]).

However, we surely have to also tackle the case of unstructured combinatorial sets of alternatives (also, polytime may be too slow in practice).

== Bounds for limited communication complexity ==

A fascinating result by [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19] shows that the worst-case lost of social welfare due to polynomial communication complexity (voters communicate at most log(#alternatives) bits) is unbounded (#alternatives2 for deterministic elicitation+voting, #alternatives for randomized).

There are caveats though. For one thing, this is a worst-case analysis. But human preferences may be more structured. Also, priors can be invoked. Plus, authors assumed that the same elicitation was applied to all voters, which is clearly suboptimal. A lot more research on the communication-complexity versus social-welfare tradeoff is definitely desired. This is exciting!

== Applications to AI Ethics ==

It has been argued to be critical to solve the problem of AI ethics ([https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12457/12204 GRTVW][https://dblp.org/rec/bibtex/conf/aaai/GreeneRTVW16 16],[http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Ethics_Conitzer_etal.pdf CSBDK][https://dblp.org/rec/bibtex/conf/isaim/ConitzerSBD018 17]). In brief, we are unlikely to agree on what ethics to program. However, we might be able to agree on how to agree on some ethics to program even though we disagree. The trick to implement some (virtual) democratic voting on moral preferences.

Interestingly, ideas along these lines have already been developed for the cases of autonomous car dilemmas [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17052/15857 NGADR+][https://dblp.org/rec/bibtex/conf/aaai/NoothigattuGADR18 18] [https://www.youtube.com/watch?v=Y6jfGZXubq0 UpAndAtom18], kidney transplant [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17384/15863 FBSDC][https://dblp.org/rec/bibtex/conf/aaai/FreedmanBSDC18 18] and food donation (called WeBuildAI [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=2ahUKEwj4y5-ggu3mAhXB2qQKHT6GDZ4QFjACegQIARAC&url=https%3A%2F%2Fwww.cs.cmu.edu%2F~akahng%2Fpapers%2Fwebuildai.pdf&usg=AOvVaw2BknquyvgNufy-JlCoPj_G LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19]).

Note that in all such applications, the set of alternatives is combinatorially large. The trick to perform voting with limited elicitation from voters is to collect binary-choice-based preferences, and to then extrapolate preferences for other cases using machine learning (with some inductive bias). Another way to interpret this is to consider that voters get substituted by digital surrogates, whose task is to answer just as the voters would. This is kind of like representative democracy, where voters are replaced by their representatives. But machine learning can allow individual representatives through customized surrogates!

To build trust in the surrogate, WeBuildAI proposes to voters to test their surrogates, and to replace it, if needs be, by some computational model of their owns. They show that such interactions build trust from voters. They also propose some [[interpretability]] framework, where voters are given the implications of the vote of their surrogates.

Now what Lê can't wait for, is for all such frameworks to be applied to problems that really matter, because they influence billions of people. Yes, Lê is (again!) talking about recommender algorithms of social medias like YouTube. How should hate speech be moderated? What should be shown to someone who wants to learn about climate change? Should there be an additional tax on, say, car advertisements? Should angering videos be less viral?

Lê would be thrilled to see social choice theory applied to such critical moral questions.

== Scaled voting ==

One frequent remark that is being made is whether we really can (and should) agree on ethical issues. For instance, [https://www.nature.com/articles/s41586-018-0637-6 ADKSH+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Moral+Machine+experiment+nature+2018&btnG= 18] showed that Japanese prefer to save walkers, while Chinese prefer to save car passengers. Should we really enforce a common ethics worldwide?

Well, we probably don't need to. Cars could be programmed to save Japanese walkers and Chinese car passengers. They could be made to defend freedom in US and baguette in France. While humans usually have preferences for what happens elsewhere in the world, they usually have stronger preferences for what happens near their home. This probably is something that should be considered when designing voting-based ethics.

One proposal to reflect such nuances is [https://en.wikipedia.org/wiki/Quadratic_voting quadratic voting] [https://www.sss.ias.edu/files/pdfs/Rodrik/workshop%2014-15/Weyl-Quadratic_Voting.pdf LalleyWeyl][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=quadratic+voting+lalley+weyl&btnG=&oq=quadratic+voting 18], which can be made secure [https://link.springer.com/article/10.1007/s11127-017-0407-2 ParkRivest][https://dblp.org/rec/bibtex/journals/iacr/ParkR16 16]. In quadratic voting, a voter who wants its vote to weigh n times more must pay n2. This guarantees (asymptotic) efficiency (utilitarian outcome) and incentive-compatibility. But quadratic voting only applies to 2-alternative votes (typically statu quo vs new law) and is manipulable by collusion.

Another interesting point to be made about multidimensional voting is that the (geometric) median is strategy-proof for voters with a peaked preference, and a valuation that decreases with the distance to the peaked preference. The geometric median is particularly suited for, say, determining budget allocation through social choice. Weirdly, we don't know of a neat paper on this, though using sum of distance minimizer is well-known (need citation).

== Preferences versus volitions ==

It's been argued [https://intelligence.org/files/CEV.pdf Yudkowsky][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+yudkowsky&btnG= 04], [https://www.izmemar.com/files/CEV-MachineEthics.pdf Tarleton][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+tarleton&btnG= 10], [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.674.6424&rep=rep1&type=pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=the+value+learning+problem+soares&btnG= 15], [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19] that we surely should not aggregate today's human moral preferences, because of [[cognitive biases]] [https://www.amazon.com/s?k=thinking+fast+and+slow&ref=nb_sb_noss KahnemanBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=thinking+fast+and+slow+kahneman+2011&btnG= 11]. Mostly, our preferences are inconsistent, manipulable via framing, time-dependent, subject to addictions, and so on. We are likely to regret today's claimed preferences in the future, or as soon as we better understand their consequences. Instead, it is argued, we should program human [[volition|volitions]], which corresponds to what we would prefer to prefer, instead of what we simply prefer.

Unfortunately, a lot more research is needed to better formalize and analyze the concept of volition, and how it diverges from preferences. One fruitful path may be to analyze the difference between what's learned through [[inverse reinforcement learning]], as opposed to through (well-framed) elicitation. See [[volition]] for a lot more discussion on this problem.

Social choice

2020-02-22T22:49:05Z

Lê Nguyên Hoang: /* Harsanyi's Utilitarian Theorem */

Social choice is the study of how to elicit, aggregate and explain human preferences for collective decision-making. This is critical to AI ethics, as we will need to decide collectively on which ethics an AI will follow. For instance, what video should be recommended by the YouTube algorithm when a user queries "Trump", "vaccine" or "social justice"?

== Background ==

Social choice theory arguably started with a remarkable memoire by Condorcet [http://classiques.uqac.ca/classiques/condorcet/Essai_application_discours_preliminaire/discours_preliminaire.pdf Condorcet][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Essai+sur+l%27application+de+l%27analyse+%C3%A0+la+probabilit%C3%A9+des+d%C3%A9cisions+rendues+%C3%A0+la+pluralit%C3%A9+des+voix.+1785&btnG= 1785]. He argued that if one alternative is preferred to any other alternative by a majority then it should be selected. This is the [https://en.wikipedia.org/wiki/Condorcet_criterion Condorcet principle] (see [https://www.youtube.com/watch?v=hI89r4LqaCc MrPhi17aFR], [https://www.youtube.com/watch?v=ZZb4TjvupkI MrPhi17bFR]).

Unfortunately, social choice theory is plagued with impossibility results, like the Condorcet paradox ([https://www.youtube.com/watch?v=HoAnYQZrNrQ PBSInfinite17], [https://www.youtube.com/watch?v=v8-2YdUqQqM MicMaths15FR]), Arrow's impossibility theorem ([https://s3.amazonaws.com/academia.edu.documents/40888103/arrow.pdf?response-content-disposition=inline%3B%20filename%3DArrow.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200118T214628Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=283c657c6dc5c2c225f77e14996a77846974b6dd3a2008f4e299131c6255fd75 Arrow][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Difficulty+in+the+Concept+of+Social+Welfare&btnG= 50], [https://www.youtube.com/watch?v=AhVR7gFMKNg PBSInfinite17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]) and the Gibbard-Satterthwaite theorem ([https://www.youtube.com/watch?v=m5crte26fiw Wandida17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]). Different voting systems yield different winners ([https://www.youtube.com/watch?v=vfTJ4vmIsO4 StatChat16FR], [https://www.youtube.com/watch?v=fBYCoPAmpr4&t=371s S4A17aFR]).

Today's most convincing social choice mechanisms are probably the [https://en.wikipedia.org/wiki/Approval_voting approval voting] ([https://dblp.org/rec/bibtex/books/daglib/0017739 BramsFishburnBook07], [https://www.youtube.com/watch?v=orybDrUj4vA CGPGrey]), [https://en.wikipedia.org/wiki/Majority_judgment majority judgment] ([https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=majority+judgment+laraki+balinski&btnG= BalinksiLarakiBook11], [https://www.youtube.com/watch?v=ZoGH7d51bvc ScienceÉtonnante16FR], [https://www.youtube.com/watch?v=_MAo8pUl0U4 S4A17cFR]) and the randomized Condorcet voting system ([https://dspace.mit.edu/bitstream/handle/1721.1/107673/355_2017_1031_ReferencePDF.pdf?sequence=1&isAllowed=y Hoang][https://dblp.org/rec/bibtex/journals/scw/Hoang17 17], [https://www.youtube.com/watch?v=wKimU8jy2a8 S4A17dFR], [https://www.youtube.com/watch?v=vAdGZkXhlNM S4A17eFR]).

== Harsanyi's Utilitarian Theorem ==

[http://darp.lse.ac.uk/papersDB/Harsanyi_(JPolE_55).pdf Harsanyi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=cardinal+welfare%2C+individualistic+ethics%2C+harsanyi&btnG= 55] proved that weighted sums of individuals' utilities are the only social choice mechanisms that aggregate [[Von Neumann-Morgenstern preferences]] to yield Von Neumann-Morgenstern group preferences in such a way that, if every individual of the group is indifferent between probabilistic outcomes <math>\mu</math> and <math>\nu</math>, then so is the group. This is a compelling argument for a simple addition of individuals' preferences.

To prove it, consider a finite set of alternatives. Denote <math>u_{ij}</math> is the utility of individual <math>i</math> for alternative <math>j</math>. Individual <math>i</math>'s utility for a probability <math>\mu</math> equals <math>u_i^T \mu</math>. Denoting <math>v_j</math> the group's utility for alternative <math>j</math>, the indifference property says that <math>u (\mu-\nu) = 0</math> implies <math>v^T (\mu-\nu) = 0</math>. Using linear algebra and the fact that this holds for all <math>\mu, \nu</math> then implies that <math>v</math> belongs to the vector space spanned by the <math>u_i</math>'s. This results in saying that <math>w = a + \sum w_i u_i</math>, for nonnegative weights <math>w_i</math>'s (and arbitrary constant <math>a</math>).

However, there are major caveats to applying this mechanism in practice. First note that the scaling of different individuals' utility functions (or equivalently of their weights) remains to be settled, which does not seem straightforward to be done. But most importantly, this social choice mechanism is not incentive-compatible. If implemented, individuals will have incentives to exaggerate their preferences (or to tell their representatives to do so). Finally, such expected utility maximization will surely turn into the maximization of some expected proxies, which would then be prone to [[Goodhart's law]].

== The (Huge) Flaw of Classical Social Choice ==

Unfortunately, such approaches are limited because they can only handle a reasonable amount of alternatives. If we are to design AI ethics collectively, we need to choose a code (or, say, guidelines or texts of laws). Yet there are combinatorially many such codes! If we consider 1,000-line codes, this would represent ~210,000 alternatives. Classical voting systems won't do the trick.

Now, there are already lots of results in social choice for structured combinatorial sets of alternatives, mostly derived from auction theory ([https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism VCG mechanism] [https://dblp.org/rec/bibtex/books/cu/NRTV2007 NRTV07] [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR], Myerson's auction [https://dblp.org/rec/bibtex/journals/mor/Myerson81 Myerson81] [https://www.youtube.com/watch?v=FjP5JMUVXxw S4A17gFR], Gale-Shapley [https://dblp.org/rec/bibtex/journals/tamm/GaleS13 GaleShapley62] [https://www.youtube.com/watch?v=Qcv1IqHWAzg Numberphile14] [https://www.youtube.com/watch?v=oHYcOXi06uY S4A17hFR]...). Most impressively, in a series of papers [https://dblp.org/rec/bibtex/journals/sigecom/CaiDW11 CDW11], [https://dblp.org/rec/bibtex/conf/stoc/CaiDW12 CDW12a], [https://dblp.org/rec/bibtex/conf/focs/CaiDW12 CDW12b], [https://dblp.org/rec/bibtex/conf/soda/CaiDW13 CDW13a] and [https://dblp.org/rec/bibtex/conf/focs/CaiDW13 CDW13b], Cai, [https://en.wikipedia.org/wiki/Constantinos_Daskalakis Daskalakis] and Weinberg proved that the polynomial tractability of a Bayesian social choice approximation problem (i.e. with incentive-compatibility constraints) is equivalent to that of the full-information problem with an additional social welfare term to be optimized (see [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR]).

However, we surely have to also tackle the case of unstructured combinatorial sets of alternatives (also, polytime may be too slow in practice).

== Bounds for limited communication complexity ==

A fascinating result by [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19] shows that the worst-case lost of social welfare due to polynomial communication complexity (voters communicate at most log(#alternatives) bits) is unbounded (#alternatives2 for deterministic elicitation+voting, #alternatives for randomized).

There are caveats though. For one thing, this is a worst-case analysis. But human preferences may be more structured. Also, priors can be invoked. Plus, authors assumed that the same elicitation was applied to all voters, which is clearly suboptimal. A lot more research on the communication-complexity versus social-welfare tradeoff is definitely desired. This is exciting!

== Applications to AI Ethics ==

It has been argued to be critical to solve the problem of AI ethics ([https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12457/12204 GRTVW][https://dblp.org/rec/bibtex/conf/aaai/GreeneRTVW16 16],[http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Ethics_Conitzer_etal.pdf CSBDK][https://dblp.org/rec/bibtex/conf/isaim/ConitzerSBD018 17]). In brief, we are unlikely to agree on what ethics to program. However, we might be able to agree on how to agree on some ethics to program even though we disagree. The trick to implement some (virtual) democratic voting on moral preferences.

Interestingly, ideas along these lines have already been developed for the cases of autonomous car dilemmas [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17052/15857 NGADR+][https://dblp.org/rec/bibtex/conf/aaai/NoothigattuGADR18 18] [https://www.youtube.com/watch?v=Y6jfGZXubq0 UpAndAtom18], kidney transplant [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17384/15863 FBSDC][https://dblp.org/rec/bibtex/conf/aaai/FreedmanBSDC18 18] and food donation (called WeBuildAI [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=2ahUKEwj4y5-ggu3mAhXB2qQKHT6GDZ4QFjACegQIARAC&url=https%3A%2F%2Fwww.cs.cmu.edu%2F~akahng%2Fpapers%2Fwebuildai.pdf&usg=AOvVaw2BknquyvgNufy-JlCoPj_G LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19]).

Note that in all such applications, the set of alternatives is combinatorially large. The trick to perform voting with limited elicitation from voters is to collect binary-choice-based preferences, and to then extrapolate preferences for other cases using machine learning (with some inductive bias). Another way to interpret this is to consider that voters get substituted by digital surrogates, whose task is to answer just as the voters would. This is kind of like representative democracy, where voters are replaced by their representatives. But machine learning can allow individual representatives through customized surrogates!

To build trust in the surrogate, WeBuildAI proposes to voters to test their surrogates, and to replace it, if needs be, by some computational model of their owns. They show that such interactions build trust from voters. They also propose some [[interpretability]] framework, where voters are given the implications of the vote of their surrogates.

Now what Lê can't wait for, is for all such frameworks to be applied to problems that really matter, because they influence billions of people. Yes, Lê is (again!) talking about recommender algorithms of social medias like YouTube. How should hate speech be moderated? What should be shown to someone who wants to learn about climate change? Should there be an additional tax on, say, car advertisements? Should angering videos be less viral?

Lê would be thrilled to see social choice theory applied to such critical moral questions.

== Scaled voting ==

One frequent remark that is being made is whether we really can (and should) agree on ethical issues. For instance, [https://www.nature.com/articles/s41586-018-0637-6 ADKSH+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Moral+Machine+experiment+nature+2018&btnG= 18] showed that Japanese prefer to save walkers, while Chinese prefer to save car passengers. Should we really enforce a common ethics worldwide?

Well, we probably don't need to. Cars could be programmed to save Japanese walkers and Chinese car passengers. They could be made to defend freedom in US and baguette in France. While humans usually have preferences for what happens elsewhere in the world, they usually have stronger preferences for what happens near their home. This probably is something that should be considered when designing voting-based ethics.

One proposal to reflect such nuances is [https://en.wikipedia.org/wiki/Quadratic_voting quadratic voting] [https://www.sss.ias.edu/files/pdfs/Rodrik/workshop%2014-15/Weyl-Quadratic_Voting.pdf LalleyWeyl][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=quadratic+voting+lalley+weyl&btnG=&oq=quadratic+voting 18], which can be made secure [https://link.springer.com/article/10.1007/s11127-017-0407-2 ParkRivest][https://dblp.org/rec/bibtex/journals/iacr/ParkR16 16]. In quadratic voting, a voter who wants its vote to weigh n times more must pay n2. This guarantees (asymptotic) efficiency (utilitarian outcome) and incentive-compatibility. But quadratic voting only applies to 2-alternative votes (typically statu quo vs new law) and is manipulable by collusion.

Another interesting point to be made about multidimensional voting is that the (geometric) median is strategy-proof for voters with a peaked preference, and a valuation that decreases with the distance to the peaked preference. The geometric median is particularly suited for, say, determining budget allocation through social choice. Weirdly, we don't know of a neat paper on this, though using sum of distance minimizer is well-known (need citation).

== Preferences versus volitions ==

It's been argued [https://intelligence.org/files/CEV.pdf Yudkowsky][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+yudkowsky&btnG= 04], [https://www.izmemar.com/files/CEV-MachineEthics.pdf Tarleton][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+tarleton&btnG= 10], [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.674.6424&rep=rep1&type=pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=the+value+learning+problem+soares&btnG= 15], [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19] that we surely should not aggregate today's human moral preferences, because of [[cognitive biases]] [https://www.amazon.com/s?k=thinking+fast+and+slow&ref=nb_sb_noss KahnemanBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=thinking+fast+and+slow+kahneman+2011&btnG= 11]. Mostly, our preferences are inconsistent, manipulable via framing, time-dependent, subject to addictions, and so on. We are likely to regret today's claimed preferences in the future, or as soon as we better understand their consequences. Instead, it is argued, we should program human [[volition|volitions]], which corresponds to what we would prefer to prefer, instead of what we simply prefer.

Unfortunately, a lot more research is needed to better formalize and analyze the concept of volition, and how it diverges from preferences. One fruitful path may be to analyze the difference between what's learned through [[inverse reinforcement learning]], as opposed to through (well-framed) elicitation. See [[volition]] for a lot more discussion on this problem.

Social choice

2020-02-22T22:48:47Z

Lê Nguyên Hoang:

Social choice is the study of how to elicit, aggregate and explain human preferences for collective decision-making. This is critical to AI ethics, as we will need to decide collectively on which ethics an AI will follow. For instance, what video should be recommended by the YouTube algorithm when a user queries "Trump", "vaccine" or "social justice"?

== Background ==

Social choice theory arguably started with a remarkable memoire by Condorcet [http://classiques.uqac.ca/classiques/condorcet/Essai_application_discours_preliminaire/discours_preliminaire.pdf Condorcet][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Essai+sur+l%27application+de+l%27analyse+%C3%A0+la+probabilit%C3%A9+des+d%C3%A9cisions+rendues+%C3%A0+la+pluralit%C3%A9+des+voix.+1785&btnG= 1785]. He argued that if one alternative is preferred to any other alternative by a majority then it should be selected. This is the [https://en.wikipedia.org/wiki/Condorcet_criterion Condorcet principle] (see [https://www.youtube.com/watch?v=hI89r4LqaCc MrPhi17aFR], [https://www.youtube.com/watch?v=ZZb4TjvupkI MrPhi17bFR]).

Unfortunately, social choice theory is plagued with impossibility results, like the Condorcet paradox ([https://www.youtube.com/watch?v=HoAnYQZrNrQ PBSInfinite17], [https://www.youtube.com/watch?v=v8-2YdUqQqM MicMaths15FR]), Arrow's impossibility theorem ([https://s3.amazonaws.com/academia.edu.documents/40888103/arrow.pdf?response-content-disposition=inline%3B%20filename%3DArrow.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200118T214628Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=283c657c6dc5c2c225f77e14996a77846974b6dd3a2008f4e299131c6255fd75 Arrow][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Difficulty+in+the+Concept+of+Social+Welfare&btnG= 50], [https://www.youtube.com/watch?v=AhVR7gFMKNg PBSInfinite17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]) and the Gibbard-Satterthwaite theorem ([https://www.youtube.com/watch?v=m5crte26fiw Wandida17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]). Different voting systems yield different winners ([https://www.youtube.com/watch?v=vfTJ4vmIsO4 StatChat16FR], [https://www.youtube.com/watch?v=fBYCoPAmpr4&t=371s S4A17aFR]).

Today's most convincing social choice mechanisms are probably the [https://en.wikipedia.org/wiki/Approval_voting approval voting] ([https://dblp.org/rec/bibtex/books/daglib/0017739 BramsFishburnBook07], [https://www.youtube.com/watch?v=orybDrUj4vA CGPGrey]), [https://en.wikipedia.org/wiki/Majority_judgment majority judgment] ([https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=majority+judgment+laraki+balinski&btnG= BalinksiLarakiBook11], [https://www.youtube.com/watch?v=ZoGH7d51bvc ScienceÉtonnante16FR], [https://www.youtube.com/watch?v=_MAo8pUl0U4 S4A17cFR]) and the randomized Condorcet voting system ([https://dspace.mit.edu/bitstream/handle/1721.1/107673/355_2017_1031_ReferencePDF.pdf?sequence=1&isAllowed=y Hoang][https://dblp.org/rec/bibtex/journals/scw/Hoang17 17], [https://www.youtube.com/watch?v=wKimU8jy2a8 S4A17dFR], [https://www.youtube.com/watch?v=vAdGZkXhlNM S4A17eFR]).

== Harsanyi's Utilitarian Theorem ==

[Harsanyi http://darp.lse.ac.uk/papersDB/Harsanyi_(JPolE_55).pdf][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=cardinal+welfare%2C+individualistic+ethics%2C+harsanyi&btnG= 55] proved that weighted sums of individuals' utilities are the only social choice mechanisms that aggregate [[Von Neumann-Morgenstern preferences]] to yield Von Neumann-Morgenstern group preferences in such a way that, if every individual of the group is indifferent between probabilistic outcomes <math>\mu</math> and <math>\nu</math>, then so is the group. This is a compelling argument for a simple addition of individuals' preferences.

To prove it, consider a finite set of alternatives. Denote <math>u_{ij}</math> is the utility of individual <math>i</math> for alternative <math>j</math>. Individual <math>i</math>'s utility for a probability <math>\mu</math> equals <math>u_i^T \mu</math>. Denoting <math>v_j</math> the group's utility for alternative <math>j</math>, the indifference property says that <math>u (\mu-\nu) = 0</math> implies <math>v^T (\mu-\nu) = 0</math>. Using linear algebra and the fact that this holds for all <math>\mu, \nu</math> then implies that <math>v</math> belongs to the vector space spanned by the <math>u_i</math>'s. This results in saying that <math>w = a + \sum w_i u_i</math>, for nonnegative weights <math>w_i</math>'s (and arbitrary constant <math>a</math>).

However, there are major caveats to applying this mechanism in practice. First note that the scaling of different individuals' utility functions (or equivalently of their weights) remains to be settled, which does not seem straightforward to be done. But most importantly, this social choice mechanism is not incentive-compatible. If implemented, individuals will have incentives to exaggerate their preferences (or to tell their representatives to do so). Finally, such expected utility maximization will surely turn into the maximization of some expected proxies, which would then be prone to [[Goodhart's law]].

== The (Huge) Flaw of Classical Social Choice ==

Unfortunately, such approaches are limited because they can only handle a reasonable amount of alternatives. If we are to design AI ethics collectively, we need to choose a code (or, say, guidelines or texts of laws). Yet there are combinatorially many such codes! If we consider 1,000-line codes, this would represent ~210,000 alternatives. Classical voting systems won't do the trick.

Now, there are already lots of results in social choice for structured combinatorial sets of alternatives, mostly derived from auction theory ([https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism VCG mechanism] [https://dblp.org/rec/bibtex/books/cu/NRTV2007 NRTV07] [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR], Myerson's auction [https://dblp.org/rec/bibtex/journals/mor/Myerson81 Myerson81] [https://www.youtube.com/watch?v=FjP5JMUVXxw S4A17gFR], Gale-Shapley [https://dblp.org/rec/bibtex/journals/tamm/GaleS13 GaleShapley62] [https://www.youtube.com/watch?v=Qcv1IqHWAzg Numberphile14] [https://www.youtube.com/watch?v=oHYcOXi06uY S4A17hFR]...). Most impressively, in a series of papers [https://dblp.org/rec/bibtex/journals/sigecom/CaiDW11 CDW11], [https://dblp.org/rec/bibtex/conf/stoc/CaiDW12 CDW12a], [https://dblp.org/rec/bibtex/conf/focs/CaiDW12 CDW12b], [https://dblp.org/rec/bibtex/conf/soda/CaiDW13 CDW13a] and [https://dblp.org/rec/bibtex/conf/focs/CaiDW13 CDW13b], Cai, [https://en.wikipedia.org/wiki/Constantinos_Daskalakis Daskalakis] and Weinberg proved that the polynomial tractability of a Bayesian social choice approximation problem (i.e. with incentive-compatibility constraints) is equivalent to that of the full-information problem with an additional social welfare term to be optimized (see [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR]).

However, we surely have to also tackle the case of unstructured combinatorial sets of alternatives (also, polytime may be too slow in practice).

== Bounds for limited communication complexity ==

A fascinating result by [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19] shows that the worst-case lost of social welfare due to polynomial communication complexity (voters communicate at most log(#alternatives) bits) is unbounded (#alternatives2 for deterministic elicitation+voting, #alternatives for randomized).

There are caveats though. For one thing, this is a worst-case analysis. But human preferences may be more structured. Also, priors can be invoked. Plus, authors assumed that the same elicitation was applied to all voters, which is clearly suboptimal. A lot more research on the communication-complexity versus social-welfare tradeoff is definitely desired. This is exciting!

== Applications to AI Ethics ==

It has been argued to be critical to solve the problem of AI ethics ([https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12457/12204 GRTVW][https://dblp.org/rec/bibtex/conf/aaai/GreeneRTVW16 16],[http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Ethics_Conitzer_etal.pdf CSBDK][https://dblp.org/rec/bibtex/conf/isaim/ConitzerSBD018 17]). In brief, we are unlikely to agree on what ethics to program. However, we might be able to agree on how to agree on some ethics to program even though we disagree. The trick to implement some (virtual) democratic voting on moral preferences.

Interestingly, ideas along these lines have already been developed for the cases of autonomous car dilemmas [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17052/15857 NGADR+][https://dblp.org/rec/bibtex/conf/aaai/NoothigattuGADR18 18] [https://www.youtube.com/watch?v=Y6jfGZXubq0 UpAndAtom18], kidney transplant [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17384/15863 FBSDC][https://dblp.org/rec/bibtex/conf/aaai/FreedmanBSDC18 18] and food donation (called WeBuildAI [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=2ahUKEwj4y5-ggu3mAhXB2qQKHT6GDZ4QFjACegQIARAC&url=https%3A%2F%2Fwww.cs.cmu.edu%2F~akahng%2Fpapers%2Fwebuildai.pdf&usg=AOvVaw2BknquyvgNufy-JlCoPj_G LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19]).

Note that in all such applications, the set of alternatives is combinatorially large. The trick to perform voting with limited elicitation from voters is to collect binary-choice-based preferences, and to then extrapolate preferences for other cases using machine learning (with some inductive bias). Another way to interpret this is to consider that voters get substituted by digital surrogates, whose task is to answer just as the voters would. This is kind of like representative democracy, where voters are replaced by their representatives. But machine learning can allow individual representatives through customized surrogates!

To build trust in the surrogate, WeBuildAI proposes to voters to test their surrogates, and to replace it, if needs be, by some computational model of their owns. They show that such interactions build trust from voters. They also propose some [[interpretability]] framework, where voters are given the implications of the vote of their surrogates.

Now what Lê can't wait for, is for all such frameworks to be applied to problems that really matter, because they influence billions of people. Yes, Lê is (again!) talking about recommender algorithms of social medias like YouTube. How should hate speech be moderated? What should be shown to someone who wants to learn about climate change? Should there be an additional tax on, say, car advertisements? Should angering videos be less viral?

Lê would be thrilled to see social choice theory applied to such critical moral questions.

== Scaled voting ==

One frequent remark that is being made is whether we really can (and should) agree on ethical issues. For instance, [https://www.nature.com/articles/s41586-018-0637-6 ADKSH+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Moral+Machine+experiment+nature+2018&btnG= 18] showed that Japanese prefer to save walkers, while Chinese prefer to save car passengers. Should we really enforce a common ethics worldwide?

Well, we probably don't need to. Cars could be programmed to save Japanese walkers and Chinese car passengers. They could be made to defend freedom in US and baguette in France. While humans usually have preferences for what happens elsewhere in the world, they usually have stronger preferences for what happens near their home. This probably is something that should be considered when designing voting-based ethics.

One proposal to reflect such nuances is [https://en.wikipedia.org/wiki/Quadratic_voting quadratic voting] [https://www.sss.ias.edu/files/pdfs/Rodrik/workshop%2014-15/Weyl-Quadratic_Voting.pdf LalleyWeyl][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=quadratic+voting+lalley+weyl&btnG=&oq=quadratic+voting 18], which can be made secure [https://link.springer.com/article/10.1007/s11127-017-0407-2 ParkRivest][https://dblp.org/rec/bibtex/journals/iacr/ParkR16 16]. In quadratic voting, a voter who wants its vote to weigh n times more must pay n2. This guarantees (asymptotic) efficiency (utilitarian outcome) and incentive-compatibility. But quadratic voting only applies to 2-alternative votes (typically statu quo vs new law) and is manipulable by collusion.

Another interesting point to be made about multidimensional voting is that the (geometric) median is strategy-proof for voters with a peaked preference, and a valuation that decreases with the distance to the peaked preference. The geometric median is particularly suited for, say, determining budget allocation through social choice. Weirdly, we don't know of a neat paper on this, though using sum of distance minimizer is well-known (need citation).

== Preferences versus volitions ==

It's been argued [https://intelligence.org/files/CEV.pdf Yudkowsky][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+yudkowsky&btnG= 04], [https://www.izmemar.com/files/CEV-MachineEthics.pdf Tarleton][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+tarleton&btnG= 10], [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.674.6424&rep=rep1&type=pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=the+value+learning+problem+soares&btnG= 15], [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19] that we surely should not aggregate today's human moral preferences, because of [[cognitive biases]] [https://www.amazon.com/s?k=thinking+fast+and+slow&ref=nb_sb_noss KahnemanBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=thinking+fast+and+slow+kahneman+2011&btnG= 11]. Mostly, our preferences are inconsistent, manipulable via framing, time-dependent, subject to addictions, and so on. We are likely to regret today's claimed preferences in the future, or as soon as we better understand their consequences. Instead, it is argued, we should program human [[volition|volitions]], which corresponds to what we would prefer to prefer, instead of what we simply prefer.

Unfortunately, a lot more research is needed to better formalize and analyze the concept of volition, and how it diverges from preferences. One fruitful path may be to analyze the difference between what's learned through [[inverse reinforcement learning]], as opposed to through (well-framed) elicitation. See [[volition]] for a lot more discussion on this problem.

Von Neumann-Morgenstern preferences

2020-02-22T22:36:52Z

Lê Nguyên Hoang: /* Formal theorem */

A Von Neumann-Morgenstern preference [https://pdfs.semanticscholar.org/0375/379194a6f34b818962ea947bff153adf621c.pdf VonneumannMorgensternBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Theory+of+games+and+economic+behaviors+von+neumann+morgenstern&btnG= 44] is a consistent order relation over probabilistic outcomes. The Von Neumann-Morgenstern theorem states that any Von Neumann-Morgenstern preference is equivalently described by expected scores.

== Formal theorem ==

Consider a set <math>X</math> of outcomes. We denote <math>\Delta(X)</math> the set of probability distributions over <math>X</math>. For countable sets <math>X</math>, a probabilistic outcome <math>\mu \in \Delta(X)</math> is thus a function that assigns a probability <math>\mu(x)</math> to any outcome <math>x \in X</math>, and that satisfies <math>\sum_{x \in X} \mu(x) = 1</math>.

A preference is a total order relation over <math>\Delta(X)</math>. To be a von Neumann-Morgenstern preference, it needs to satisfy the following very natural axioms:
* Completeness: <math>\mu \succ \nu</math>, <math>\nu \prec \mu</math> or <math>\mu \sim \nu</math>.
* Transitivity: if <math>\mu \succeq \nu</math> and <math>\nu \succeq \rho</math>, then <math>\mu \succeq \rho</math>.
* Continuity: if <math>\mu \succeq \nu \succeq \rho</math>, then there exists <math>p \in [0,1]</math> such that <math>p \mu + (1-p) \rho \sim \nu</math>.
* Independence of irrelevant alternatives (IIA): <math>\mu \succeq \nu</math> if and only if <math>p \mu + (1-p) \rho \succeq p \nu + (1-p) \rho</math>.

The von Neumann-Morgenstern theorem states that any von Neumann-Morgenstern preference is equivalently descrbied by an expected score (called utility). In other words, there exists a utility function <math>u : X \rightarrow \mathbb R</math> such that <math>\mu \succeq \nu</math> if and only if <math>\mathbb E_{x \leftarrow \mu} [u(x)] \geq \mathbb E_{x \leftarrow \nu} [u(x)]</math>.

It was also shown that this utility function is nearly unique. It is defined up to a positive affine transformation, i.e., <math>v</math> is a utility function for the von Neumann-Morgenster preference if and only if there exists <math>a \in \mathbb R</math> and <math>b > 0</math> such that <math>v = a+bu</math>.

Von Neumann-Morgenstern preferences

2020-02-22T22:36:01Z

Lê Nguyên Hoang: Created page with "A Von Neumann-Morgenstern preference [https://pdfs.semanticscholar.org/0375/379194a6f34b818962ea947bff153adf621c.pdf VonneumannMorgensternBook][https://scholar.google.ch/schol..."

A Von Neumann-Morgenstern preference [https://pdfs.semanticscholar.org/0375/379194a6f34b818962ea947bff153adf621c.pdf VonneumannMorgensternBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Theory+of+games+and+economic+behaviors+von+neumann+morgenstern&btnG= 44] is a consistent order relation over probabilistic outcomes. The Von Neumann-Morgenstern theorem states that any Von Neumann-Morgenstern preference is equivalently described by expected scores.

== Formal theorem ==

Consider a set <math>X</math> of outcomes. We denote <math>\Delta(X)</math> the set of probability distributions over <math>X</math>. For countable sets <math>X</math>, a probabilistic outcome <math>\mu \in \Delta(X)</math> is thus a function that assigns a probability <math>\mu(x)</math> to any outcome <math>x \in X</math>, and that satisfies <math>\sum_{x \in X} \mu(x) = 1</math>.

A preference is a total order relation over <math>\Delta(X)</math>. To be a von Neumann-Morgenstern preference, it needs to satisfy the following very natural axioms:
* Completeness: <math>\mu \succ \nu</math>, <math>\nu \prec \mu</math> or <math>\mu \sim \nu</math>.
* Transitivity: if <math>\mu \succeq \nu</math> and <math>\nu \succeq \rho</math>, then <math>\mu \succeq \rho</math>.
* Continuity: if <math>\mu \succeq \nu \succeq \rho</math>, then there exists <math>p \in [0,1]</math> such that <math>p \mu + (1-p) \rho \sim \nu</math>.
* Independence: <math>\mu \succeq \nu</math> if and only if <math>p \mu + (1-p) \rho \succeq p \nu + (1-p) \rho</math>.

The von Neumann-Morgenstern theorem states that any von Neumann-Morgenstern preference is equivalently descrbied by an expected score (called utility). In other words, there exists a utility function <math>u : X \rightarrow \mathbb R</math> such that <math>\mu \succeq \nu</math> if and only if <math>\mathbb E_{x \leftarrow \mu} [u(x)] \geq \mathbb E_{x \leftarrow \nu} [u(x)]</math>.

It was also shown that this utility function is nearly unique. It is defined up to a positive affine transformation, i.e., <math>v</math> is a utility function for the von Neumann-Morgenster preference if and only if there exists <math>a \in \mathbb R</math> and <math>b > 0</math> such that <math>v = a+bu</math>.

Preference learning from comparisons

2020-02-22T22:34:04Z

Lê Nguyên Hoang:

It has been argued that we humans are much more effective at comparing alternatives than at scoring them [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18] [https://www.youtube.com/watch?v=bmD-myeu19Q RB5]. Besides, implicit observations of humans mostly provide such choice data, in the form of like/no-like, share/no-share, click/no-click, and so on. Therefore, preference learning from comparisons seems to be an important approach to preference learning.

== Classical models ==

The classical models for preference learning are due to [http://mlab.no/blog/wp-content/uploads/2009/07/thurstone94law.pdf Thurston][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+law+of+comparative+judgment+thurstone&btnG= 27], [https://idp.springer.com/authorize/casa?redirect_uri=https://link.springer.com/content/pdf/10.1007/BF01180541.pdf&casa_token=dMSQZQpkjt4AAAAA:4HFh3rzi3S6VqH29Pg00S64qMuFB9b6VpF8pRikHmTNWf3REi10i_dj9_Eps3Ivp6l8c-ZETTU-Cx_-L6g Zermelo][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Die+Berechnung+der+Turnier-Ergebnisse+als+ein+Maximumproblem+der+Wahrscheinlichkeitsrechnung.&btnG= 29], [https://link.springer.com/content/pdf/10.1007/BF02289115.pdf Mosteller][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=mosteller+Remarks+on+the+method+of+paired+comparisons&btnG= 51], [https://www.jstor.org/stable/pdf/2334029.pdf?casa_token=Q0ROYjxwOZEAAAAA:fuuqbCntxaNmZB0hlKk8iXfzOkLzQs9H677gh2UOQdMI4496B3YZHlUaiuMgD9zONaHYLHgtSmjMHziL29cx9d9tv2f_QwyqN0wxYUJrExI_eK8QR7sa BradleyTerry][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Rank+analysis+of+incomplete+block+designs%3A+I.+The+method+of+paired+comparisons&btnG= 52], [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Individual+Choice+Behavior%3A+A+Theoretical+Analysis+luce+wiley+1959&btnG= LuceBook59], [https://apps.dtic.mil/dtic/tr/fulltext/u2/a417190.pdf#page=15 DavidBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Method+of+Paired+Comparisons+david&btnG= 63].

In these models, in a choice between 1 and 2, the human implicitly the scores <math>\theta_1</math> and <math>\theta_2</math> that she assigns to each alternative. But his computation of <math>\theta_1 - \theta_2</math> is noisy, and is accompanied with some noise <math>\varepsilon</math>, to yield <math>x_{12} = \theta_1-\theta_2+\varepsilon</math>. In the above models, the sign of <math>x_{12}</math> then determines the choice of the human.

This approach allows to explain some of the inconsistencies in humans' decision-making. Equivalently, this corresponds to saying that the probability that the human chooses 1 over 2 is a function of the intrinsic difference <math>\theta_1-\theta_2</math>, which we can write <math>\mathbb P[1 \succ 2] = \mathbb P[x_{12}>0] = \phi(\theta_1-\theta_2)</math>. Intuitively, the greater the intrinsic difference between 1 and 2, the less likely it is for the human to say he prefers 2 to 1.

Different models assume different noise models for <math>\varepsilon</math>, or, equivalently, for the function <math>\phi</math>. Thurstone's model assumes that <math>\varepsilon</math> is normally distributed, which is equivalent to saying that <math>\phi(z) = \Phi(z/\sigma)</math>, where <math>\sigma^2</math> is the variance of <math>\varepsilon</math> (which may depend on the choice of 1 and 2), and where <math>\Phi</math> is the cumulative density function of the standard normal distribution.

The Bradley-Terry model assumes that <math>\varepsilon</math> follows a [https://en.wikipedia.org/wiki/Gumbel_distribution Gumbel distribution]. Equivalently, it sets <math>\phi(z)= \frac{1}{1+\exp(-z)}</math> (up to variance scaling). Luce's model generalizes the Bradley-Terry models, by considering choices among numerous alternatives and setting <math>\phi(z_2, z_3, ...)= \frac{1}{1+\exp(-z_2)+\exp(-z_3)+...}</math>, where this quantity is the probability of choosing option 1, and where <math>z_i = \theta_1-\theta_i</math>. Interestingly, Luce proved that this framework was equivalent to demanding independence of irrelevant alternatives.

== Inference ==

The inference problem is then the problem of inferring the values of parameters <math>\theta_i</math> given observational data <math>\mathcal D</math> of choices <math>i</math> out of a set of alternatives <math>\mathcal A</math>. Bayesian inference would suggest computing <math>\mathbb P[\theta|\mathcal D]</math>. But as often, this approach might be too computationally costly in practice. Approximate Bayesian methods are needed.

Note that in the Luce model, the log-likelihood <math>\log \mathbb P[\mathcal D|\theta] = \sum \left( \theta_i - \log \sum_{(j \succ i) \in \mathcal D} \exp \theta_j \right)</math> is a strictly concave function in <math>\theta</math> if the comparison graph is strongly connected (for any <math>i,j</math>, there exists <math>k_1, ... k_m</math> such that there are some of the data that says <math>i \succ k_1 \succ k_2 \succ ... \succ k_m \succ j</math>). This proves the existence and uniqueness of the maximum likelihood estimator under this assumption.

== Markov chain for preference learning ==

[http://papers.nips.cc/paper/4701-iterative-ranking-from-pair-wise-comparisons.pdf NOS][https://dblp.org/rec/bibtex/conf/nips/NegahbanOS12 12] proposed a Markov chain approach to compute the maximum likelihood estimator, in the same vein as the PageRank algorithm [http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf PBMW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+PageRank+citation+ranking%3A+Bringing+order+to+the+Web+page+brin+motwani+winograd&btnG= 99]. Namely, by choosing adequately the transition rates of the comparison graph, the stationary distribution of the Markov chain thereby constructed turns out to equal the vector <math>(\exp \theta_i^\star)</math>, where <math>\theta^\star</math> is the maximum likelihood estimator.

Note however, that this approach is too data and computationally expensive if the set of alternatives is combinatorial. In fact, for most recommender systems, it may be inapplicable, since most users have never even been exposed to most of the video or music contents of platforms like YouTube or Spotify.

== Gaussian process ==

Perhaps one of the most promising avenues to scalable comparison-based preference learning consists of building upon the assumption that, a priori, the intrinsic preference <math>\theta_1-\theta_2</math> between two alternatives <math>z_1</math> and <math>z_2</math> is a Gaussian process. By then invoking the similarity between having to choose between <math>x = (z_1,z_2)</math> and <math>x' = (z_1',z_2')</math>, we can then generalize from observed data by means of a [[kernel method]].

To implement this approach, we first need a measure of the similarity between two choices <math>x</math> and <math>x'</math>. Intuitively, <math>x</math> and <math>x'</math> will be similar if both <math>z_1 \approx z_1'</math> and <math>z_2 \approx z_2'</math>. Note that <math>x</math> and <math>x'</math> can also be "anti-similar", if <math>z_1 \approx z_2'</math> and <math>z_2 \approx z_1'</math>. Let us call <math>K(x,x')</math> the similarity between <math>x</math> and </math>x'</math>. Gaussian process prior and Bayesian methods then enable to infer a posterior from revealed preferences on the preference for other dilemmas [https://dl.acm.org/doi/pdf/10.1145/1102351.1102369?download=true ChuGhahramani][https://dblp.org/rec/bibtex/conf/icml/ChuG05 05a] [http://www.jmlr.org/papers/volume6/chu05a/chu05a.pdf ChuGharamani][https://dblp.org/rec/bibtex/journals/jmlr/ChuG05 05b].

Note that such kernels can be approximately implemented by [[representational learning|vector representation]], for instance by some deep neural network <math>f</math>. We could then have <math>K(x,x') = f(x)^Tf(x')</math> (anti-similarity could be naturally enforced if <math>f(z_1,z_2) = - f(z_2,z_1)</math>, which is guaranteed if <math>f(z_1,z_2) = g(z_1)-g(z_2)</math> for some other vector representation <math>g</math>). The use of such vector representations could be useful to accelerate computation, as opposed to the kernel method which requires going through all the training data to make a prediction.

== Connection to sports ==

Chess [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Rating+Of+ChessPlayers+past+and+present+Players+elo+arco&btnG= Elo78], football [http://ceur-ws.org/Vol-1842/paper_04.pdf MKFG][https://dblp.org/rec/bibtex/conf/pkdd/MaystreKFG16 16].

Welcome to the Robustly Beneficial Wiki

2020-02-22T22:33:14Z

Lê Nguyên Hoang: /* Why AI safety and ethics is harder than meets the eye */

Welcome to the [[Robustly beneficial|Robustly Beneficial]] wiki!!

This wiki aims to better grasp the scope and the limits of current AI ethics research. It lists references, key ideas and relevant open questions to make algorithms robustly beneficial. Please check also our [https://www.youtube.com/watch?v=WWbw4cla2jw&list=PLgqL_7nXb23FKk_rUfs7vnvyrPshYPfA8 Robustly Beneficial Podcast] ([https://podcasts.apple.com/fr/podcast/robustly-beneficial-podcast/id1496159681 iTunes], [https://playlists.podmytube.com/UCgl_MmjatQif8juz3Lt6iPw/PLgqL_7nXb23FKk_rUfs7vnvyrPshYPfA8.xml RSS]), our [https://www.youtube.com/playlist?list=PLgqL_7nXb23HvhToBb9FwFxj83navY6oq&playnext=1&index=1 Robustly Beneficial Talks] and our [https://twitter.com/robustlyb Twitter account].

The wiki has just been launched, so most pages are still being written. But they will never be finished — this is the whole point of a wiki!

== The structure of the wiki ==

The wiki can be roughly divided into 4 main categories.

=== Why AI ethics is becoming critical ===

If you are new to AI ethics, you should probably start with the [[AI risks]] page. You could then go into arguably today's most important case of AI ethics, namely [[YouTube]]. Note that algorithms also offer formidable [[AI opportunities]] that are definitely worth considering. Find out more by reading about [[online polarization]], [[misinformation]], [[addiction]], [[mental health]] or [[hate]]. And as an example of an urgent AI ethics dilemma, check [https://twitter.com/le_science4all/status/1227690739104174080 this Twitter thread] on responses to a "is climate change a hoax?" search.

And if you know little about the current state of algorithmic research, you might want to check the latest [[impressive advances in AI]]. Or you could check some [[funny applications of AI]]. You can also read Lê's [https://www.lesswrong.com/posts/bwqDrSZvhEDKxRf6z/a-rant-against-robots rant against robots].

=== How today's (and probably tomorrow's) AIs work ===

The most important principle of today's AI is surely [[machine learning]]. Today, it mostly relies on [[stochastic gradient descent]] for (deep) [[neural networks]], which allow [[representational learning]] (see [[convolutional neural network]], [[residual network]], [[transformer]]). See also [[Turing 1950]], [[convexity]], [[generative adversarial network]], [[specialized hardware]] and [[linear systems]].

[[Bayesianism]] has been argued to be the ideal form of supervised and unsupervised learning, if we had infinite computational power (see [[Solomonoff's demon]], [[Laplace 1814]]). It has numerous desirable properties, like [[statistical admissibility]], [[Bayesian agreement]] or the [[Bayesian brain]] hypothesis. See also [[Bayesian examination]] and [[conjugate priors]].

A branch of learning called [[reinforcement learning]], which relies on [[Q-learning]] or [[policy learning]], seems likely to become the core framework of today's and tomorrow's AIs. [[AIXI]] achieves the upper-bound for [[Legg-Hutter intelligence]], which aims to measure [[artificial general intelligence|general intelligence]].

To understand the gap between Bayesianism/AIXI and practical machine learning, we need to understand the constraints of computational [[complexity]] theory. By building upon the [[Church-Turing thesis]], the [[Kolmogorov-Solomonoff complexity]] and knowledge from [[human brain computations]], this allows some insights into [[human-level AI]], in addition to [[experts' AI predictions]]. See also [[entropy]] and [[sophistication]].

AIs are already doing [[distributed learning]], which raises numerous challenges, like [[Byzantine fault tolerance]] and [[model drift]].

=== Why AI safety and ethics is harder than meets the eye ===

We want to get algorithms to do what we would really want them to do. But this turns out to raise numerous highly nontrivial problems, like [[Goodhart's law]], [[overfitting]], [[robust statistics]], [[confounding variables]], [[adversarial attacks]], [[algorithmic bias]], [[cognitive bias]], [[backfire effect]], [[distributional shift]], [[privacy]], [[human liabilities]], [[interpretability]], [[reward hacking]], [[wireheading]] and [[instrumental convergence]]. Because of all such problems, it seems crucial that algorithms be able to reason about their ignorance, using [[Bayesianism|Bayesian]] principles, [[moral uncertainty]] and [[second opinion querying]]. Algorithms must be [[robustly beneficial]].

AI ethics also demands that we solve thorny philosophical dilemmas, like the [[repugnant conclusion]], [[Newcomb's paradox]] and [[moral realism]]. Unfortunately, we have numerous [[cognitive bias|cognitive biases]], which seem critical to understand to solve AI ethics. Results about [[counterfactual]], [[von Neumann-Morgenstern preferences]] and [[Dutch book]] also seem useful to consider.

=== How to solve AI ethics (hopefully) ===

To solve AI ethics, [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19a] proposed the [[ABCDE roadmap]], which decomposes the [[alignment]] problem into numerous (hopefully) orthogonal and complementary subproblems. Such subproblems include [[data certification]], perhaps through [[Blockchain]], [[world model inference]] through [[Bayesianism]] and/or [[representational learning]], [[volition]] learning perhaps from [[Preference learning from comparisons|comparisons]] and [[social choice]] solutions, [[corrigibility]] and safe [[reinforcement learning]].

The fabulous endeavor to make AIs robustly beneficial can seem overwhelming, given how extraordinarily interdisciplinary it is. While it is worthwhile to have an overview of the problem, we believe it is also useful for aspiring contributors to identify more precise problems they can contribute to. In this wiki, we propose targeted research directions for different expertises and research interests. Please check the following pages that may be of interest to you.
<ul>
<li>[[how newcomers can contribute]]</li>
<li>[[how mathematicians can contribute]]</li>
<li>[[how computer scientists can contribute]]</li>
<li>[[how software engineers can contribute]]</li>
<li>[[how neuroscientists can contribute]]</li>
<li>[[how philosophers can contribute]]</li>
<li>[[how social scientists can contribute]]</li>
<li>[[how economists can contribute]]</li>
<li>[[how policy-makers can contribute]]</li>
<li>[[how journalists can contribute]]</li>
<li>[[how medical doctors can contribute]]</li>
<li>[[how physicists can contribute]]</li>
<li>[[how educators can contribute]]</li>
<li>[[how science communicators can contribute]]</li>
<li>[[how community builders can contribute]]</li>
</ul>

== About the authors ==

This wiki is written and edited mostly by members of the [[Robustly Beneficial group]], which regularly meets at EPFL, in Lausanne, Switzerland. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join]. So far, the main authors are [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Louis_Faucon|Louis Faucon]].

Lê and Mahdi recently co-wrote the book The Fabulous Endeavor: Make Artificial Intelligence Robustly Beneficial [https://laboutique.edpsciences.fr/produit/1107/9782759824304/Le%20fabuleux%20chantier HoangElmhamdi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Le+fabuleux+chantier%3A+Rendre+l%27intelligence+artificielle+robustement+b%C3%A9n%C3%A9fique&btnG= 19FR] (the English version is pending).

Social choice

2020-02-22T22:22:14Z

Lê Nguyên Hoang:

Social choice is the study of how to elicit, aggregate and explain human preferences for collective decision-making. This is critical to AI ethics, as we will need to decide collectively on which ethics an AI will follow. For instance, what video should be recommended by the YouTube algorithm when a user queries "Trump", "vaccine" or "social justice"?

== Background ==

Social choice theory arguably started with a remarkable memoire by Condorcet [http://classiques.uqac.ca/classiques/condorcet/Essai_application_discours_preliminaire/discours_preliminaire.pdf Condorcet][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Essai+sur+l%27application+de+l%27analyse+%C3%A0+la+probabilit%C3%A9+des+d%C3%A9cisions+rendues+%C3%A0+la+pluralit%C3%A9+des+voix.+1785&btnG= 1785]. He argued that if one alternative is preferred to any other alternative by a majority then it should be selected. This is the [https://en.wikipedia.org/wiki/Condorcet_criterion Condorcet principle] (see [https://www.youtube.com/watch?v=hI89r4LqaCc MrPhi17aFR], [https://www.youtube.com/watch?v=ZZb4TjvupkI MrPhi17bFR]).

Unfortunately, social choice theory is plagued with impossibility results, like the Condorcet paradox ([https://www.youtube.com/watch?v=HoAnYQZrNrQ PBSInfinite17], [https://www.youtube.com/watch?v=v8-2YdUqQqM MicMaths15FR]), Arrow's impossibility theorem ([https://s3.amazonaws.com/academia.edu.documents/40888103/arrow.pdf?response-content-disposition=inline%3B%20filename%3DArrow.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200118T214628Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=283c657c6dc5c2c225f77e14996a77846974b6dd3a2008f4e299131c6255fd75 Arrow][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Difficulty+in+the+Concept+of+Social+Welfare&btnG= 50], [https://www.youtube.com/watch?v=AhVR7gFMKNg PBSInfinite17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]) and the Gibbard-Satterthwaite theorem ([https://www.youtube.com/watch?v=m5crte26fiw Wandida17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]). Different voting systems yield different winners ([https://www.youtube.com/watch?v=vfTJ4vmIsO4 StatChat16FR], [https://www.youtube.com/watch?v=fBYCoPAmpr4&t=371s S4A17aFR]).

Today's most convincing social choice mechanisms are probably the [https://en.wikipedia.org/wiki/Approval_voting approval voting] ([https://dblp.org/rec/bibtex/books/daglib/0017739 BramsFishburnBook07], [https://www.youtube.com/watch?v=orybDrUj4vA CGPGrey]), [https://en.wikipedia.org/wiki/Majority_judgment majority judgment] ([https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=majority+judgment+laraki+balinski&btnG= BalinksiLarakiBook11], [https://www.youtube.com/watch?v=ZoGH7d51bvc ScienceÉtonnante16FR], [https://www.youtube.com/watch?v=_MAo8pUl0U4 S4A17cFR]) and the randomized Condorcet voting system ([https://dspace.mit.edu/bitstream/handle/1721.1/107673/355_2017_1031_ReferencePDF.pdf?sequence=1&isAllowed=y Hoang][https://dblp.org/rec/bibtex/journals/scw/Hoang17 17], [https://www.youtube.com/watch?v=wKimU8jy2a8 S4A17dFR], [https://www.youtube.com/watch?v=vAdGZkXhlNM S4A17eFR]).

== Harsanyi's Utilitarian Theorem ==

[Harsanyi http://darp.lse.ac.uk/papersDB/Harsanyi_(JPolE_55).pdf][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=cardinal+welfare%2C+individualistic+ethics%2C+harsanyi&btnG= 55] proved that weighted sums of individuals' utilities are the only social choice mechanisms that aggregate [[Von Neumann-Morgenstern preferences]] to yield Von Neumann-Morgenstern group preferences in such a way that, if every individual of the group is indifferent between probabilistic outcomes <math>\mu</math> and <math>\nu</math>, then so is the group. This is a compelling argument for a simple addition of individuals' preferences.

However, there are major caveats to applying this mechanism in practice. First note that the scaling of different individuals' utility functions (or equivalently of their weights) remains to be settled, which does not seem straightforward to be done. But most importantly, this social choice mechanism is not incentive-compatible. If implemented, individuals will have incentives to exaggerate their preferences (or to tell their representatives to do so). Finally, such expected utility maximization will surely turn into the maximization of some expected proxies, which would then be prone to [[Goodhart's law]].

== The (Huge) Flaw of Classical Social Choice ==

Unfortunately, such approaches are limited because they can only handle a reasonable amount of alternatives. If we are to design AI ethics collectively, we need to choose a code (or, say, guidelines or texts of laws). Yet there are combinatorially many such codes! If we consider 1,000-line codes, this would represent ~210,000 alternatives. Classical voting systems won't do the trick.

Now, there are already lots of results in social choice for structured combinatorial sets of alternatives, mostly derived from auction theory ([https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism VCG mechanism] [https://dblp.org/rec/bibtex/books/cu/NRTV2007 NRTV07] [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR], Myerson's auction [https://dblp.org/rec/bibtex/journals/mor/Myerson81 Myerson81] [https://www.youtube.com/watch?v=FjP5JMUVXxw S4A17gFR], Gale-Shapley [https://dblp.org/rec/bibtex/journals/tamm/GaleS13 GaleShapley62] [https://www.youtube.com/watch?v=Qcv1IqHWAzg Numberphile14] [https://www.youtube.com/watch?v=oHYcOXi06uY S4A17hFR]...). Most impressively, in a series of papers [https://dblp.org/rec/bibtex/journals/sigecom/CaiDW11 CDW11], [https://dblp.org/rec/bibtex/conf/stoc/CaiDW12 CDW12a], [https://dblp.org/rec/bibtex/conf/focs/CaiDW12 CDW12b], [https://dblp.org/rec/bibtex/conf/soda/CaiDW13 CDW13a] and [https://dblp.org/rec/bibtex/conf/focs/CaiDW13 CDW13b], Cai, [https://en.wikipedia.org/wiki/Constantinos_Daskalakis Daskalakis] and Weinberg proved that the polynomial tractability of a Bayesian social choice approximation problem (i.e. with incentive-compatibility constraints) is equivalent to that of the full-information problem with an additional social welfare term to be optimized (see [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR]).

However, we surely have to also tackle the case of unstructured combinatorial sets of alternatives (also, polytime may be too slow in practice).

== Bounds for limited communication complexity ==

A fascinating result by [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19] shows that the worst-case lost of social welfare due to polynomial communication complexity (voters communicate at most log(#alternatives) bits) is unbounded (#alternatives2 for deterministic elicitation+voting, #alternatives for randomized).

There are caveats though. For one thing, this is a worst-case analysis. But human preferences may be more structured. Also, priors can be invoked. Plus, authors assumed that the same elicitation was applied to all voters, which is clearly suboptimal. A lot more research on the communication-complexity versus social-welfare tradeoff is definitely desired. This is exciting!

== Applications to AI Ethics ==

It has been argued to be critical to solve the problem of AI ethics ([https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12457/12204 GRTVW][https://dblp.org/rec/bibtex/conf/aaai/GreeneRTVW16 16],[http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Ethics_Conitzer_etal.pdf CSBDK][https://dblp.org/rec/bibtex/conf/isaim/ConitzerSBD018 17]). In brief, we are unlikely to agree on what ethics to program. However, we might be able to agree on how to agree on some ethics to program even though we disagree. The trick to implement some (virtual) democratic voting on moral preferences.

Interestingly, ideas along these lines have already been developed for the cases of autonomous car dilemmas [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17052/15857 NGADR+][https://dblp.org/rec/bibtex/conf/aaai/NoothigattuGADR18 18] [https://www.youtube.com/watch?v=Y6jfGZXubq0 UpAndAtom18], kidney transplant [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17384/15863 FBSDC][https://dblp.org/rec/bibtex/conf/aaai/FreedmanBSDC18 18] and food donation (called WeBuildAI [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=2ahUKEwj4y5-ggu3mAhXB2qQKHT6GDZ4QFjACegQIARAC&url=https%3A%2F%2Fwww.cs.cmu.edu%2F~akahng%2Fpapers%2Fwebuildai.pdf&usg=AOvVaw2BknquyvgNufy-JlCoPj_G LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19]).

Note that in all such applications, the set of alternatives is combinatorially large. The trick to perform voting with limited elicitation from voters is to collect binary-choice-based preferences, and to then extrapolate preferences for other cases using machine learning (with some inductive bias). Another way to interpret this is to consider that voters get substituted by digital surrogates, whose task is to answer just as the voters would. This is kind of like representative democracy, where voters are replaced by their representatives. But machine learning can allow individual representatives through customized surrogates!

To build trust in the surrogate, WeBuildAI proposes to voters to test their surrogates, and to replace it, if needs be, by some computational model of their owns. They show that such interactions build trust from voters. They also propose some [[interpretability]] framework, where voters are given the implications of the vote of their surrogates.

Now what Lê can't wait for, is for all such frameworks to be applied to problems that really matter, because they influence billions of people. Yes, Lê is (again!) talking about recommender algorithms of social medias like YouTube. How should hate speech be moderated? What should be shown to someone who wants to learn about climate change? Should there be an additional tax on, say, car advertisements? Should angering videos be less viral?

Lê would be thrilled to see social choice theory applied to such critical moral questions.

== Scaled voting ==

One frequent remark that is being made is whether we really can (and should) agree on ethical issues. For instance, [https://www.nature.com/articles/s41586-018-0637-6 ADKSH+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Moral+Machine+experiment+nature+2018&btnG= 18] showed that Japanese prefer to save walkers, while Chinese prefer to save car passengers. Should we really enforce a common ethics worldwide?

Well, we probably don't need to. Cars could be programmed to save Japanese walkers and Chinese car passengers. They could be made to defend freedom in US and baguette in France. While humans usually have preferences for what happens elsewhere in the world, they usually have stronger preferences for what happens near their home. This probably is something that should be considered when designing voting-based ethics.

One proposal to reflect such nuances is [https://en.wikipedia.org/wiki/Quadratic_voting quadratic voting] [https://www.sss.ias.edu/files/pdfs/Rodrik/workshop%2014-15/Weyl-Quadratic_Voting.pdf LalleyWeyl][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=quadratic+voting+lalley+weyl&btnG=&oq=quadratic+voting 18], which can be made secure [https://link.springer.com/article/10.1007/s11127-017-0407-2 ParkRivest][https://dblp.org/rec/bibtex/journals/iacr/ParkR16 16]. In quadratic voting, a voter who wants its vote to weigh n times more must pay n2. This guarantees (asymptotic) efficiency (utilitarian outcome) and incentive-compatibility. But quadratic voting only applies to 2-alternative votes (typically statu quo vs new law) and is manipulable by collusion.

Another interesting point to be made about multidimensional voting is that the (geometric) median is strategy-proof for voters with a peaked preference, and a valuation that decreases with the distance to the peaked preference. The geometric median is particularly suited for, say, determining budget allocation through social choice. Weirdly, we don't know of a neat paper on this, though using sum of distance minimizer is well-known (need citation).

== Preferences versus volitions ==

It's been argued [https://intelligence.org/files/CEV.pdf Yudkowsky][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+yudkowsky&btnG= 04], [https://www.izmemar.com/files/CEV-MachineEthics.pdf Tarleton][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+tarleton&btnG= 10], [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.674.6424&rep=rep1&type=pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=the+value+learning+problem+soares&btnG= 15], [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19] that we surely should not aggregate today's human moral preferences, because of [[cognitive biases]] [https://www.amazon.com/s?k=thinking+fast+and+slow&ref=nb_sb_noss KahnemanBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=thinking+fast+and+slow+kahneman+2011&btnG= 11]. Mostly, our preferences are inconsistent, manipulable via framing, time-dependent, subject to addictions, and so on. We are likely to regret today's claimed preferences in the future, or as soon as we better understand their consequences. Instead, it is argued, we should program human [[volition|volitions]], which corresponds to what we would prefer to prefer, instead of what we simply prefer.

Unfortunately, a lot more research is needed to better formalize and analyze the concept of volition, and how it diverges from preferences. One fruitful path may be to analyze the difference between what's learned through [[inverse reinforcement learning]], as opposed to through (well-framed) elicitation. See [[volition]] for a lot more discussion on this problem.

Robustly Beneficial group

2020-02-21T13:23:03Z

Lê Nguyên Hoang: /* Past papers */

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. [https://arxiv.org/abs/1911.08265 SAHSS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Mastering+Atari%2C+Go%2C+Chess+and+Shogi+by+Planning+with+a+Learned+Model&btnG= 19].
* Intelligent Autonomous Things on the Battlefield. AI for the Internet of Everything. [https://arxiv.org/ftp/arxiv/papers/1902/1902.10086.pdf KottStump][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Intelligent+Autonomous+Things+on+the+Battlefield&btnG= 19] [https://youtu.be/gxwGkZeSg30 RB6].
* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18] [https://www.youtube.com/watch?v=bmD-myeu19Q RB5].
* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] [https://www.youtube.com/watch?v=_RuyXyekx6g RB4].
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* An fMRI Investigation of Emotional Engagement in Moral Judgment [https://science.sciencemag.org/content/sci/293/5537/2105.full.pdf?casa_token=7YSThrIwxB0AAAAA:cQJCIjltjkF3GT2V6Op-WBEExmGrwuOsvK6a93ejFZNi6pGRbrWRmoIEOlekacpUbRk04V06Jy9wC4k GSNDC][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=an+fmri+investigation+of+emotional+engagement+in+moral+judgment&btnG= 01]
* Reflections on Trusting Trust. Turing Award Lecture. [https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf Thompson][https://dblp.org/img/download.dark.hollow.16x16.png 84]

Robustly Beneficial group

2020-02-21T13:21:40Z

Lê Nguyên Hoang: /* Past papers */

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model. [https://arxiv.org/abs/1911.08265 SAHSS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Mastering+Atari%2C+Go%2C+Chess+and+Shogi+by+Planning+with+a+Learned+Model&btnG= 19].
* Intelligent Autonomous Things on the Battlefield. AI for the Internet of Everything. [https://arxiv.org/ftp/arxiv/papers/1902/1902.10086.pdf KottStump][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Intelligent+Autonomous+Things+on+the+Battlefield&btnG= 19].
* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18] [https://www.youtube.com/watch?v=bmD-myeu19Q RB5].
* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] [https://www.youtube.com/watch?v=_RuyXyekx6g RB4].
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* An fMRI Investigation of Emotional Engagement in Moral Judgment [https://science.sciencemag.org/content/sci/293/5537/2105.full.pdf?casa_token=7YSThrIwxB0AAAAA:cQJCIjltjkF3GT2V6Op-WBEExmGrwuOsvK6a93ejFZNi6pGRbrWRmoIEOlekacpUbRk04V06Jy9wC4k GSNDC][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=an+fmri+investigation+of+emotional+engagement+in+moral+judgment&btnG= 01]
* Reflections on Trusting Trust. Turing Award Lecture. [https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf Thompson][https://dblp.org/img/download.dark.hollow.16x16.png 84]

Robustly Beneficial group

2020-02-20T17:32:13Z

Lê Nguyên Hoang: /* Candidate future papers */

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* Intelligent Autonomous Things on the Battlefield. AI for the Internet of Everything. [https://arxiv.org/ftp/arxiv/papers/1902/1902.10086.pdf KottStump][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Intelligent+Autonomous+Things+on+the+Battlefield&btnG= 19].
* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18] [https://www.youtube.com/watch?v=bmD-myeu19Q RB5].
* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] [https://www.youtube.com/watch?v=_RuyXyekx6g RB4].
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* An fMRI Investigation of Emotional Engagement in Moral Judgment [https://science.sciencemag.org/content/sci/293/5537/2105.full.pdf?casa_token=7YSThrIwxB0AAAAA:cQJCIjltjkF3GT2V6Op-WBEExmGrwuOsvK6a93ejFZNi6pGRbrWRmoIEOlekacpUbRk04V06Jy9wC4k GSNDC][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=an+fmri+investigation+of+emotional+engagement+in+moral+judgment&btnG= 01]
* Reflections on Trusting Trust. Turing Award Lecture. [https://www.cs.cmu.edu/~rdriley/487/papers/Thompson_1984_ReflectionsonTrustingTrust.pdf Thompson][https://dblp.org/img/download.dark.hollow.16x16.png 84]

Impressive advances in AI

2020-02-17T21:10:20Z

Lê Nguyên Hoang: /* Image processing */

This page lists impressive results in machine learning.

== Image processing ==

Online photorealistic portrait generator is available at [http://thispersondoesnotexist.com thispersondoesnotexist.com].

[https://imglarger.com/ imglarger.com] proposes super-resolution enhancement, while [http://bgeraser.com/ bgeraser.com] proposes background erasing.

[https://pdfs.semanticscholar.org/af30/a2394c620132884bb98c78b6b9e46c791482.pdf WangKosinski][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Deep+neural+networks+are+more+accurate+than+humans+at+detecting+sexual+orientation+from+facial+images&btnG= 18] Detect gayness from picture.

Detect heart beat rate from video.

ZAO allows users to face swap from one picture, and [https://twitter.com/AllanXia/status/1168049059413643265 become Di Caprio in Titanic].

[https://arxiv.org/pdf/1902.06838 JoPark][https://dblp.org/rec/bibtex/journals/corr/abs-1902-06838 19] propose a GAN to easily [https://github.com/run-youngjoo/SC-FEGAN photoshop real images] in milliseconds. Similar inpaintings by NVIDIA are available [https://www.youtube.com/watch?v=gg0F5JjKmhA here].

[http://openaccess.thecvf.com/content_ICCV_2019/papers/Nirkin_FSGAN_Subject_Agnostic_Face_Swapping_and_Reenactment_ICCV_2019_paper.pdf NKH][https://dblp.org/rec/bibtex/journals/corr/abs-1908-05932 19] propose face swapping and reenactement techniques.

[http://openaccess.thecvf.com/content_ICCV_2019/papers/Zhou_Deep_Single-Image_Portrait_Relighting_ICCV_2019_paper.pdf ZHSJ][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Deep+Single+Image+Portrait+Relighting&btnG= 19] propose re-lightening techniques.

== Sound processing ==

[https://arxiv.org/pdf/1806.04558 JZWWS+][https://dblp.org/rec/bibtex/conf/nips/JiaZWWSRCNPLW18 18] learns to synthesize voice from 5-second samples.

[https://openai.com/blog/musenet/ Payne19] proposed MuseNet to create musical compositions.

[https://www.respeecher.com Respeecher] allows to create speeches in another person's voice.

Google Duplex.

== Natural language processing ==

[https://openai.com/blog/better-language-models/ RWAAC+19] proposed GPT2, which you can play with on [https://talktotransformer.com/ talktotransformer.com].

[https://openreview.net/pdf?id=H196sainb LCRDJ][https://dblp.org/rec/bibtex/conf/iclr/LampleCRDJ18 18] achieved state-of-the-art translations without parallel data. They simply found a rotation of the word embeddings of two languages that make the embeddings fit nicely.

== Games ==

AlphaGo, Alpha Zero.

AlphaStar.

== Social skills ==

[https://www.pnas.org/content/pnas/112/4/1036.full.pdf YKS][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Computer-based+personality+judgments+are+more+accurate+than+those+made+by+humans&btnG= 15] showed that algorithms outperformed even close relatives at predicting an individuals' personality traits derived from a combination of relatives' judgments.

[https://arxiv.org/pdf/1510.06769 EIYSDH][https://dblp.org/rec/bibtex/conf/icassp/EskimezIYSDH16 16] showed that algorithms could outperform naive humans at emotion classification based on images.

== Problem solving ==

AlphaFold [https://kstatic.googleusercontent.com/files/b4d715e8f8b6514cbfdc28a9ad83e14b6a8f86c34ea3b3cc844af8e76767d21ac3df5b0a9177d5e3f6a40b74caf7281a386af0fab8ca62f687599abaf8c8810f EJKSG+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=De+novo+structure+prediction+with+deep%C2%ADlearning+based+scoring&btnG= 18] [https://www.nature.com/articles/s41586-019-1923-7.epdf?author_access_token=Z_KaZKDqtKzbE7Wd5HtwI9RgN0jAjWel9jnR3ZoTv0MCcgAwHMgRx9mvLjNQdB2TlQQaa7l420UCtGo8vYQ39gg8lFWR9mAZtvsN_1PrccXfIbc6e-tGSgazNL_XdtQzn1PHfy21qdcxV7Pw-k3htw%3D%3D SEJKS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Improved+protein+structure+prediction+using+potentials+from+deep+learning&btnG= 20]

Deep learning for symbolic math [https://arxiv.org/pdf/1912.01412 LampleCharton][https://dblp.org/rec/bibtex/journals/corr/abs-1912-01412 19]

==

[https://www.amazon.com/Clinical-Versus-Statistical-Prediction-Theoretical/dp/0963878492 Meehl54] list examples from 1954 where statistical computations suffice to outperform clinical predictions by doctors (to verify).

Impressive advances in AI

2020-02-17T21:07:41Z

Lê Nguyên Hoang: /* Image processing */

This page lists impressive results in machine learning.

== Image processing ==

Online photorealistic portrait generator is available at [http://thispersondoesnotexist.com thispersondoesnotexist.com].

[https://imglarger.com/ imglarger.com] proposes super-resolution enhancement.

[https://pdfs.semanticscholar.org/af30/a2394c620132884bb98c78b6b9e46c791482.pdf WangKosinski][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Deep+neural+networks+are+more+accurate+than+humans+at+detecting+sexual+orientation+from+facial+images&btnG= 18] Detect gayness from picture.

Detect heart beat rate from video.

ZAO allows users to face swap from one picture, and [https://twitter.com/AllanXia/status/1168049059413643265 become Di Caprio in Titanic].

[https://arxiv.org/pdf/1902.06838 JoPark][https://dblp.org/rec/bibtex/journals/corr/abs-1902-06838 19] propose a GAN to easily [https://github.com/run-youngjoo/SC-FEGAN photoshop real images] in milliseconds. Similar inpaintings by NVIDIA are available [https://www.youtube.com/watch?v=gg0F5JjKmhA here].

[http://openaccess.thecvf.com/content_ICCV_2019/papers/Nirkin_FSGAN_Subject_Agnostic_Face_Swapping_and_Reenactment_ICCV_2019_paper.pdf NKH][https://dblp.org/rec/bibtex/journals/corr/abs-1908-05932 19] propose face swapping and reenactement techniques.

[http://openaccess.thecvf.com/content_ICCV_2019/papers/Zhou_Deep_Single-Image_Portrait_Relighting_ICCV_2019_paper.pdf ZHSJ][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Deep+Single+Image+Portrait+Relighting&btnG= 19] propose re-lightening techniques.

== Sound processing ==

[https://arxiv.org/pdf/1806.04558 JZWWS+][https://dblp.org/rec/bibtex/conf/nips/JiaZWWSRCNPLW18 18] learns to synthesize voice from 5-second samples.

[https://openai.com/blog/musenet/ Payne19] proposed MuseNet to create musical compositions.

[https://www.respeecher.com Respeecher] allows to create speeches in another person's voice.

Google Duplex.

== Natural language processing ==

[https://openai.com/blog/better-language-models/ RWAAC+19] proposed GPT2, which you can play with on [https://talktotransformer.com/ talktotransformer.com].

[https://openreview.net/pdf?id=H196sainb LCRDJ][https://dblp.org/rec/bibtex/conf/iclr/LampleCRDJ18 18] achieved state-of-the-art translations without parallel data. They simply found a rotation of the word embeddings of two languages that make the embeddings fit nicely.

== Games ==

AlphaGo, Alpha Zero.

AlphaStar.

== Social skills ==

[https://www.pnas.org/content/pnas/112/4/1036.full.pdf YKS][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Computer-based+personality+judgments+are+more+accurate+than+those+made+by+humans&btnG= 15] showed that algorithms outperformed even close relatives at predicting an individuals' personality traits derived from a combination of relatives' judgments.

[https://arxiv.org/pdf/1510.06769 EIYSDH][https://dblp.org/rec/bibtex/conf/icassp/EskimezIYSDH16 16] showed that algorithms could outperform naive humans at emotion classification based on images.

== Problem solving ==

AlphaFold [https://kstatic.googleusercontent.com/files/b4d715e8f8b6514cbfdc28a9ad83e14b6a8f86c34ea3b3cc844af8e76767d21ac3df5b0a9177d5e3f6a40b74caf7281a386af0fab8ca62f687599abaf8c8810f EJKSG+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=De+novo+structure+prediction+with+deep%C2%ADlearning+based+scoring&btnG= 18] [https://www.nature.com/articles/s41586-019-1923-7.epdf?author_access_token=Z_KaZKDqtKzbE7Wd5HtwI9RgN0jAjWel9jnR3ZoTv0MCcgAwHMgRx9mvLjNQdB2TlQQaa7l420UCtGo8vYQ39gg8lFWR9mAZtvsN_1PrccXfIbc6e-tGSgazNL_XdtQzn1PHfy21qdcxV7Pw-k3htw%3D%3D SEJKS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Improved+protein+structure+prediction+using+potentials+from+deep+learning&btnG= 20]

Deep learning for symbolic math [https://arxiv.org/pdf/1912.01412 LampleCharton][https://dblp.org/rec/bibtex/journals/corr/abs-1912-01412 19]

==

[https://www.amazon.com/Clinical-Versus-Statistical-Prediction-Theoretical/dp/0963878492 Meehl54] list examples from 1954 where statistical computations suffice to outperform clinical predictions by doctors (to verify).

Robustly Beneficial group

2020-02-17T08:05:09Z

Lê Nguyên Hoang:

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* Intelligent Autonomous Things on the Battlefield. AI for the Internet of Everything. [https://arxiv.org/ftp/arxiv/papers/1902/1902.10086.pdf KottStump][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Intelligent+Autonomous+Things+on+the+Battlefield&btnG= 19].
* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18] [https://www.youtube.com/watch?v=bmD-myeu19Q RB5].
* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] [https://www.youtube.com/watch?v=_RuyXyekx6g RB4].
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* An fMRI Investigation of Emotional Engagement in Moral Judgment [https://science.sciencemag.org/content/sci/293/5537/2105.full.pdf?casa_token=7YSThrIwxB0AAAAA:cQJCIjltjkF3GT2V6Op-WBEExmGrwuOsvK6a93ejFZNi6pGRbrWRmoIEOlekacpUbRk04V06Jy9wC4k GSNDC][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=an+fmri+investigation+of+emotional+engagement+in+moral+judgment&btnG= 01]

Impressive advances in AI

2020-02-15T16:08:21Z

Lê Nguyên Hoang:

This page lists impressive results in machine learning.

== Image processing ==

Online photorealistic portrait generator is available at [http://thispersondoesnotexist.com thispersondoesnotexist.com].

[https://pdfs.semanticscholar.org/af30/a2394c620132884bb98c78b6b9e46c791482.pdf WangKosinski][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Deep+neural+networks+are+more+accurate+than+humans+at+detecting+sexual+orientation+from+facial+images&btnG= 18] Detect gayness from picture.

Detect heart beat rate from video.

ZAO allows users to face swap from one picture, and [https://twitter.com/AllanXia/status/1168049059413643265 become Di Caprio in Titanic].

[https://arxiv.org/pdf/1902.06838 JoPark][https://dblp.org/rec/bibtex/journals/corr/abs-1902-06838 19] propose a GAN to easily [https://github.com/run-youngjoo/SC-FEGAN photoshop real images] in milliseconds. Similar inpaintings by NVIDIA are available [https://www.youtube.com/watch?v=gg0F5JjKmhA here].

[http://openaccess.thecvf.com/content_ICCV_2019/papers/Nirkin_FSGAN_Subject_Agnostic_Face_Swapping_and_Reenactment_ICCV_2019_paper.pdf NKH][https://dblp.org/rec/bibtex/journals/corr/abs-1908-05932 19] propose face swapping and reenactement techniques.

[http://openaccess.thecvf.com/content_ICCV_2019/papers/Zhou_Deep_Single-Image_Portrait_Relighting_ICCV_2019_paper.pdf ZHSJ][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Deep+Single+Image+Portrait+Relighting&btnG= 19] propose re-lightening techniques.

== Sound processing ==

[https://arxiv.org/pdf/1806.04558 JZWWS+][https://dblp.org/rec/bibtex/conf/nips/JiaZWWSRCNPLW18 18] learns to synthesize voice from 5-second samples.

[https://openai.com/blog/musenet/ Payne19] proposed MuseNet to create musical compositions.

[https://www.respeecher.com Respeecher] allows to create speeches in another person's voice.

Google Duplex.

== Natural language processing ==

[https://openai.com/blog/better-language-models/ RWAAC+19] proposed GPT2, which you can play with on [https://talktotransformer.com/ talktotransformer.com].

[https://openreview.net/pdf?id=H196sainb LCRDJ][https://dblp.org/rec/bibtex/conf/iclr/LampleCRDJ18 18] achieved state-of-the-art translations without parallel data. They simply found a rotation of the word embeddings of two languages that make the embeddings fit nicely.

== Games ==

AlphaGo, Alpha Zero.

AlphaStar.

== Social skills ==

[https://www.pnas.org/content/pnas/112/4/1036.full.pdf YKS][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Computer-based+personality+judgments+are+more+accurate+than+those+made+by+humans&btnG= 15] showed that algorithms outperformed even close relatives at predicting an individuals' personality traits derived from a combination of relatives' judgments.

[https://arxiv.org/pdf/1510.06769 EIYSDH][https://dblp.org/rec/bibtex/conf/icassp/EskimezIYSDH16 16] showed that algorithms could outperform naive humans at emotion classification based on images.

== Problem solving ==

AlphaFold [https://kstatic.googleusercontent.com/files/b4d715e8f8b6514cbfdc28a9ad83e14b6a8f86c34ea3b3cc844af8e76767d21ac3df5b0a9177d5e3f6a40b74caf7281a386af0fab8ca62f687599abaf8c8810f EJKSG+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=De+novo+structure+prediction+with+deep%C2%ADlearning+based+scoring&btnG= 18] [https://www.nature.com/articles/s41586-019-1923-7.epdf?author_access_token=Z_KaZKDqtKzbE7Wd5HtwI9RgN0jAjWel9jnR3ZoTv0MCcgAwHMgRx9mvLjNQdB2TlQQaa7l420UCtGo8vYQ39gg8lFWR9mAZtvsN_1PrccXfIbc6e-tGSgazNL_XdtQzn1PHfy21qdcxV7Pw-k3htw%3D%3D SEJKS+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Improved+protein+structure+prediction+using+potentials+from+deep+learning&btnG= 20]

Deep learning for symbolic math [https://arxiv.org/pdf/1912.01412 LampleCharton][https://dblp.org/rec/bibtex/journals/corr/abs-1912-01412 19]

==

[https://www.amazon.com/Clinical-Versus-Statistical-Prediction-Theoretical/dp/0963878492 Meehl54] list examples from 1954 where statistical computations suffice to outperform clinical predictions by doctors (to verify).

Robustly Beneficial group

2020-02-15T10:20:25Z

Lê Nguyên Hoang:

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18] [https://www.youtube.com/watch?v=bmD-myeu19Q RB5].
* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] [https://www.youtube.com/watch?v=_RuyXyekx6g RB4].
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* An fMRI Investigation of Emotional Engagement in Moral Judgment [https://science.sciencemag.org/content/sci/293/5537/2105.full.pdf?casa_token=7YSThrIwxB0AAAAA:cQJCIjltjkF3GT2V6Op-WBEExmGrwuOsvK6a93ejFZNi6pGRbrWRmoIEOlekacpUbRk04V06Jy9wC4k GSNDC][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=an+fmri+investigation+of+emotional+engagement+in+moral+judgment&btnG= 01]

Robustly Beneficial group

2020-02-14T06:54:43Z

Lê Nguyên Hoang: /* Past papers */

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] [https://www.youtube.com/watch?v=_RuyXyekx6g RB4].
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18]

Robustly Beneficial group

2020-02-14T06:54:10Z

Lê Nguyên Hoang:

The Robustly Beneficial group is an AI ethics group, started by [[User:Louis_Faucon|Louis Faucon]] and Sergei Volodin, in Lausanne, Switzerland. The group is now managed by [[User:Louis_Faucon|Louis Faucon]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]]. Every week, we discuss a paper relevant to AI ethics. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join].

== Past papers ==

* Focusing on the Long-Term: It's Good for Users and Business. KDD. [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15]
* Experimental evidence of massive-scale emotional contagion through social networks. PNAS. [https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] [https://www.youtube.com/watch?v=gQHvTow91FY RB3].
* Recent Advances in Algorithmic High-Dimensional Robust Statistics. [https://arxiv.org/pdf/1911.05911 DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].
* Algorithmic Accountability Reporting: On the Investigation of Black Boxes. [https://academiccommons.columbia.edu/doi/10.7916/D8ZK5TW2 Diakopoulos][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Algorithmic+Accountability+Reporting%3A+On+the+Investigation+of+Black+Boxes&btnG= 14] [https://www.youtube.com/watch?v=WWbw4cla2jw RB1].
* Efficient and Thrifty Voting by Any Means Necessary, NeurIPS. [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary.pdf MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19].
* The Vulnerable World Hypothesis, Global Policy. [https://nickbostrom.com/papers/vulnerable.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Vulnerable+World+Hypothesis&btnG= 19].
* Occam's razor is insufficient to infer the preferences of irrational agents, NeurIPS. [https://arxiv.org/pdf/1712.05812 ArmstrongMindermann][https://dblp.org/rec/bibtex/conf/nips/ArmstrongM18 18].
* Supervising strong learners by amplifying weak experts. [https://arxiv.org/pdf/1810.08575 CSA][https://dblp.org/rec/bibtex/journals/corr/abs-1810-08575 18].
* Embedded Agency. [https://arxiv.org/pdf/1902.09469.pdf DemskiGarrabrant][https://dblp.org/rec/bibtex/journals/corr/abs-1902-09469 19].
* Concrete Problems in AI Safety. [https://arxiv.org/pdf/1606.06565 AOSCSM][https://dblp.org/rec/bibtex/journals/corr/AmodeiOSCSM16 16].
* The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Minds and Machines. [https://www.nickbostrom.com/superintelligentwill.pdf Bostrom][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=THE+SUPERINTELLIGENT+WILL%3A+MOTIVATION+AND+INSTRUMENTAL+RATIONALITY+IN+ADVANCED+ARTIFICIAL+AGENTS&btnG= 12].
* On the Limits of Recursively Self-Improving AGI, AGI. [https://link.springer.com/content/pdf/10.1007%2F978-3-319-21365-1.pdf Yampolski][https://dblp.org/rec/bibtex/conf/agi/Yampolskiy15b 15].
* Can Intelligence Explode? [http://www.hutter1.net/publ/singularity.pdf Hutter][https://dblp.org/rec/bibtex/journals/corr/abs-1202-6177 12].
* Risks from Learned Optimization in Advanced Machine Learning Systems. [https://arxiv.org/pdf/1906.01820.pdf HMMSG][https://dblp.org/rec/bibtex/journals/corr/abs-1906-01820 19].
* The Value Learning Problem, IJCAI. [https://intelligence.org/files/ValueLearningProblem.pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Value+Learning+Problem+soares&btnG= 16].

== Candidate future papers ==

* Why Philosophers Should Care About Computational Complexity, ECCC. [https://www.scottaaronson.com/papers/philos.pdf Aaronson][https://dblp.org/rec/bibtex/journals/eccc/Aaronson11b 11].
* Facebook language predicts depression in medical records, PNAS. [https://www.pnas.org/content/pnas/115/44/11203.full.pdf ESMUC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Facebook+language+predicts+depression+in+medical+records&btnG= 18].
* WeBuildAI: Participatory Framework for Algorithmic Governance, PACMHCI. [https://www.cs.cmu.edu/~akahng/papers/webuildai.pdf LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19].
* Exposure to opposing views on social media can increase political polarization, PNAS. [https://www.pnas.org/content/pnas/115/37/9216.full.pdf BABBC+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Exposure+to+opposing+views+on+social+media+can+increase+political+polarization&btnG= 18].
* Multi-armed Bandit Models for the Optimal Design of Clinical Trials: Benefits and Challenges, Statistical science: a review journal of the Institute of Mathematical Statistics. [https://arxiv.org/pdf/1507.08025.pdf VBW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Multi-armed+Bandit+Models+for+the+Optimal+Design+of+Clinical+Trials%3A+Benefits+and+Challenges&btnG= 15].
* The complexity of agreement, STOC. [https://dl.acm.org/doi/pdf/10.1145/1060590.1060686 Aaronson][https://dblp.org/rec/bibtex/conf/stoc/Aaronson05 05].
* Reward Tampering Problems and Solutions in Reinforcement Learning. [https://arxiv.org/pdf/1908.04734.pdf EverittHutter][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Reward+Tampering+Problems+and+Solutions+in+Reinforcement+Learning&btnG= 19].
* AGI safety literature review, IJCAI. [https://arxiv.org/pdf/1805.01109 ELH][https://dblp.org/rec/bibtex/conf/ijcai/EverittLH18 18].
* The global landscape of AI ethics guidelines, Nature. [https://www.nature.com/articles/s42256-019-0088-2 JIV][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+global+landscape+of+AI+ethics+guidelines&btnG= 19].
* Tackling climate change with machine learning. [https://arxiv.org/pdf/1906.05433.pdf RDKKL+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Tackling+climate+change+with+machine+learning&btnG= 19].
* Science and Environmental Communication via Online Video: Strategically Distorted Communications on Climate Change and Climate Engineering on YouTube, Frontiers. [https://www.frontiersin.org/articles/10.3389/fcomm.2019.00036/pdf Allgaier][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Science+and+Environmental+Communication+via+Online+Video%3A+Strategically+Distorted+Communications+on+Climate+Change+and+Climate+Engineering+on+YouTube&btnG= 19]
* Efficient Learning from Comparisons. [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18]

Welcome to the Robustly Beneficial Wiki

2020-02-13T07:27:23Z

Lê Nguyên Hoang: /* Why AI ethics is becoming critical */

Welcome to the [[Robustly beneficial|Robustly Beneficial]] wiki!!

This wiki aims to better grasp the scope and the limits of current AI ethics research. It lists references, key ideas and relevant open questions to make algorithms robustly beneficial. Please check also our [https://www.youtube.com/watch?v=WWbw4cla2jw&list=PLgqL_7nXb23FKk_rUfs7vnvyrPshYPfA8 Robustly Beneficial Podcast] ([https://podcasts.apple.com/fr/podcast/robustly-beneficial-podcast/id1496159681 iTunes], [https://playlists.podmytube.com/UCgl_MmjatQif8juz3Lt6iPw/PLgqL_7nXb23FKk_rUfs7vnvyrPshYPfA8.xml RSS]), our [https://www.youtube.com/playlist?list=PLgqL_7nXb23HvhToBb9FwFxj83navY6oq&playnext=1&index=1 Robustly Beneficial Talks] and our [https://twitter.com/robustlyb Twitter account].

The wiki has just been launched, so most pages are still being written. But they will never be finished — this is the whole point of a wiki!

== The structure of the wiki ==

The wiki can be roughly divided into 4 main categories.

=== Why AI ethics is becoming critical ===

If you are new to AI ethics, you should probably start with the [[AI risks]] page. You could then go into arguably today's most important case of AI ethics, namely [[YouTube]]. Note that algorithms also offer formidable [[AI opportunities]] that are definitely worth considering. Find out more by reading about [[online polarization]], [[misinformation]], [[addiction]], [[mental health]] or [[hate]]. And as an example of an urgent AI ethics dilemma, check [https://twitter.com/le_science4all/status/1227690739104174080 this Twitter thread] on responses to a "is climate change a hoax?" search.

And if you know little about the current state of algorithmic research, you might want to check the latest [[impressive advances in AI]]. Or you could check some [[funny applications of AI]]. You can also read Lê's [https://www.lesswrong.com/posts/bwqDrSZvhEDKxRf6z/a-rant-against-robots rant against robots].

=== How today's (and probably tomorrow's) AIs work ===

The most important principle of today's AI is surely [[machine learning]]. Today, it mostly relies on [[stochastic gradient descent]] for (deep) [[neural networks]], which allow [[representational learning]] (see [[convolutional neural network]], [[residual network]], [[transformer]]). See also [[Turing 1950]], [[convexity]], [[generative adversarial network]], [[specialized hardware]] and [[linear systems]].

[[Bayesianism]] has been argued to be the ideal form of supervised and unsupervised learning, if we had infinite computational power (see [[Solomonoff's demon]], [[Laplace 1814]]). It has numerous desirable properties, like [[statistical admissibility]], [[Bayesian agreement]] or the [[Bayesian brain]] hypothesis. See also [[Bayesian examination]] and [[conjugate priors]].

A branch of learning called [[reinforcement learning]], which relies on [[Q-learning]] or [[policy learning]], seems likely to become the core framework of today's and tomorrow's AIs. [[AIXI]] achieves the upper-bound for [[Legg-Hutter intelligence]], which aims to measure [[artificial general intelligence|general intelligence]].

To understand the gap between Bayesianism/AIXI and practical machine learning, we need to understand the constraints of computational [[complexity]] theory. By building upon the [[Church-Turing thesis]], the [[Kolmogorov-Solomonoff complexity]] and knowledge from [[human brain computations]], this allows some insights into [[human-level AI]], in addition to [[experts' AI predictions]]. See also [[entropy]] and [[sophistication]].

AIs are already doing [[distributed learning]], which raises numerous challenges, like [[Byzantine fault tolerance]] and [[model drift]].

=== Why AI safety and ethics is harder than meets the eye ===

We want to get algorithms to do what we would really want them to do. But this turns out to raise numerous highly nontrivial problems, like [[Goodhart's law]], [[overfitting]], [[robust statistics]], [[confounding variables]], [[adversarial attacks]], [[algorithmic bias]], [[cognitive bias]], [[backfire effect]], [[distributional shift]], [[privacy]], [[human liabilities]], [[interpretability]], [[reward hacking]], [[wireheading]] and [[instrumental convergence]]. Because of all such problems, it seems crucial that algorithms be able to reason about their ignorance, using [[Bayesianism|Bayesian]] principles, [[moral uncertainty]] and [[second opinion querying]]. Algorithms must be [[robustly beneficial]].

AI ethics also demands that we solve thorny philosophical dilemmas, like the [[repugnant conclusion]], [[Newcomb's paradox]] and [[moral realism]]. Unfortunately, we have numerous [[cognitive bias|cognitive biases]], which seem critical to understand to solve AI ethics. Results about [[counterfactual]], [[von Neumann-Morgenstern theorem]] and [[Dutch book]] also seem useful to consider.

=== How to solve AI ethics (hopefully) ===

To solve AI ethics, [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19a] proposed the [[ABCDE roadmap]], which decomposes the [[alignment]] problem into numerous (hopefully) orthogonal and complementary subproblems. Such subproblems include [[data certification]], perhaps through [[Blockchain]], [[world model inference]] through [[Bayesianism]] and/or [[representational learning]], [[volition]] learning perhaps from [[Preference learning from comparisons|comparisons]] and [[social choice]] solutions, [[corrigibility]] and safe [[reinforcement learning]].

The fabulous endeavor to make AIs robustly beneficial can seem overwhelming, given how extraordinarily interdisciplinary it is. While it is worthwhile to have an overview of the problem, we believe it is also useful for aspiring contributors to identify more precise problems they can contribute to. In this wiki, we propose targeted research directions for different expertises and research interests. Please check the following pages that may be of interest to you.
<ul>
<li>[[how newcomers can contribute]]</li>
<li>[[how mathematicians can contribute]]</li>
<li>[[how computer scientists can contribute]]</li>
<li>[[how software engineers can contribute]]</li>
<li>[[how neuroscientists can contribute]]</li>
<li>[[how philosophers can contribute]]</li>
<li>[[how social scientists can contribute]]</li>
<li>[[how economists can contribute]]</li>
<li>[[how policy-makers can contribute]]</li>
<li>[[how journalists can contribute]]</li>
<li>[[how medical doctors can contribute]]</li>
<li>[[how physicists can contribute]]</li>
<li>[[how educators can contribute]]</li>
<li>[[how science communicators can contribute]]</li>
<li>[[how community builders can contribute]]</li>
</ul>

== About the authors ==

This wiki is written and edited mostly by members of the [[Robustly Beneficial group]], which regularly meets at EPFL, in Lausanne, Switzerland. Please feel free to [https://groups.google.com/forum/#!forum/lausannealignment ask to join]. So far, the main authors are [[User:Lê_Nguyên_Hoang|Lê Nguyên Hoang]], [[User:El_Mahdi_El_Mhamdi|El Mahdi El Mhamdi]] and [[User:Louis_Faucon|Louis Faucon]].

Lê and Mahdi recently co-wrote the book The Fabulous Endeavor: Make Artificial Intelligence Robustly Beneficial [https://laboutique.edpsciences.fr/produit/1107/9782759824304/Le%20fabuleux%20chantier HoangElmhamdi][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Le+fabuleux+chantier%3A+Rendre+l%27intelligence+artificielle+robustement+b%C3%A9n%C3%A9fique&btnG= 19FR] (the English version is pending).

Social choice

2020-02-11T16:55:36Z

Lê Nguyên Hoang: /* Applications to AI Ethics */

Social choice is the study of how to elicit, aggregate and explain human preferences for collective decision-making. This is critical to AI ethics, as we will need to decide collectively on which ethics an AI will follow. For instance, what video should be recommended by the YouTube algorithm when a user queries "Trump", "vaccine" or "social justice"?

== Background ==

Social choice theory arguably started with a remarkable memoire by Condorcet [http://classiques.uqac.ca/classiques/condorcet/Essai_application_discours_preliminaire/discours_preliminaire.pdf Condorcet][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Essai+sur+l%27application+de+l%27analyse+%C3%A0+la+probabilit%C3%A9+des+d%C3%A9cisions+rendues+%C3%A0+la+pluralit%C3%A9+des+voix.+1785&btnG= 1785]. He argued that if one alternative is preferred to any other alternative by a majority then it should be selected. This is the [https://en.wikipedia.org/wiki/Condorcet_criterion Condorcet principle] (see [https://www.youtube.com/watch?v=hI89r4LqaCc MrPhi17aFR], [https://www.youtube.com/watch?v=ZZb4TjvupkI MrPhi17bFR]).

Unfortunately, social choice theory is plagued with impossibility results, like the Condorcet paradox ([https://www.youtube.com/watch?v=HoAnYQZrNrQ PBSInfinite17], [https://www.youtube.com/watch?v=v8-2YdUqQqM MicMaths15FR]), Arrow's impossibility theorem ([https://s3.amazonaws.com/academia.edu.documents/40888103/arrow.pdf?response-content-disposition=inline%3B%20filename%3DArrow.pdf&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIWOWYYGZ2Y53UL3A%2F20200118%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20200118T214628Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=283c657c6dc5c2c225f77e14996a77846974b6dd3a2008f4e299131c6255fd75 Arrow][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+Difficulty+in+the+Concept+of+Social+Welfare&btnG= 50], [https://www.youtube.com/watch?v=AhVR7gFMKNg PBSInfinite17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]) and the Gibbard-Satterthwaite theorem ([https://www.youtube.com/watch?v=m5crte26fiw Wandida17], [https://www.youtube.com/watch?v=VNcj7-XUhoc S4A17bFR]). Different voting systems yield different winners ([https://www.youtube.com/watch?v=vfTJ4vmIsO4 StatChat16FR], [https://www.youtube.com/watch?v=fBYCoPAmpr4&t=371s S4A17aFR]).

Today's most convincing social choice mechanisms are probably the [https://en.wikipedia.org/wiki/Approval_voting approval voting] ([https://dblp.org/rec/bibtex/books/daglib/0017739 BramsFishburnBook07], [https://www.youtube.com/watch?v=orybDrUj4vA CGPGrey]), [https://en.wikipedia.org/wiki/Majority_judgment majority judgment] ([https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=majority+judgment+laraki+balinski&btnG= BalinksiLarakiBook11], [https://www.youtube.com/watch?v=ZoGH7d51bvc ScienceÉtonnante16FR], [https://www.youtube.com/watch?v=_MAo8pUl0U4 S4A17cFR]) and the randomized Condorcet voting system ([https://dspace.mit.edu/bitstream/handle/1721.1/107673/355_2017_1031_ReferencePDF.pdf?sequence=1&isAllowed=y Hoang][https://dblp.org/rec/bibtex/journals/scw/Hoang17 17], [https://www.youtube.com/watch?v=wKimU8jy2a8 S4A17dFR], [https://www.youtube.com/watch?v=vAdGZkXhlNM S4A17eFR]).

== The (Huge) Flaw of Classical Social Choice ==

Unfortunately, such approaches are limited because they can only handle a reasonable amount of alternatives. If we are to design AI ethics collectively, we need to choose a code (or, say, guidelines or texts of laws). Yet there are combinatorially many such codes! If we consider 1,000-line codes, this would represent ~210,000 alternatives. Classical voting systems won't do the trick.

Now, there are already lots of results in social choice for structured combinatorial sets of alternatives, mostly derived from auction theory ([https://en.wikipedia.org/wiki/Vickrey%E2%80%93Clarke%E2%80%93Groves_mechanism VCG mechanism] [https://dblp.org/rec/bibtex/books/cu/NRTV2007 NRTV07] [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR], Myerson's auction [https://dblp.org/rec/bibtex/journals/mor/Myerson81 Myerson81] [https://www.youtube.com/watch?v=FjP5JMUVXxw S4A17gFR], Gale-Shapley [https://dblp.org/rec/bibtex/journals/tamm/GaleS13 GaleShapley62] [https://www.youtube.com/watch?v=Qcv1IqHWAzg Numberphile14] [https://www.youtube.com/watch?v=oHYcOXi06uY S4A17hFR]...). Most impressively, in a series of papers [https://dblp.org/rec/bibtex/journals/sigecom/CaiDW11 CDW11], [https://dblp.org/rec/bibtex/conf/stoc/CaiDW12 CDW12a], [https://dblp.org/rec/bibtex/conf/focs/CaiDW12 CDW12b], [https://dblp.org/rec/bibtex/conf/soda/CaiDW13 CDW13a] and [https://dblp.org/rec/bibtex/conf/focs/CaiDW13 CDW13b], Cai, [https://en.wikipedia.org/wiki/Constantinos_Daskalakis Daskalakis] and Weinberg proved that the polynomial tractability of a Bayesian social choice approximation problem (i.e. with incentive-compatibility constraints) is equivalent to that of the full-information problem with an additional social welfare term to be optimized (see [https://www.youtube.com/watch?v=qruxfBdYTh8 S4A17fFR]).

However, we surely have to also tackle the case of unstructured combinatorial sets of alternatives (also, polytime may be too slow in practice).

== Bounds for limited communication complexity ==

A fascinating result by [http://papers.nips.cc/paper/8939-efficient-and-thrifty-voting-by-any-means-necessary MPSW][https://dblp.org/rec/bibtex/conf/nips/MandalP0W19 19] shows that the worst-case lost of social welfare due to polynomial communication complexity (voters communicate at most log(#alternatives) bits) is unbounded (#alternatives2 for deterministic elicitation+voting, #alternatives for randomized).

There are caveats though. For one thing, this is a worst-case analysis. But human preferences may be more structured. Also, priors can be invoked. Plus, authors assumed that the same elicitation was applied to all voters, which is clearly suboptimal. A lot more research on the communication-complexity versus social-welfare tradeoff is definitely desired. This is exciting!

== Applications to AI Ethics ==

It has been argued to be critical to solve the problem of AI ethics ([https://www.aaai.org/ocs/index.php/AAAI/AAAI16/paper/view/12457/12204 GRTVW][https://dblp.org/rec/bibtex/conf/aaai/GreeneRTVW16 16],[http://isaim2018.cs.virginia.edu/papers/ISAIM2018_Ethics_Conitzer_etal.pdf CSBDK][https://dblp.org/rec/bibtex/conf/isaim/ConitzerSBD018 17]). In brief, we are unlikely to agree on what ethics to program. However, we might be able to agree on how to agree on some ethics to program even though we disagree. The trick to implement some (virtual) democratic voting on moral preferences.

Interestingly, ideas along these lines have already been developed for the cases of autonomous car dilemmas [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17052/15857 NGADR+][https://dblp.org/rec/bibtex/conf/aaai/NoothigattuGADR18 18] [https://www.youtube.com/watch?v=Y6jfGZXubq0 UpAndAtom18], kidney transplant [https://www.aaai.org/ocs/index.php/AAAI/AAAI18/paper/view/17384/15863 FBSDC][https://dblp.org/rec/bibtex/conf/aaai/FreedmanBSDC18 18] and food donation (called WeBuildAI [https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=3&cad=rja&uact=8&ved=2ahUKEwj4y5-ggu3mAhXB2qQKHT6GDZ4QFjACegQIARAC&url=https%3A%2F%2Fwww.cs.cmu.edu%2F~akahng%2Fpapers%2Fwebuildai.pdf&usg=AOvVaw2BknquyvgNufy-JlCoPj_G LKKKY+][https://dblp.org/rec/bibtex/journals/pacmhci/LeeKKKYCSNLPP19 19]).

Note that in all such applications, the set of alternatives is combinatorially large. The trick to perform voting with limited elicitation from voters is to collect binary-choice-based preferences, and to then extrapolate preferences for other cases using machine learning (with some inductive bias). Another way to interpret this is to consider that voters get substituted by digital surrogates, whose task is to answer just as the voters would. This is kind of like representative democracy, where voters are replaced by their representatives. But machine learning can allow individual representatives through customized surrogates!

To build trust in the surrogate, WeBuildAI proposes to voters to test their surrogates, and to replace it, if needs be, by some computational model of their owns. They show that such interactions build trust from voters. They also propose some [[interpretability]] framework, where voters are given the implications of the vote of their surrogates.

Now what Lê can't wait for, is for all such frameworks to be applied to problems that really matter, because they influence billions of people. Yes, Lê is (again!) talking about recommender algorithms of social medias like YouTube. How should hate speech be moderated? What should be shown to someone who wants to learn about climate change? Should there be an additional tax on, say, car advertisements? Should angering videos be less viral?

Lê would be thrilled to see social choice theory applied to such critical moral questions.

== Scaled voting ==

One frequent remark that is being made is whether we really can (and should) agree on ethical issues. For instance, [https://www.nature.com/articles/s41586-018-0637-6 ADKSH+][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Moral+Machine+experiment+nature+2018&btnG= 18] showed that Japanese prefer to save walkers, while Chinese prefer to save car passengers. Should we really enforce a common ethics worldwide?

Well, we probably don't need to. Cars could be programmed to save Japanese walkers and Chinese car passengers. They could be made to defend freedom in US and baguette in France. While humans usually have preferences for what happens elsewhere in the world, they usually have stronger preferences for what happens near their home. This probably is something that should be considered when designing voting-based ethics.

One proposal to reflect such nuances is [https://en.wikipedia.org/wiki/Quadratic_voting quadratic voting] [https://www.sss.ias.edu/files/pdfs/Rodrik/workshop%2014-15/Weyl-Quadratic_Voting.pdf LalleyWeyl][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=quadratic+voting+lalley+weyl&btnG=&oq=quadratic+voting 18], which can be made secure [https://link.springer.com/article/10.1007/s11127-017-0407-2 ParkRivest][https://dblp.org/rec/bibtex/journals/iacr/ParkR16 16]. In quadratic voting, a voter who wants its vote to weigh n times more must pay n2. This guarantees (asymptotic) efficiency (utilitarian outcome) and incentive-compatibility. But quadratic voting only applies to 2-alternative votes (typically statu quo vs new law) and is manipulable by collusion.

Another interesting point to be made about multidimensional voting is that the (geometric) median is strategy-proof for voters with a peaked preference, and a valuation that decreases with the distance to the peaked preference. The geometric median is particularly suited for, say, determining budget allocation through social choice. Weirdly, we don't know of a neat paper on this, though using sum of distance minimizer is well-known (need citation).

== Preferences versus volitions ==

It's been argued [https://intelligence.org/files/CEV.pdf Yudkowsky][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+yudkowsky&btnG= 04], [https://www.izmemar.com/files/CEV-MachineEthics.pdf Tarleton][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=coherent+extrapolated+volition+tarleton&btnG= 10], [http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.674.6424&rep=rep1&type=pdf Soares][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=the+value+learning+problem+soares&btnG= 15], [http://ceur-ws.org/Vol-2301/paper_1.pdf Hoang][https://dblp.org/rec/bibtex/conf/aaai/Hoang19 19] that we surely should not aggregate today's human moral preferences, because of [[cognitive biases]] [https://www.amazon.com/s?k=thinking+fast+and+slow&ref=nb_sb_noss KahnemanBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=thinking+fast+and+slow+kahneman+2011&btnG= 11]. Mostly, our preferences are inconsistent, manipulable via framing, time-dependent, subject to addictions, and so on. We are likely to regret today's claimed preferences in the future, or as soon as we better understand their consequences. Instead, it is argued, we should program human [[volition|volitions]], which corresponds to what we would prefer to prefer, instead of what we simply prefer.

Unfortunately, a lot more research is needed to better formalize and analyze the concept of volition, and how it diverges from preferences. One fruitful path may be to analyze the difference between what's learned through [[inverse reinforcement learning]], as opposed to through (well-framed) elicitation. See [[volition]] for a lot more discussion on this problem.

Convexity

2020-02-11T15:50:08Z

Lê Nguyên Hoang: /* Neural networks and convexity */

A convex set is one such that the segment between any two points of the set still belongs to the set. In other words, if <math>x,y \in S</math> and <math>\lambda \in [0,1]</math>, then <math>\lambda x + (1-\lambda) y \in S</math>. A convex function <math>f</math> is such that <math>f(\lambda x + (1-\lambda) y) \leq \lambda f(x) + (1-\lambda) f(y)</math></math>. Put differently, the image of the average is below the average of the images.

Convexity play a central role in optimization, because optimization of convex functions over convex sets can be done efficiently, for instance using (variants of) [[stochastic gradient descent|gradient descents]]. Yet, weirdly, though neural networks learning is not a convex problem, it has become the most successful approach for machine learning.

== Examples of convex optimization ==

MSE linear regression, cross-entropy, SVM, logistic regression.

== Neural networks and convexity ==

[http://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf JHG][https://dblp.org/rec/bibtex/conf/nips/JacotHG18 18] showed that, in the infinite-width limit, neural networks could be regarded as a convex optimization of neural networks in the functional space [https://www.youtube.com/watch?v=7WmOSqcBDr0 ZettaBytes19].

On another note, the recent discovery of [[overfitting|double descent]] [https://openreview.net/pdf?id=Sy8gdB9xx ZBHRV][https://dblp.org/rec/bibtex/conf/iclr/ZhangBHRV17 17] [https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19] suggest that overparameterized neural networks are better at learning and generalization. The learning of overparameterized neural networks then seems "nearly convex" [https://arxiv.org/pdf/1811.08888 ZCZG][https://dblp.org/rec/bibtex/journals/corr/abs-1811-08888 18] [https://openreview.net/pdf?id=BylIciRcYQ ZYZLT][https://dblp.org/rec/bibtex/conf/iclr/ZhouYZLT19 19] [http://proceedings.mlr.press/v97/allen-zhu19a/allen-zhu19a.pdf ZLS][https://dblp.org/rec/bibtex/conf/icml/Allen-ZhuLS19 19] [http://proceedings.mlr.press/v97/du19c/du19c.pdf DLLWZ][https://dblp.org/rec/bibtex/conf/icml/DuLL0Z19 19].

Robust statistics

2020-02-11T08:50:26Z

Lê Nguyên Hoang: /* Poisoning models */

Robust statistics is the problem of estimating parameters from unreliable empirical data. Typically, suppose that a fraction of the training dataset is compromised. Can we design algorithms that nevertheless succeed in learning adequately from such a partially compromised dataset?

This question has arguably become crucial, as large-scale algorithms perform learning from users' data. Yet, clearly, if the algorithm is used by thousands, millions or billions of users, many of the data will likely be corrupted, because of bugs [https://www.youtube.com/watch?v=yb2zkxHDfUE standupmaths20], or because some users will maliciously want to exploit or attack the algorithm. This latter case is known as a poisoning attack.

Over the last three years, there have been fascinating recent advances, both for classical statistical tasks [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] and [[modern machine learning]] [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] especially in very high dimensional settings such as [[training neural networks]] [http://proceedings.mlr.press/v80/mhamdi18a EGR]. We discussed robust statistics in [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].

== Example of the median ==

Suppose the data are real numbers, and we want to estimate the mean of the true data (which we shall call inliers). Note that the naive empirical mean estimate would be a bad idea here, as a single malicious user could completely upset the empirical mean estimate. In fact, by choosing its input data adequately (called outliers), the malicious user can make the empirical mean estimate equal whatever the malicious user wants it to be.

It turns out that using the median of the dataset is a robust way to do so. Indeed, even if 45% of the data are outliers, the median will still be a quantile of the inliers, which should not be too far from the actual mean. The median is said to have a 0.5 statistical breakdown point [https://www.researchgate.net/profile/Peter_Rousseeuw/publication/303193534_Robust_Regression_Outlier_Detection_John_Wiley_Sons/links/5d7a06b64585151ee4af67da/Robust-Regression-Outlier-Detection-John-Wiley-Sons.pdf RousseeuwLeroy][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+regression+and+outlier+detection+rousseeuw+leroy&btnG= 05]. No statistical method can achieve a better breakdown point, but other methods also achieve 0.5 statistical breakdown, like trimmed mean (we remove sufficiently many extreme values on both sides and take the mean of the rest).

Another way to quantify robustness is to compute a high-probability upper bound between the empirical median and the mean μ of the true distribution of inliers. Call ε the fraction of outliers. It turns out that, assuming the true distribution is a normal distribution <math>\mathcal N(\mu,1)</math>, given <math>n=\Omega\left( \frac{d+\log(1/\tau)}{\varepsilon^2}\right)</math> data, we can guarantee <math>|median-\mu| = O(\varepsilon)</math> with probability <math>1-\tau</math>. This asymptotic bound is also best possible [https://projecteuclid.org/download/pdf_1/euclid.aoms/1177703732 Huber][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+estimation+of+a+location+parameter+huber&btnG= 92].

== Poisoning models ==

The above model holds for arguably the strongest poisoning model. This is one where an adversary gets to read the full dataset before we can, and is able to erase a fraction <math>\varepsilon</math> of the data, and to replace them by any imaginable input. The dataset is then analyzed by our (robust) statistics algorithm.

A weaker, but still widespread, model is one where a fraction <math>1-\varepsilon</math> comes from the true distribution, while the remaining <math>\varepsilon</math> is chosen by the adversary [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BMGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17].

Other models include an adversary with only erasing power, or an adversary that must choose its outliers without knowledge of the values of the inliers. Evidently, any guarantee for such weaker poisoning models will also hold for stronger poisoning models [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19].

Perhaps the most general form of poisoning attack is the following. Consider a "true dataset" <math>D</math>. However, the attacker gets to distort the dataset using a distort function <math>f \in \mathcal F</math>, thereby yielding <math>f(D)</math>. Suppose we now have a best-possible machine learning algorithm <math>ML</math> that learns from data. It would ideally compute <math>ML(D)</math>. But we can only exploit <math>f(D)</math>, by some hopefully robust machine learning algorithm <math>RML(f(D))</math>. What we would like is to guarantee that <math>d(ML(D),RML(f(D))) < bound</math>, for a suitable distance <math>d</math> and any <math>f \in \mathcal F</math>, where <math>RML</math> is tractable.

A further generalization of this could consist in assuming a prior probabilistic belief on the set <math>\mathcal F</math> of attack models that we need to defend against. This would correspond to the study of Byzantine Bayesian learning, which we may model as <math>\max_{RML} \mathbb E_{\mathcal F} \left[ \min_{f \in \mathcal F} \mathbb E[u|D,RML(f(D))] \right]</math>.

It is noteworthy that robustness to such attacks are useful, even if there are no adversary. Indeed, data may still get corrupted, because of bugs, crashes or misuse by a human operator (see for instance the case of gene mutations caused by Excel that plagued genetic research [https://www.sciencemag.org/news/2016/08/one-five-genetics-papers-contains-errors-thanks-microsoft-excel Boddy16]). Our algorithms need to remain performant despite such issues. An algorithm robust to strong attacks will be robust to such weaker flaws.

== Robustness to additive poisoning ==

Unfortunately, results that hold for small dimensions generalize poorly to high dimensions, either because of weak robustness guarantees or computational slowness. Typically, the coordinate-wise median and the [[geometric median]] both yield <math>\Omega(\varepsilon \sqrt{d})</math>-error, even in the limit of infinite-size datasets and assuming normality for inliers. This is very bad, as today's [[neural networks]] often have <math>d\sim 10^6</math>, if not <math>10^9</math> or <math>10^{12}</math> parameters.

On the other hand, assuming the true distribution is a spherical normal distribution <math>\mathcal N(0,I)</math>, Tukey proposed another approach based on identifying the directions of largest variances, since these are likely to be the "attack line" of the adversary [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Mathematics+and+the+picturing+of+data+tukey&btnG= Tukey75]. Tukey's median yields <math>O(\varepsilon)</math>-error with high probability <math>1-\tau</math> for <math>n=\Omega\left( \frac{d+\log(1/\tau)}{\varepsilon^2}\right)</math> data points. Unfortunately, Tukey's median is NP-hard to compute, and is typically exponential in d.

[https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] proved that such a bound can be achieved for additive poisoning in quasi-linear-time, even for heavy-tailed distribution with bounded but unknown variance. More precisely, by using an approach akin to median-of-means, for <math>K = \Omega(\varepsilon n)</math>, they designed an algorithm that achieves an error <math>O\left( \sqrt{\frac{Tr(\Sigma)}{n}} + \sqrt{\frac{||\Sigma||_{op} K}{n}} \right)</math> in time <math>\tilde O(nd + uKd)</math>. Here, <math>u</math> is an integer parameter, which is needed to guarantee a subgaussian decay rate of errors, which will be in <math>1-\exp(-\Theta(K+u))</math>. Note that <math>Tr(\Sigma)</math> is essentially the "effective dimension" of the data points.

Their technique relies on partitioning data into <math>K</math> buckets, computing the means for each bucket, and replacing the computation of median of the means by a covering SDP that fits all configuration of bucket poisoning. It turns out that approximations of such an covering SDP can be founded in quasi-linear-time [https://arxiv.org/pdf/1201.5135 PTZ][https://dblp.org/rec/bibtex/conf/spaa/PengT12 12]. It turns out that, rather than being used to directly compute a mean estimator, this is actually used to perform gradient descent, starting from the coordinate-wise median, and then descending along the direction provided by the covering SDP. [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] proved that only <math>O(\log d)</math> such steps were needed to guarantee their bound.

== Robustness to strong poisoning ==

Note that all the papers of this section seem to strongly rely on the knowledge of the covariance matrix of inliers by the algorithm.

For robustness to strong poisoning, Tukey's ideas can be turned into an polynomial-time algorithm for robust statistics mean estimate. The trick is to identify worst-case "attack line" by computing the largest eigenvalue of the empirical covariance matrix, and to remove extremal points along such lines to reduce variance. [https://arxiv.org/pdf/1604.06443.pdf DKKLMS][https://dblp.org/rec/bibtex/conf/focs/DiakonikolasKK016 16] [http://proceedings.mlr.press/v70/diakonikolas17a/diakonikolas17a.pdf DKKLMS][https://dblp.org/rec/bibtex/conf/icml/DiakonikolasKK017 17] show that, for <math>n=\Omega(d/\varepsilon^2)</math>, this yields <math>O(\varepsilon \sqrt{\log(1/\varepsilon)})</math>-error with high probability for sub-Gaussian inliers, and <math>O(\sigma \sqrt{\varepsilon})</math> for inliners whose true covariance matrix <math>\Sigma</math> is such that <math>\sigma^2 I - \Sigma</math> is semidefinite positive.

The asymptotical optimal bound <math>O(\varepsilon)</math> has been achieved a more sophisticated filtering polynomial-time algorithm by [https://epubs.siam.org/doi/pdf/10.1137/1.9781611975031.171 DKKLM+][https://dblp.org/rec/bibtex/conf/soda/DiakonikolasKK018 18] for Gaussian distribution in the additive poisoning model, while [https://ieeexplore.ieee.org/document/8104048 DKS][https://dblp.org/rec/bibtex/conf/focs/DiakonikolasKS17 17] showed that no polynomial-time can achieve better than <math>O(\varepsilon \sqrt{\log(1/\varepsilon)})</math> in the Statistical Query Model with strong poisoning.

Quasi-linear-time robust mean estimators have been designed by [https://epubs.siam.org/doi/pdf/10.1137/1.9781611975482.171 CDG][https://dblp.org/rec/bibtex/conf/soda/0002D019 19], i.e. with <math>\tilde O(nd)</math> up to logarithmic factors, based on filtering methods of extremal points on variance-maximizing directions.

Note that all such results can be applied to robust linear regression, by applying robust mean estimator to gradient descent estimator (with the mean taken over data points), assuming that the covariance matrix of the distribution of features <math>x</math> is known, and that the noise is of known mean and variance. Robust covariance matrix estimation can also be addressed by the framework, as it too can be regarded as a robust mean estimation problem (the fourth moment then needs to be assumed to be upper-bounded).

== What if there are more outliers than inliers? ==

Another puzzling question concerns the setting where outliers may be more numerous than inliers. A simple argument shows that methods cited above just won't work. What can be done for such a setting? Can we still learn something from the data, or do we have to throw away the dataset altogether? Let's cite three approaches that may be fruitful.

First, [https://dl.acm.org/doi/pdf/10.1145/1374376.1374474?casa_token=CY0VwFuDnskAAAAA:Q_eNEhY5XxJ8_rll_t1TeRYQiTB6fJeTXQG_OJGwVeghBAcpD6rwFCExUqrmwX5SP-N9iKd8n2CxKQ BBV][https://dblp.org/rec/bibtex/conf/stoc/BalcanBV08 08] introduced list-decodable learning, which consists in returning several hypothesis, and [https://arxiv.org/pdf/1711.07211 DKS][https://dblp.org/rec/bibtex/conf/stoc/DiakonikolasKS18 18] provided polynomial-time for robust list-decodable mean estimation.

Second, one might try to a apply a robust Bayesian inference to the data, which would yield a set of posterior beliefs. This framework has yet to be defined.

Third, we may assume that the data are more or less [[data certification|certified]]. This is a natural setting, as we humans often judge the reliability of raw data depending on its source, and we usually consider a continuum of reliability, rather than a clear cut binary classification. Algorithms should probably do the same at some point, but there does not yet seem to be an algorithmic framework to pose this problem. In particular, it could be interesting to analyze threat models where different degrees or sorts of certification have different levels of liability.

== Robust statistics for neural networks ==

The main application of robust statistics (at least relevant to AI ethics) seems to be the aggregation of [[stochastic gradient descent|stochastic gradients]] for [[neural networks]]. In this setting, even a linear-time algorithm in <math>\Omega(nd)</math> is impractical if we demand <math>n \geq d</math> (which is necessary to have dimension-independent guarantees). In practice, this setting is often carried out with batches whose size <math>n</math> is significantly smaller than <math>d</math>. In fact, despite conventional wisdom and PAC-learning theory, it seems that <math>n \ll d</math> may be a desirable setting to do neural network learning (see [[overfitting]] where we discuss double descent). For <math>n \ll d</math>, is there a gain in using algorithms more complex than coordinate-wise median?

[http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] proposed Krum and multi-Krum, aggregation algorithms for this setting that have weaker robustness guarantees but are more efficient. Is it possible to improve upon them?

== Robust statistics for agreement and multi-agent settings ==

Another venue where robust statistics are needed is the increasingly multi-agent setting that modern AI is built on. [https://dl.acm.org/doi/10.1145/2488608.2488657 MH2013]

Convexity

2020-02-11T08:10:10Z

Lê Nguyên Hoang: Created page with "A convex set is one such that the segment between any two points of the set still belongs to the set. In other words, if <math>x,y \in S</math> and <math>\lambda \in [0,1]</ma..."

A convex set is one such that the segment between any two points of the set still belongs to the set. In other words, if <math>x,y \in S</math> and <math>\lambda \in [0,1]</math>, then <math>\lambda x + (1-\lambda) y \in S</math>. A convex function <math>f</math> is such that <math>f(\lambda x + (1-\lambda) y) \leq \lambda f(x) + (1-\lambda) f(y)</math></math>. Put differently, the image of the average is below the average of the images.

Convexity play a central role in optimization, because optimization of convex functions over convex sets can be done efficiently, for instance using (variants of) [[stochastic gradient descent|gradient descents]]. Yet, weirdly, though neural networks learning is not a convex problem, it has become the most successful approach for machine learning.

== Examples of convex optimization ==

MSE linear regression, cross-entropy, SVM, logistic regression.

== Neural networks and convexity ==

[http://papers.nips.cc/paper/8076-neural-tangent-kernel-convergence-and-generalization-in-neural-networks.pdf JHG][https://dblp.org/rec/bibtex/conf/nips/JacotHG18 18] showed that, in the infinite-width limit, neural networks could be regarded as a convex optimization of neural networks in the functional space [https://www.youtube.com/watch?v=7WmOSqcBDr0 ZettaBytes19].

On another note, the recent discovery of [[overfitting|double descent]] [https://openreview.net/pdf?id=Sy8gdB9xx ZBHRV][https://dblp.org/rec/bibtex/conf/iclr/ZhangBHRV17 17] [https://arxiv.org/pdf/1912.02292 NKBYBS][https://dblp.org/rec/bibtex/journals/corr/abs-1912-02292 19] suggest that overparameterized neural networks are better at learning and generalization. It may or may not be that the learning of overparameterized neural networks is "nearly convex". A conjecture that we might make is that any random initialization of a very overparameterized neural network is, with high probability, in a convex region where the loss function of the learning problem is convex, and where its minimum is zero.

Mental health

2020-02-06T18:25:42Z

Lê Nguyên Hoang: /* Impact */

One of the most exciting [[AI opportunities]] is tackling large-scale mental health.

== Loneliness ==

[https://www.youtube.com/watch?v=n3Xv_g3g-mA Kurzgesagt19] stresses the fact that loneliness is a growing problem, which leads to other undesirable consequences like increased cancer risks, ageing and aggressivity.

== Impact ==

[https://www.pnas.org/content/pnas/111/24/8788.full.pdf KGH][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Experimental+evidence+of+massive-scale+emotional+contagion+through+social+networks&btnG= 14] tweaked the Facebook recommendation algorithm for a week, for three groups of subjects. One was the controlled group. For a second group, 10% of the posts with negative words were removed. For a third group, 10% of the posts with positive words were removed. The experiment showed that the second group then wrote more positive posts and fewer negative posts, while the third group wrote less positive posts and more negative posts. Effects are small. But the treatment and its length were small too. We discussed this paper in [https://www.youtube.com/watch?v=gQHvTow91FY RB3].

On the other hand, [https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43887.pdf HOT][https://dblp.org/rec/bibtex/conf/kdd/HohnholdOT15 15] suggest that longer exposure could yield important effects. More precisely, they test learned ads "blindness" or "sightedness". Namely, they exposed some test group to greater ad loads, and tracked the click-through-rate of the test group, as opposed to, on each day, a randomly selected control group of users who got exposed to increased ad load on this single day. Crucially, they found that the learned ad blindness (or sightedness) was acquired after months. They modeled the learning by a function of the form <math>\alpha (1-e^{t/\tau})</math>, where <math>\tau \approx 83</math> days was the best fit.

Adversarial attacks

2020-02-05T11:42:07Z

Lê Nguyên Hoang: /* Evasion attacks */

Adversarial attacks encompass a large range of users' behaviors trying to hack an algorithm's vulnerabilities for their advantages.

== Evasion attacks ==

An evasion attack is the vulnerability of an algorithm to imperceptible alterations of their inputs. Typically, while the algorithm successfully classifies cat images as such 99.999% of the time, for any cat image, there may be a slight perturbation of the image such that the algorithm no longer classifies the perturbed cat image as a cat image. This vulnerability has become critical to large-scale algorithms, like [[YouTube]]'s paedophilia moderation algorithm [https://www.wired.co.uk/article/youtube-pedophile-videos-advertising Wired19].

[https://arxiv.org/pdf/1412.6572.pdf GSS][https://dblp.org/rec/bibtex/journals/corr/GoodfellowSS14 14] highlighted the vulnerabilities of state-of-the-art machine learning algorithms to evasion attacks, with an example that has since become iconic.

[https://arxiv.org/pdf/1907.05418.pdf CXYFY][https://dblp.org/rec/bibtex/journals/corr/abs-1907-05418 19]

In February 2020, an artist caused a traffic jam by crossing a bridge with a trolley filled with 99 phones, thereby making Google Maps believe that there was a jam on a bridge and redirecting numerous drivers [https://techbriefly.com/2020/02/03/an-artist-created-fake-traffic-jams-on-google-maps-using-99-phones/amp/ TechBriefly20].

== Poisoning attacks ==

Poisoning attacks consist in contaminating a machine learning algorithm's training data. [[Robust statistics]] consists of developing learning algorithms that successfully learn from poisoned datasets, hopefully nearly as well as if the datasets were not poisoned in the first place. There have been remarkable recent progress in this domain [https://arxiv.org/pdf/1911.05911.pdf DiakonikolasKane][https://dblp.org/rec/bibtex/journals/corr/abs-1911-05911 19] [https://arxiv.org/pdf/1906.03058 DepersinLecué][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Robust+subgaussian+estimation+of+a+mean+vector+in+nearly+linear+time&btnG= 19] [http://papers.nips.cc/paper/6617-machine-learning-with-adversaries-byzantine-tolerant-gradient-descent.pdf BEGS][https://dblp.org/rec/bibtex/conf/nips/BlanchardMGS17 17] [https://www.youtube.com/watch?v=QguWgfGsG-k RB2].

== Astroturfing attacks ==

Astroturfing attacks and SEO-optimization exploit vulnerabilities of recommender systems to promote specific contents, for instance by creating fake accounts or exploiting compromised accounts to tweet hashtags (and immediately erase the tweet to prevent detection) [https://arxiv.org/pdf/1910.07783.pdf EOOR][https://dblp.org/rec/bibtex/journals/corr/abs-1910-07783 19].

Church-Turing thesis

2020-02-04T16:03:19Z

Lê Nguyên Hoang:

The Church-Turing thesis claims that the universal Turing machine is the most general form of computation [https://www.youtube.com/watch?v=PLVCscCY4xI UpAndAtom20].

== The different versions of the thesis ==

There are several slightly distinct versions of this thesis.

Perhaps the most important one is to claim that it is the most general form of computation of any machine that can be built in our universe. In particular, this would mean that the halting problem cannot be solved in our space-time.

The computational complexity Church-Turing thesis is widely believed to be wrong, because of quantum mechanics. But it can be upgraded to a quantum complexity Church-Turing thesis.

== Justifications from physics ==

Gandy theorem.

Bekenstein bound.

[https://arxiv.org/pdf/quant-ph/0110141.pdf Lloyd][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Computational+capacity+of+the+universe+lloyd&btnG= 02] estimate computational capacity measures of the universe.

== Implications ==

It's all about information and information processing!

No "fundamental law" of physics. Any universal Turing machine is.

Nothing special about intelligence.

Computational moral philosophy.

Preference learning from comparisons

2020-02-04T14:21:56Z

Lê Nguyên Hoang: /* Gaussian process */

It has been argued that we humans are much more effective at comparing alternatives than at scoring them [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18]. Besides, implicit observations of humans mostly provide such choice data, in the form of like/no-like, share/no-share, click/no-click, and so on. Therefore, preference learning from comparisons seems to be an important approach to preference learning.

== Classical models ==

The classical models for preference learning are due to [http://mlab.no/blog/wp-content/uploads/2009/07/thurstone94law.pdf Thurston][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+law+of+comparative+judgment+thurstone&btnG= 27], [https://idp.springer.com/authorize/casa?redirect_uri=https://link.springer.com/content/pdf/10.1007/BF01180541.pdf&casa_token=dMSQZQpkjt4AAAAA:4HFh3rzi3S6VqH29Pg00S64qMuFB9b6VpF8pRikHmTNWf3REi10i_dj9_Eps3Ivp6l8c-ZETTU-Cx_-L6g Zermelo][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Die+Berechnung+der+Turnier-Ergebnisse+als+ein+Maximumproblem+der+Wahrscheinlichkeitsrechnung.&btnG= 29], [https://link.springer.com/content/pdf/10.1007/BF02289115.pdf Mosteller][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=mosteller+Remarks+on+the+method+of+paired+comparisons&btnG= 51], [https://www.jstor.org/stable/pdf/2334029.pdf?casa_token=Q0ROYjxwOZEAAAAA:fuuqbCntxaNmZB0hlKk8iXfzOkLzQs9H677gh2UOQdMI4496B3YZHlUaiuMgD9zONaHYLHgtSmjMHziL29cx9d9tv2f_QwyqN0wxYUJrExI_eK8QR7sa BradleyTerry][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Rank+analysis+of+incomplete+block+designs%3A+I.+The+method+of+paired+comparisons&btnG= 52], [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Individual+Choice+Behavior%3A+A+Theoretical+Analysis+luce+wiley+1959&btnG= LuceBook59], [https://apps.dtic.mil/dtic/tr/fulltext/u2/a417190.pdf#page=15 DavidBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Method+of+Paired+Comparisons+david&btnG= 63].

In these models, in a choice between 1 and 2, the human implicitly the scores <math>\theta_1</math> and <math>\theta_2</math> that she assigns to each alternative. But his computation of <math>\theta_1 - \theta_2</math> is noisy, and is accompanied with some noise <math>\varepsilon</math>, to yield <math>x_{12} = \theta_1-\theta_2+\varepsilon</math>. In the above models, the sign of <math>x_{12}</math> then determines the choice of the human.

This approach allows to explain some of the inconsistencies in humans' decision-making. Equivalently, this corresponds to saying that the probability that the human chooses 1 over 2 is a function of the intrinsic difference <math>\theta_1-\theta_2</math>, which we can write <math>\mathbb P[1 \succ 2] = \mathbb P[x_{12}>0] = \phi(\theta_1-\theta_2)</math>. Intuitively, the greater the intrinsic difference between 1 and 2, the less likely it is for the human to say he prefers 2 to 1.

Different models assume different noise models for <math>\varepsilon</math>, or, equivalently, for the function <math>\phi</math>. Thurstone's model assumes that <math>\varepsilon</math> is normally distributed, which is equivalent to saying that <math>\phi(z) = \Phi(z/\sigma)</math>, where <math>\sigma^2</math> is the variance of <math>\varepsilon</math> (which may depend on the choice of 1 and 2), and where <math>\Phi</math> is the cumulative density function of the standard normal distribution.

The Bradley-Terry model assumes that <math>\varepsilon</math> follows a [https://en.wikipedia.org/wiki/Gumbel_distribution Gumbel distribution]. Equivalently, it sets <math>\phi(z)= \frac{1}{1+\exp(-z)}</math> (up to variance scaling). Luce's model generalizes the Bradley-Terry models, by considering choices among numerous alternatives and setting <math>\phi(z_2, z_3, ...)= \frac{1}{1+\exp(-z_2)+\exp(-z_3)+...}</math>, where this quantity is the probability of choosing option 1, and where <math>z_i = \theta_1-\theta_i</math>. Interestingly, Luce proved that this framework was equivalent to demanding independence of irrelevant alternatives.

== Inference ==

The inference problem is then the problem of inferring the values of parameters <math>\theta_i</math> given observational data <math>\mathcal D</math> of choices <math>i</math> out of a set of alternatives <math>\mathcal A</math>. Bayesian inference would suggest computing <math>\mathbb P[\theta|\mathcal D]</math>. But as often, this approach might be too computationally costly in practice. Approximate Bayesian methods are needed.

Note that in the Luce model, the log-likelihood <math>\log \mathbb P[\mathcal D|\theta] = \sum \left( \theta_i - \log \sum_{(j \succ i) \in \mathcal D} \exp \theta_j \right)</math> is a strictly concave function in <math>\theta</math> if the comparison graph is strongly connected (for any <math>i,j</math>, there exists <math>k_1, ... k_m</math> such that there are some of the data that says <math>i \succ k_1 \succ k_2 \succ ... \succ k_m \succ j</math>). This proves the existence and uniqueness of the maximum likelihood estimator under this assumption.

== Markov chain for preference learning ==

[http://papers.nips.cc/paper/4701-iterative-ranking-from-pair-wise-comparisons.pdf NOS][https://dblp.org/rec/bibtex/conf/nips/NegahbanOS12 12] proposed a Markov chain approach to compute the maximum likelihood estimator, in the same vein as the PageRank algorithm [http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf PBMW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+PageRank+citation+ranking%3A+Bringing+order+to+the+Web+page+brin+motwani+winograd&btnG= 99]. Namely, by choosing adequately the transition rates of the comparison graph, the stationary distribution of the Markov chain thereby constructed turns out to equal the vector <math>(\exp \theta_i^\star)</math>, where <math>\theta^\star</math> is the maximum likelihood estimator.

Note however, that this approach is too data and computationally expensive if the set of alternatives is combinatorial. In fact, for most recommender systems, it may be inapplicable, since most users have never even been exposed to most of the video or music contents of platforms like YouTube or Spotify.

== Gaussian process ==

Perhaps one of the most promising avenues to scalable comparison-based preference learning consists of building upon the assumption that, a priori, the intrinsic preference <math>\theta_1-\theta_2</math> between two alternatives <math>z_1</math> and <math>z_2</math> is a Gaussian process. By then invoking the similarity between having to choose between <math>x = (z_1,z_2)</math> and <math>x' = (z_1',z_2')</math>, we can then generalize from observed data by means of a [[kernel method]].

To implement this approach, we first need a measure of the similarity between two choices <math>x</math> and <math>x'</math>. Intuitively, <math>x</math> and <math>x'</math> will be similar if both <math>z_1 \approx z_1'</math> and <math>z_2 \approx z_2'</math>. Note that <math>x</math> and <math>x'</math> can also be "anti-similar", if <math>z_1 \approx z_2'</math> and <math>z_2 \approx z_1'</math>. Let us call <math>K(x,x')</math> the similarity between <math>x</math> and </math>x'</math>. Gaussian process prior and Bayesian methods then enable to infer a posterior from revealed preferences on the preference for other dilemmas [https://dl.acm.org/doi/pdf/10.1145/1102351.1102369?download=true ChuGhahramani][https://dblp.org/rec/bibtex/conf/icml/ChuG05 05a] [http://www.jmlr.org/papers/volume6/chu05a/chu05a.pdf ChuGharamani][https://dblp.org/rec/bibtex/journals/jmlr/ChuG05 05b].

Note that such kernels can be approximately implemented by [[representational learning|vector representation]], for instance by some deep neural network <math>f</math>. We could then have <math>K(x,x') = f(x)^Tf(x')</math> (anti-similarity could be naturally enforced if <math>f(z_1,z_2) = - f(z_2,z_1)</math>, which is guaranteed if <math>f(z_1,z_2) = g(z_1)-g(z_2)</math> for some other vector representation <math>g</math>). The use of such vector representations could be useful to accelerate computation, as opposed to the kernel method which requires going through all the training data to make a prediction.

== Connection to sports ==

Chess [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Rating+Of+ChessPlayers+past+and+present+Players+elo+arco&btnG= Elo78], football [http://ceur-ws.org/Vol-1842/paper_04.pdf MKFG][https://dblp.org/rec/bibtex/conf/pkdd/MaystreKFG16 16].

Preference learning from comparisons

2020-02-04T13:02:51Z

Lê Nguyên Hoang: /* Classical models */

It has been argued that we humans are much more effective at comparing alternatives than at scoring them [https://infoscience.epfl.ch/record/255399/files/EPFL_TH8637.pdf MaystrePhD][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Efficient+Learning+from+Comparisons+maystre&btnG= 18]. Besides, implicit observations of humans mostly provide such choice data, in the form of like/no-like, share/no-share, click/no-click, and so on. Therefore, preference learning from comparisons seems to be an important approach to preference learning.

== Classical models ==

The classical models for preference learning are due to [http://mlab.no/blog/wp-content/uploads/2009/07/thurstone94law.pdf Thurston][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=A+law+of+comparative+judgment+thurstone&btnG= 27], [https://idp.springer.com/authorize/casa?redirect_uri=https://link.springer.com/content/pdf/10.1007/BF01180541.pdf&casa_token=dMSQZQpkjt4AAAAA:4HFh3rzi3S6VqH29Pg00S64qMuFB9b6VpF8pRikHmTNWf3REi10i_dj9_Eps3Ivp6l8c-ZETTU-Cx_-L6g Zermelo][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Die+Berechnung+der+Turnier-Ergebnisse+als+ein+Maximumproblem+der+Wahrscheinlichkeitsrechnung.&btnG= 29], [https://link.springer.com/content/pdf/10.1007/BF02289115.pdf Mosteller][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=mosteller+Remarks+on+the+method+of+paired+comparisons&btnG= 51], [https://www.jstor.org/stable/pdf/2334029.pdf?casa_token=Q0ROYjxwOZEAAAAA:fuuqbCntxaNmZB0hlKk8iXfzOkLzQs9H677gh2UOQdMI4496B3YZHlUaiuMgD9zONaHYLHgtSmjMHziL29cx9d9tv2f_QwyqN0wxYUJrExI_eK8QR7sa BradleyTerry][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Rank+analysis+of+incomplete+block+designs%3A+I.+The+method+of+paired+comparisons&btnG= 52], [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=Individual+Choice+Behavior%3A+A+Theoretical+Analysis+luce+wiley+1959&btnG= LuceBook59], [https://apps.dtic.mil/dtic/tr/fulltext/u2/a417190.pdf#page=15 DavidBook][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Method+of+Paired+Comparisons+david&btnG= 63].

In these models, in a choice between 1 and 2, the human implicitly the scores <math>\theta_1</math> and <math>\theta_2</math> that she assigns to each alternative. But his computation of <math>\theta_1 - \theta_2</math> is noisy, and is accompanied with some noise <math>\varepsilon</math>, to yield <math>x_{12} = \theta_1-\theta_2+\varepsilon</math>. In the above models, the sign of <math>x_{12}</math> then determines the choice of the human.

This approach allows to explain some of the inconsistencies in humans' decision-making. Equivalently, this corresponds to saying that the probability that the human chooses 1 over 2 is a function of the intrinsic difference <math>\theta_1-\theta_2</math>, which we can write <math>\mathbb P[1 \succ 2] = \mathbb P[x_{12}>0] = \phi(\theta_1-\theta_2)</math>. Intuitively, the greater the intrinsic difference between 1 and 2, the less likely it is for the human to say he prefers 2 to 1.

Different models assume different noise models for <math>\varepsilon</math>, or, equivalently, for the function <math>\phi</math>. Thurstone's model assumes that <math>\varepsilon</math> is normally distributed, which is equivalent to saying that <math>\phi(z) = \Phi(z/\sigma)</math>, where <math>\sigma^2</math> is the variance of <math>\varepsilon</math> (which may depend on the choice of 1 and 2), and where <math>\Phi</math> is the cumulative density function of the standard normal distribution.

The Bradley-Terry model assumes that <math>\varepsilon</math> follows a [https://en.wikipedia.org/wiki/Gumbel_distribution Gumbel distribution]. Equivalently, it sets <math>\phi(z)= \frac{1}{1+\exp(-z)}</math> (up to variance scaling). Luce's model generalizes the Bradley-Terry models, by considering choices among numerous alternatives and setting <math>\phi(z_2, z_3, ...)= \frac{1}{1+\exp(-z_2)+\exp(-z_3)+...}</math>, where this quantity is the probability of choosing option 1, and where <math>z_i = \theta_1-\theta_i</math>. Interestingly, Luce proved that this framework was equivalent to demanding independence of irrelevant alternatives.

== Inference ==

The inference problem is then the problem of inferring the values of parameters <math>\theta_i</math> given observational data <math>\mathcal D</math> of choices <math>i</math> out of a set of alternatives <math>\mathcal A</math>. Bayesian inference would suggest computing <math>\mathbb P[\theta|\mathcal D]</math>. But as often, this approach might be too computationally costly in practice. Approximate Bayesian methods are needed.

Note that in the Luce model, the log-likelihood <math>\log \mathbb P[\mathcal D|\theta] = \sum \left( \theta_i - \log \sum_{(j \succ i) \in \mathcal D} \exp \theta_j \right)</math> is a strictly concave function in <math>\theta</math> if the comparison graph is strongly connected (for any <math>i,j</math>, there exists <math>k_1, ... k_m</math> such that there are some of the data that says <math>i \succ k_1 \succ k_2 \succ ... \succ k_m \succ j</math>). This proves the existence and uniqueness of the maximum likelihood estimator under this assumption.

== Markov chain for preference learning ==

[http://papers.nips.cc/paper/4701-iterative-ranking-from-pair-wise-comparisons.pdf NOS][https://dblp.org/rec/bibtex/conf/nips/NegahbanOS12 12] proposed a Markov chain approach to compute the maximum likelihood estimator, in the same vein as the PageRank algorithm [http://ilpubs.stanford.edu:8090/422/1/1999-66.pdf PBMW][https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+PageRank+citation+ranking%3A+Bringing+order+to+the+Web+page+brin+motwani+winograd&btnG= 99]. Namely, by choosing adequately the transition rates of the comparison graph, the stationary distribution of the Markov chain thereby constructed turns out to equal the vector <math>(\exp \theta_i^\star)</math>, where <math>\theta^\star</math> is the maximum likelihood estimator.

Note however, that this approach is too data and computationally expensive if the set of alternatives is combinatorial. In fact, for most recommender systems, it may be inapplicable, since most users have never even been exposed to most of the video or music contents of platforms like YouTube or Spotify.

== Gaussian process ==

Perhaps one of the most promising avenues to scalable comparison-based preference learning consists of building upon the assumption that, a priori, the intrinsic preference <math>\theta_1-\theta_2</math> between two alternatives <math>z_1</math> and <math>z_2</math> is a Gaussian process. By then invoking the similarity between having to choose between <math>x = (z_1,z_2)</math> and <math>x' = (z_1',z_2')</math>, we can then generalize from observed data by means of a [[kernel method]].

To implement this approach, we first need a measure of the similarity between two choices <math>x</math> and <math>x'</math>. Intuitively, <math>x</math> and <math>x'</math> will be similar if both <math>z_1 \approx z_1'</math> and <math>z_2 \approx z_2'</math>. Note that <math>x</math> and <math>x'</math> can also be "anti-similar", if <math>z_1 \approx z_2'</math> and <math>z_2 \approx z_1'</math>. Let us call <math>K(x,x')</math> this property.

Note that such kernels can be approximately implemented by [[representational learning|vector representation]], for instance by some deep neural network <math>f</math>. We could then have <math>K(x,x') = f(x)^Tf(x')</math> (anti-similarity could be naturally enforced if <math>f(z_1,z_2) = - f(z_2,z_1)</math>, which is guaranteed if <math>f(z_1,z_2) = g(z_1)-g(z_2)</math> for some other vector representation <math>g</math>). The use of such vector representations could be useful to accelerate computation, as opposed to the kernel method which requires going through all the training data to make a prediction.

NOTE: This section needs references.

== Connection to sports ==

Chess [https://scholar.google.ch/scholar?hl=en&as_sdt=0%2C5&q=The+Rating+Of+ChessPlayers+past+and+present+Players+elo+arco&btnG= Elo78], football [http://ceur-ws.org/Vol-1842/paper_04.pdf MKFG][https://dblp.org/rec/bibtex/conf/pkdd/MaystreKFG16 16].