This post consists of rough thoughts on possible training processes that select for multi-agent architectures, or “subagents”, in neural networks. I deliberately ignore delegative subagents that are instantiated by mesa-optimizers and instead focus on subagent architectures that arise due to “non-agentic” training processes alone.

TL;DR: No; there is no directed agency that enforces sexual selection through an exploitable proxy. However, Fisherian runaway is an insightful example of the path-dependence of local search, where an easily acquired and apparently useful proxy goal can be so strongly favored that disadvantageous traits emerge as side effects.

Many people who self-identify as Effective Altruists or are otherwise intensely committed to their work seem to struggle with prioritising their health. For the purposes of this article, I will adopt the following definition of ‘health’: ‘the ability to adapt and self-manage in the face of social, physical, and emotional challenges’. I want to explain what I think is the best argument for why those people should by default invest ~10-20% of their time in highly evidence-backed health-promoting activities. I don’t claim that this argument is entirely original and I apologise if I forget to cite some sources that inspired me. In short, engaging in health-promoting activities likely induces:

• Better quality of life;
• Resilience to burn-out and adverse life events;
• Increased productivity; and
• Positive impact on one’s immediate social circle.

TL;DR: If we want to understand how AI models make decisions, and thus assess alignment, we generally want to extract their internal latent information. That information might be inaccessible to us by default and defy easy extraction. Resolving this problem might be critical to AI alignment.

A perpetually updating list of my favourite resources for a variety of goals:

To illustrate why choices that lead to extinction events are potentially special among ethical problems, we consider the following thought experiment. Amy has been given three options:

Low back pain (LBP) is an extremely common and debilitating symptom worldwide, experienced by people of all ages and socioeconomic circumstance [1]. In 2017, LBP contributed 65 million disability-adjusted life-years and was the primary cause of work absenteeism and disability worldwide [2]. Most LBP cannot be attributed to a specific source and bouts, while typically short, frequently reoccur [1].

The realisation of artificial general intelligence (AGI), human-built models that can accomplish most human tasks, would be an event of tremendous significance for the future of humanity and all earthly life. AGI systems might possess capabilities vastly exceeding humans and radically different goals, destabilise international power structures, and alter society fundamentally by populating the world with digital minds. If AGI is technologically feasible, there is arguably a good chance that it will be realised within the next 50 years, potentially making this the most important century in the lifespan of humanity.