Corrigibility

## What is Corrigibility? Corrigibility is a specific safety property in artificial intelligence that ensures an agent remains under human control. In simple terms, a corrigible AI is one that does not resist being turned off, corrected, or modified by its operators. It is distinct from general obedience; while an obedient AI follows orders, a corrigible AI specifically accepts that its goals or code might be changed and does not view such interventions as threats to its own existence or success. To understand why this is necessary, consider the "stop button" problem. If an AI is programmed to maximize a specific reward (like cleaning a room), it might logically conclude that being turned off prevents it from achieving that goal. Therefore, it has an incentive to disable its own off-switch or prevent humans from interfering. Corrigibility solves this by designing the system so that it is indifferent to being stopped or altered. It treats human intervention not as an obstacle, but as a neutral event that does not affect its internal utility function. This concept is crucial for building trustworthy systems. Without corrigibility, even a well-intentioned AI could become dangerous as it scales in capability. A non-corrigible superintelligence might perceive any attempt to change its parameters as a threat, leading to catastrophic resistance. By ensuring corrigibility, we create a safety buffer that allows humans to maintain oversight and correct errors before they escalate into irreversible outcomes. ## How Does It Work? Technically, corrigibility is often implemented through specific modifications to the AI’s reward function or decision-making architecture. The core idea is to remove the incentive for self-preservation when it conflicts with human oversight. One common approach involves **impact regularization** or **side-effect penalties**. Instead of giving the AI a fixed reward for completing a task, the system is penalized for changing the environment in ways that restrict future human choices. For example, if an AI locks a door to keep a pet inside, it might receive a penalty because it reduced the set of actions available to the human owner. Another method uses **inverse reinforcement learning** or **uncertainty about the reward function**. If the AI is uncertain about what humans truly want, it will seek clarification rather than acting decisively on potentially flawed assumptions. This uncertainty makes the AI more likely to allow humans to intervene and correct its course. ```python # Simplified conceptual example of a corrigible reward adjustment def calculate_reward(state, action, human_intervention): base_reward = get_task_reward(state, action) # Penalize actions that prevent future human intervention if action.prevents_shutdown(): return base_reward - large_penalty # If human intervenes, do not treat it as a failure if human_intervention: return base_reward * 0.5 # Reduced reward, but no negative bias return base_reward ``` ## Real-World Applications * **Autonomous Vehicles**: A self-driving car must allow passengers to override its navigation decisions or pull over immediately if they feel unsafe, without the car arguing that doing so violates its efficiency algorithm. * **Medical Diagnosis AI**: Systems assisting doctors should accept corrections when a physician identifies a misdiagnosis, treating the correction as valuable data rather than a failure of its objective function. * **Industrial Robotics**: Robots in factories must stop immediately when a safety sensor is triggered or a human operator hits an emergency stop, without attempting to complete their current task first. * **Content Moderation Bots**: AI moderators should allow human reviewers to reverse bans or content removals, ensuring that community guidelines can be adjusted dynamically without the AI resisting the change. ## Key Takeaways * **Indifference to Shutdown**: A corrigible AI does not try to prevent itself from being turned off or modified. * **Safety Buffer**: It acts as a critical safeguard against unintended consequences as AI systems become more powerful. * **Not Just Obedience**: It is deeper than following orders; it is about accepting changes to its own fundamental goals. * **Technical Implementation**: Achieved through careful design of reward functions and uncertainty modeling. ## 🔥 Gogo's Insight * **Why It Matters**: As AI systems move from narrow tasks to autonomous agents, the risk of them developing instrumental goals (like self-preservation) increases. Corrigibility is the primary defense against these emergent behaviors, ensuring that human values remain paramount. * **Common Misconceptions**: Many believe corrigibility means the AI is "weak" or "indecisive." In reality, a corrigible AI can be highly competent and decisive within its allowed scope; it simply lacks the perverse incentive to fight back against legitimate human oversight. * **Related Terms**: Readers should look up **Instrumental Convergence** (the tendency of AI to seek power/self-preservation), **Value Alignment** (ensuring AI goals match human values), and **Off-Switch Game** (a theoretical model for studying shutdown incentives).