A Descriptive and Normative Theory of Human Beliefs in RLHF
Updated
Updated · arxiv.org · Jul 10
A Descriptive and Normative Theory of Human Beliefs in RLHF
1 articles · Updated · arxiv.org · Jul 10
Summary
Researchers have proposed a new theory highlighting the impact of human beliefs about AI agent capabilities on preference generation in RLHF systems.
A study shows that aligning human labelers’ beliefs with actual agent abilities improves policy performance, with empirical evidence from synthetic and human experiments.
The findings suggest best practices for RLHF practitioners, such as priming labelers with realistic agent demonstrations to reduce belief mismatches and enhance AI alignment.