Do People Think Fast or Slow When Training AI?

Artificial intelligence (AI) plays a crucial role in decision making. In doing so, it often learns to make choices from human behavior, assuming that people provide unbiased training data. However, studies show that people change their behavior when they are aware they are training AI. It remains unknown whether these modifications are intuitive, driven by social norms, or deliberate, aimed at maximizing personal gain. Across three experiments, we investigated the extent people deliberate when training AI using the ultimatum game. In this game, participants decided whether to accept monetary rewards. Some participants were informed they would train an AI to respond to their or other participants' proposals made in a follow-up session, while others were not. Those training AI could intuitively reject unfair offers or deliberately accept them to maximize current and future rewards. We found that participants rejected unfair offers, suggesting they were more inclined to rely on intuition when training AI. This reveals that people often embed their biases into AI, posing a challenge for AI designed to make optimal decisions.