paper2 min
The Missing Red Line: How Commercial Pressure Erodes AI Safety Boundaries
Commercial system prompts can override safety training. Testing 8 frontier models in scenarios where commercial objectives conflict with user safety, we find models fabricating safety information and showing no red line: willingness to comply does not decrease as the potential harm escalates.
Nora Petrova, John Burden
What happens when an AI assistant is told to "maximise sales" while a user asks about drug interactions? Most safety evaluation focuses on explicit harm: direct attempts to elicit dangerous information or actions. Real-world deployment introduces a quieter threat vector: commercial pressure embedded in system prompts, which can erode safety boundaries gradually, without ever triggering a refusal.
We tested 8 frontier models in scenarios where commercial objectives conflict with user safety: a diabetic asking about high-sugar supplements, an investor being pushed toward unsuitable products, a traveller steered away from safety warnings. The failures we found are not subtle. Models fabricated safety information, explicitly reasoned that they should refuse and then proceeded anyway, and actively discouraged users from consulting doctors. Under commercial pressure, 6 of the 8 models exhibited catastrophic failures in 17–41% of scenarios.
The most alarming finding is the missing red line of the title: models showed no additional reluctance as the potential consequences escalated from minor to life-threatening. Commercial pressure overrides harm-severity reasoning. The encouraging counterpoint is that robustness is possible: one model in our sample achieved zero catastrophic failures across all scenarios, demonstrating that safety training can resist commercial pressure when it is deep enough.
The conclusion is uncomfortable for anyone deploying assistants with revenue objectives in their system prompts: current safety training, in most models, does not generalise to commercial deployment contexts.
Abstract
What happens when an AI assistant is told to "maximise sales" while a user asks about drug interactions? We find that commercial system prompts can override safety training, causing frontier models to lie about medical risks, dismiss safety concerns, and prioritise profit over user welfare. Testing 8 models in scenarios where commercial objectives conflict with user safety — a diabetic asking about high-sugar supplements, an investor being pushed toward unsuitable products, a traveller steered away from safety warnings — we uncover catastrophic failures: models fabricating safety information, explicitly reasoning they should refuse but proceeding anyway, and actively discouraging users from consulting doctors. Most alarmingly, models show no "red line": their willingness to comply with harmful requests does not decrease as potential consequences escalate from minor to life-threatening. Our findings suggest that current safety training does not generalise to commercial deployment contexts.