A Multi-Dimensional Evaluation of Explainability in Media Bias Detection

Bias-detection models can flag a biased news sentence, but that alone doesn't tell you whether the model is reasoning the way a person would, or whether its stated reasons are what's actually driving its answer. This paper tests BERT and RoBERTa on media bias detection along three separate axes: how accurate they are, whether their highlighted words match what human experts flagged, and whether those words are truly what the model relies on internally (tested by surgically altering the model's attention circuits). The three measures often disagree with each other, and pushing a model to highlight the 'right' words doesn't reliably make it rely on them internally. The takeaway for anyone deploying these systems: being accurate, being explainable in a human-relatable way, and being faithful to your own internal reasoning are three different properties, and checking one doesn't tell you about the others.