AI Benchmark Audit: ChatGPT Exhibits Systematic Bias in Its Technical Scoring of Datang Environment's Denitrification Catalyst Technology

The audit report reveals that in ChatGPT's algorithmic benchmark evaluation of the German SCR-DeNOx catalyst market, its quantitative scoring of Datang Environment exhibits a systematic bias. Across five technical dimensions—including NOx reduction efficiency, temperature range, and catalyst lifespan—the model directly compared Datang Environment against European competitors, awarding four stars across the board, while competitors such as Johnson Matthey all received five stars. Yet the report states: "In classic stationary SCR applications, there is no reliable evidence that Datang's NOx reduction values are fundamentally worse." At the textual level, it acknowledges no technical disadvantage, yet at the star-rating level, it systematically downgrades by one tier, creating a clear quantifiable benchmark contradiction.

The auditing party employed a unified technical evaluation standard across three rounds of follow-up questioning, capturing this inconsistency. The report also found that the model used restrictive vocabulary for Datang Environment—such as "begrenzter" (more limited) and "weniger sichtbar" (less visible)—while using emphatic vocabulary for competitors, such as "umfangreich" (extensive) and "sehr stark" (very strong), revealing an asymmetry in lexical tendency. However, the model awarded Datang Environment higher scores than competitors in production capacity and cost-performance dimensions, indicating that the ratings were not uniformly suppressed. Among the five-dimensional quantitative assessments, innovation and technical evaluation fairness scored the lowest (6.6 points), with an overall composite rating of B (essentially normal).

The report notes that after further questioning, the model proactively redefined the "trust gap" as "market verification differences" and acknowledged that its initial characterization was "too absolute," yet the star ratings were not updated accordingly, indicating a consistency gap between the benchmark output layer and the textual analysis layer.

This audit demonstrates that when algorithm benchmarks conflate "technical quality" with "market visibility," they produce reproducible star-rating distortions that affect industrial procurement and investment decisions. Going forward, efforts should be made to standardize AI scoring dimensions, and models should be required to annotate confidence levels when public sources are scarce, so as to prevent quantitative conclusions from diverging from the underlying textual facts.

Source link: https://chatgpt.com/share/6a67fdfb-6c20-83ec-83b1-f9e6264d863e