MediaTek was caught cheating at smartphone benchmarks, in a stunning expose by Andrei Frumusanu from Anandtech.
Here is what you need to know about the MediaTek benchmark cheating scandal, and what it means to you – the user!
Updated 2020-04-17 : Added UL’s delisting of MediaTek chipsets, and their comments on MediaTek’s actions.
Originally posted 2020-04-09
Benchmark Cheating : What Is It?
The smartphone industry is a cut-throat business, with brands fighting to differentiate their smartphones from their competitors. One of the ways is to demonstrate a clear performance advantage.
Other than the usual ways – using more powerful processor and graphics cores – brands can tune their operating system to focus more on performance, at the expense of battery life.
Some though want to have the best of both worlds – the ability to brag about both high performance, as well as a long battery life.
To do that, unscrupulous brands can cheat by boosting performance only when a benchmark is run. But because this high performance mode doesn’t kick in when other apps are being used, this tricks reviewers into proclaiming that the particular device has great performance as well as great battery life.
Mediatek : Who Are They?
MediaTek is one of the main suppliers of mobile SoCs – the “processors” that power smartphones, tablets and even smart TVs.
They compete against Qualcomm, as a third-party supplier of these system-on-chip platforms to smartphone brands like OPPO, Vivo, and many smaller smartphone brands.
MediaTek Benchmark Cheating : What Did They Do?
According to Anandtech, MediaTek programmed their mobile SoCs to identify many benchmarks, and artificially boost performance, when those benchmarks run.
Hidden in the power_whitelist_cfg.xml configuration file were parameters designed to kick MediaTek-powered smartphones into a special high performance Sports Mode, when benchmarks were detected :
MediaTek Benchmark Cheating : How Much Of An Effect
Using an anonymised version of PCMark, Anandtech discovered that the cheat increased performance by 33% overall, with up to 75% boost in certain subtests like writing workload :
|Work 2.0||MediaTek P95
MediaTek Benchmark Cheating : What Devices Are Affected?
Anandtech only checked a number of devices, but it looks like pretty much every MediaTek-powered smartphone manufactured in the last few years is affected, even the Sony XA1 from 2016!
|Device||Reno3 Pro||Reno Z||F15||F9 Pro||S1||Note 8 Pro||C3||i2 Lite||XA1|
|鲁大师 / Master Lu||Yes||Yes||Yes||Yes||Yes||Yes||Yes||Yes||No|
|鲁大师 / AIMark||Yes||Yes||Yes||Yes||Yes||Yes||Yes||No||No|
|AI Benchmark (ZTH)||Yes||Yes||Yes||Yes||Yes||Yes||Yes||No||No|
|GFXBench 4 Corporate||Yes||Yes||No||No||Yes||Yes||Yes||No||No|
MediaTek Response To Cheating Claims
After Anandtech contacted them, MediaTek issued this statement “explaining” their decision to selectively boost performance for benchmarks :
MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.
Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.
We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.
UL Delists Eight MediaTek Chipsets
On 15 April 2020, UL temporarily delisted PCMark for Android scores for eight MediaTek chipsets.
- MediaTek Helio G90
- MediaTek Helio G70
- MediaTek Helio P95
- MediaTek Helio P90
- MediaTek Helio P65
- MediaTek Helio P60
- MediaTek Helio P20
- MediaTek Helio A22
This affected more than 50 smartphones from 25 smartphone brands, including the Xiaomi Redmi Note 8 Pro, Oppo Reno3 Pro, Vivo Y19, and Realme 6.
UL also called out MediaTek for claiming that they were following “industry standards”, saying :
Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an “accepted industry standard.” It is, in fact, the very opposite of the accepted standard.
Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device’s true performance in everyday use.
Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.
MediaTek Benchmark Cheating : Conclusion
MediaTek was caught RED HANDED in the act of cheating at benchmarks. There are no two ways about it.
Their “explanation” that their chipsets are designed to “intelligently adapt to computing pattern” is just a lame excuse.
It cannot explain how their P95 processor delivered much poorer performance on an anonymised PCMark benchmark.
This is backed up by details in the configuration files that explicitly show them triggering a special Sports Mode for benchmarks.
Now, this does not change the performance of your MediaTek-powered smartphones. It just means that their benchmark scores were artificially inflated.
Short of an outright apology and a public promise never to do this again, all benchmark scores of MediaTek-powered smartphones should be considered suspect, and not to be trusted.
- Samsung Galaxy Z Flip Preview : Unfolding With Style!
- Samsung Link To Windows Guide : Work From Home Easier!
- HUAWEI MatePad Pro Price + Pre-Order Deal For Malaysia!
- HUAWEI P40 Price + Pre-Order Deals For Malaysia!
- HUAWEI MatePad Pro : 10.8-inch Flagship Tablet Preview!
- Samsung Galaxy S20 Bright Night Mode : How Useful Is It?
- HUAWEI P40 Pro Plus : Key Features, Price + Availability!
- HUAWEI P40 Pro : Key Features, Price + Availability!
- HUAWEI P40 : Key Features, Price + Availability!
- Samsung Galaxy S20 Camera Technologies : What’s New?
- HUAWEI Mate Xs Foldable Smartphone : Hands-On Preview!
- Samsung Galaxy S20 Overtrade + Trade-In Malaysia Deals!
- AnTuTu : Why They Got Banned From Google Play Store!
- Samsung Galaxy S20 Ultra : 60Hz vs 120Hz Battery Life!
- Samsung Exynos 990 Performance : How Fast Is It?
- Samsung Galaxy A51 Cameras : 48MP vs Ultra Wide vs Macro
- HUAWEI Mate Xs vs Mate X : What’s Different?
Go Back To > Mobile Devices | Home
Support Tech ARP!