MediaTek Benchmark Cheating : What Happened So Far

Spread the love

MediaTek was caught cheating at smartphone benchmarks, in a stunning expose by Andrei Frumusanu from Anandtech.

Here is what you need to know about the MediaTek benchmark cheating scandal, and what it means to you – the user!

MediaTek Benchmark Cheating : What You Need To Know!

Updated 2020-04-17 : Added UL’s delisting of MediaTek chipsets, and their comments on MediaTek’s actions.

Originally posted 2020-04-09

 

Benchmark Cheating : What Is It?

The smartphone industry is a cut-throat business, with brands fighting to differentiate their smartphones from their competitors. One of the ways is to demonstrate a clear performance advantage.

Other than the usual ways – using more powerful processor and graphics cores – brands can tune their operating system to focus more on performance, at the expense of battery life.

Some though want to have the best of both worlds – the ability to brag about both high performance, as well as a long battery life.

To do that, unscrupulous brands can cheat by boosting performance only when a benchmark is run. But because this high performance mode doesn’t kick in when other apps are being used, this tricks reviewers into proclaiming that the particular device has great performance as well as great battery life.

MediaTek mobile platform generic

 

Mediatek : Who Are They?

MediaTek is one of the main suppliers of mobile SoCs – the “processors” that power smartphones, tablets and even smart TVs.

They compete against Qualcomm, as a third-party supplier of these system-on-chip platforms to smartphone brands like OPPO, Vivo, and many smaller smartphone brands.

 

MediaTek Benchmark Cheating : What Did They Do?

According to Anandtech, MediaTek programmed their mobile SoCs to identify many benchmarks, and artificially boost performance, when those benchmarks run.

Hidden in the power_whitelist_cfg.xml configuration file were parameters designed to kick MediaTek-powered smartphones into a special high performance Sports Mode, when benchmarks were detected :

MediaTek configuration files
Credit : Anandtech

 

MediaTek Benchmark Cheating : How Much Of An Effect

Using an anonymised version of PCMark, Anandtech discovered that the cheat increased performance by 33% overall, with up to 75% boost in certain subtests like writing workload :

Work 2.0MediaTek P95
(Real Performance)
MediaTek P95
(Sports Mode)
Difference
Performance Score
(Average)
67819048+33.4%
Web Browsing54849011+64.3%
Video Editing53065950+12.1%
Writing573410034+75.0%
Photo Editing1489618804+26.2%
Data Manipulation57675995+4.0%

 

MediaTek Benchmark Cheating : What Devices Are Affected?

Anandtech only checked a number of devices, but it looks like pretty much every MediaTek-powered smartphone manufactured in the last few years is affected, even the Sony XA1 from 2016!

BrandOPPOOPPOOPPOOPPOVivoXiaomiRealmeiVoomiSony
DeviceReno3 ProReno ZF15F9 ProS1Note 8 ProC3i2 LiteXA1
MediaTek SoCP95P90P70P60P65G90G70A22P20
AndroBench2YesYesYesYesYesYesYesYesYes
PCMarkYesYesYesYesYesYesYesYesYes
AnTuTuYesYesYesYesYesYesYesYesYes
GeekBenchYesYesYesYesYesYesYesYesYes
QuadrantYesYesYesYesYesYesYesYesYes
Quadrant ProfessionalYesYesYesYesYesYesYesYesYes
鲁大师 / Master LuYesYesYesYesYesYesYesYesNo
鲁大师 / AIMarkYesYesYesYesYesYesYesNoNo
AI Benchmark (ZTH)YesYesYesYesYesYesYesNoNo
NeuralScope BenchmarkYesYesYesYesYesYesYesNoNo
GFXBench 4 CorporateYesYesNoNoYesYesYesNoNo

 

MediaTek Response To Cheating Claims

After Anandtech contacted them, MediaTek issued this statement “explaining” their decision to selectively boost performance for benchmarks :

MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.

Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.

We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.

 

UL Delists Eight MediaTek Chipsets

On 15 April 2020, UL temporarily delisted PCMark for Android scores for eight MediaTek chipsets.

  • MediaTek Helio G90
  • MediaTek Helio G70
  • MediaTek Helio P95
  • MediaTek Helio P90
  • MediaTek Helio P65
  • MediaTek Helio P60
  • MediaTek Helio P20
  • MediaTek Helio A22

This affected more than 50 smartphones from 25 smartphone brands, including the Xiaomi Redmi Note 8 Pro, Oppo Reno3 Pro, Vivo Y19, and Realme 6.

UL also called out MediaTek for claiming that they were following “industry standards”, saying :

Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an “accepted industry standard.” It is, in fact, the very opposite of the accepted standard.

Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device’s true performance in everyday use. 

Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.

 

MediaTek Benchmark Cheating : Conclusion

MediaTek was caught RED HANDED in the act of cheating at benchmarks. There are no two ways about it.

Their “explanation” that their chipsets are designed to “intelligently adapt to computing pattern” is just a lame excuse.

It cannot explain how their P95 processor delivered much poorer performance on an anonymised PCMark benchmark.

This is backed up by details in the configuration files that explicitly show them triggering a special Sports Mode for benchmarks.

Now, this does not change the performance of your MediaTek-powered smartphones. It just means that their benchmark scores were artificially inflated.

Short of an outright apology and a public promise never to do this again, all benchmark scores of MediaTek-powered smartphones should be considered suspect, and not to be trusted.

 

Recommended Reading

Go Back To > Mobile Devices | Home

 

Support Tech ARP!

If you like our work, you can help support our work by visiting our sponsors, participating in the Tech ARP Forums, or even donating to our fund. Any help you can render is greatly appreciated!


About The Author

Leave a Reply