MediaTek Benchmark Cheating : What Happened So Far

MediaTek was caught cheating at smartphone benchmarks, in a stunning expose by Andrei Frumusanu from Anandtech.

Here is what you need to know about the MediaTek benchmark cheating scandal, and what it means to you – the user!

MediaTek Benchmark Cheating : What You Need To Know!

Updated 2020-04-17 : Added UL’s delisting of MediaTek chipsets, and their comments on MediaTek’s actions.

Originally posted 2020-04-09

 

Benchmark Cheating : What Is It?

The smartphone industry is a cut-throat business, with brands fighting to differentiate their smartphones from their competitors. One of the ways is to demonstrate a clear performance advantage.

Other than the usual ways – using more powerful processor and graphics cores – brands can tune their operating system to focus more on performance, at the expense of battery life.

Some though want to have the best of both worlds – the ability to brag about both high performance, as well as a long battery life.

To do that, unscrupulous brands can cheat by boosting performance only when a benchmark is run. But because this high performance mode doesn’t kick in when other apps are being used, this tricks reviewers into proclaiming that the particular device has great performance as well as great battery life.

MediaTek mobile platform generic

 

Mediatek : Who Are They?

MediaTek is one of the main suppliers of mobile SoCs – the “processors” that power smartphones, tablets and even smart TVs.

They compete against Qualcomm, as a third-party supplier of these system-on-chip platforms to smartphone brands like OPPO, Vivo, and many smaller smartphone brands.

 

MediaTek Benchmark Cheating : What Did They Do?

According to Anandtech, MediaTek programmed their mobile SoCs to identify many benchmarks, and artificially boost performance, when those benchmarks run.

Hidden in the power_whitelist_cfg.xml configuration file were parameters designed to kick MediaTek-powered smartphones into a special high performance Sports Mode, when benchmarks were detected :

MediaTek configuration files
Credit : Anandtech

 

MediaTek Benchmark Cheating : How Much Of An Effect

Using an anonymised version of PCMark, Anandtech discovered that the cheat increased performance by 33% overall, with up to 75% boost in certain subtests like writing workload :

Work 2.0 MediaTek P95
(Real Performance)
MediaTek P95
(Sports Mode)
Difference
Performance Score
(Average)
6781 9048 +33.4%
Web Browsing 5484 9011 +64.3%
Video Editing 5306 5950 +12.1%
Writing 5734 10034 +75.0%
Photo Editing 14896 18804 +26.2%
Data Manipulation 5767 5995 +4.0%

 

MediaTek Benchmark Cheating : What Devices Are Affected?

Anandtech only checked a number of devices, but it looks like pretty much every MediaTek-powered smartphone manufactured in the last few years is affected, even the Sony XA1 from 2016!

Brand OPPO OPPO OPPO OPPO Vivo Xiaomi Realme iVoomi Sony
Device Reno3 Pro Reno Z F15 F9 Pro S1 Note 8 Pro C3 i2 Lite XA1
MediaTek SoC P95 P90 P70 P60 P65 G90 G70 A22 P20
AndroBench2 Yes Yes Yes Yes Yes Yes Yes Yes Yes
PCMark Yes Yes Yes Yes Yes Yes Yes Yes Yes
AnTuTu Yes Yes Yes Yes Yes Yes Yes Yes Yes
GeekBench Yes Yes Yes Yes Yes Yes Yes Yes Yes
Quadrant Yes Yes Yes Yes Yes Yes Yes Yes Yes
Quadrant Professional Yes Yes Yes Yes Yes Yes Yes Yes Yes
鲁大师 / Master Lu Yes Yes Yes Yes Yes Yes Yes Yes No
鲁大师 / AIMark Yes Yes Yes Yes Yes Yes Yes No No
AI Benchmark (ZTH) Yes Yes Yes Yes Yes Yes Yes No No
NeuralScope Benchmark Yes Yes Yes Yes Yes Yes Yes No No
GFXBench 4 Corporate Yes Yes No No Yes Yes Yes No No

 

MediaTek Response To Cheating Claims

After Anandtech contacted them, MediaTek issued this statement “explaining” their decision to selectively boost performance for benchmarks :

MediaTek follows accepted industry standards and is confident that benchmarking tests accurately represent the capabilities of our chipsets. We work closely with global device makers when it comes to testing and benchmarking devices powered by our chipsets, but ultimately brands have the flexibility to configure their own devices as they see fit. Many companies design devices to run on the highest possible performance levels when benchmarking tests are running in order to show the full capabilities of the chipset. This reveals what the upper end of performance capabilities are on any given chipset.

Of course, in real world scenarios there are a multitude of factors that will determine how chipsets perform. MediaTek’s chipsets are designed to optimize power and performance to provide the best user experience possible while maximizing battery life. If someone is running a compute-intensive program like a demanding game, the chipset will intelligently adapt to computing patterns to deliver sustained performance. This means that a user will see different levels of performance from different apps as the chipset dynamically manages the CPU, GPU and memory resources according to the power and performance that is required for a great user experience. Additionally, some brands have different types of modes turned on in different regions so device performance can vary based on regional market requirements.

We believe that showcasing the full capabilities of a chipset in benchmarking tests is in line with the practices of other companies and gives consumers an accurate picture of device performance.

 

UL Delists Eight MediaTek Chipsets

On 15 April 2020, UL temporarily delisted PCMark for Android scores for eight MediaTek chipsets.

  • MediaTek Helio G90
  • MediaTek Helio G70
  • MediaTek Helio P95
  • MediaTek Helio P90
  • MediaTek Helio P65
  • MediaTek Helio P60
  • MediaTek Helio P20
  • MediaTek Helio A22

This affected more than 50 smartphones from 25 smartphone brands, including the Xiaomi Redmi Note 8 Pro, Oppo Reno3 Pro, Vivo Y19, and Realme 6.

UL also called out MediaTek for claiming that they were following “industry standards”, saying :

Using hidden mechanisms to detect benchmarking apps by name and make app-specific performance optimizations is not an “accepted industry standard.” It is, in fact, the very opposite of the accepted standard.

Likewise, benchmark scores based on hidden app-specific optimizations and settings that are enabled by default and not available to the user do not accurately reflect a device’s true performance in everyday use. 

Simply put, a device must run a benchmark as if it was any other application. Performance gains must come from reacting to the nature of the workloads in the test rather than the name of the app itself.

 

MediaTek Benchmark Cheating : Conclusion

MediaTek was caught RED HANDED in the act of cheating at benchmarks. There are no two ways about it.

Their “explanation” that their chipsets are designed to “intelligently adapt to computing pattern” is just a lame excuse.

It cannot explain how their P95 processor delivered much poorer performance on an anonymised PCMark benchmark.

This is backed up by details in the configuration files that explicitly show them triggering a special Sports Mode for benchmarks.

Now, this does not change the performance of your MediaTek-powered smartphones. It just means that their benchmark scores were artificially inflated.

Short of an outright apology and a public promise never to do this again, all benchmark scores of MediaTek-powered smartphones should be considered suspect, and not to be trusted.

 

Recommended Reading

Go Back To > Mobile Devices | Home

 

Support Tech ARP!

If you like our work, you can help support our work by visiting our sponsors, participating in the Tech ARP Forums, or even donating to our fund. Any help you can render is greatly appreciated!


Comments

comments

About The Author

Leave a Reply

%d bloggers like this: