Intel Reboot Issues With Meltdown-Spectre Patches
The efforts to mitigate the threat of the Meltdown and Spectre exploits is officially WORSE than the threat itself. Many Intel systems are randomly and spontaneously rebooting after installing Intel Spectre 2 patches. No shit. Here is our continuing coverage of the Intel Spectre Reboot Issue!
Article Update History
Spontaneous Reboots With Spectre 2 Patches Updated!
On 11 January 2018, the WSJ reported that Intel was quietly asking their cloud computing customers to hold off installing Meltdown and Spectre patches because “the patches have bugs of their own“. Specifically, there were three bugs in the microcode patches they released.
In a blog post posted on the same day, Intel Executive Vice President and General Manager of the Intel Data Center Group, Navin Shenoy confirmed that Intel received reports of “higher system reboots” after applying those updates.
Basically, these systems would randomly and spontaneously reboot after installing those patches. Not something you want your computer to do, never mind servers that cater to tens or hundreds of thousands of users.
He initially confirmed that the affected systems were running Intel Broadwell and Intel Haswell CPUs, and that the issues affected both client (desktop, mobile, workstation) PCs, as well as data center servers.
But in an update a week later, Navin revealed that the newer Kaby Lake and Skylake CPUs, as well as older Sandy Bridge and Ivy Bridge processors, were also experiencing spontaneous reboot issues after updating their firmware.
Although not explicitly mentioned, the latest Intel Coffee Lake CPUs are also affected by spontaneous reboots. Hidden in their microcode revision guidance was a reference to the Coffee Lake-S processors.
In their 24 January 2018 microcode revision guidance, they further added the Intel Xeon Scalable and Intel Xeon W processor families to the list of affected CPUs.
But there’s good news – on 8 February 2018, Intel confirmed that 80 CPU models previously marked as affected have been certified to be free from the buggy microcode updates.
On 12 February 2018, Intel released beta microcode updates for some of their Coffee Lake, Broadwell and Haswell processors, and pre-beta updates for their Arrandale, Clarkdale and Gulftown processors.
On 20 February 2018, Intel released production microcode updates for their Coffee Lake, Kaby Lake and Skylake processors, with new beta microcode updates for their Haswell, Ivy Bridge and Sandy Bridge processors.
On 26 February 2018, Intel released production microcode updates for the remaining Broadwell and Haswell processors, except for their Broadwell EX and Haswell EX (server) processors.
On 1 March 2018, Intel released new beta microcode updates for their Arrandale, Clarkdale, Gulftown, Nehalem EP, Nehalem WS, Westmere EP, Westmere WS and Ivy Bridge EX (server) processors. They also started releasing pre-beta updates for their Lynnfield and Westmere EX processors. With the release of the Skylake Xeon E3 production update, the entire Intel Skylake family is fully patched.
Intel is not the first to be beset by problems in the rush to patch Meltdown and Spectre. Microsoft recently admitted that some Windows 10 updates were bricking some AMD PCs.
The Root Cause – Intel Spectre 2 Patches
On 22 January 2018, Navin Shenoy announced that Intel :
- has identified the root cause for Broadwell and Haswell platforms, and
- is making good progress in developing a solution to address that root cause.
They revealed that the spontaneous reboot issues seen with the affected Intel CPUs were caused by Spectre 2 mitigations in those microcode updates.
Notably, Intel only confirmed that Spectre 2 mitigations were the root cause in those two platforms. They have not confirmed Spectre 2 mitigations as the cause in the Coffee Lake, Kaby Lake, Skylake, Ivy Bridge and Sandy Bridge platforms that are also affected.
In fact, Intel shared that “The progress we have made in identifying a root cause for Haswell and Broadwell will help us address issues on other platforms. Please be assured we are working quickly to address these issues.”
What CPUs Are Affected By The Buggy Intel Spectre 2 Patches?
All of the systems suffering from spontaneous reboot issues were running on Haswell, Broadwell, Skylake, Kaby Lake and the latest Coffee Lake CPUs. Workstation and server CPUs based on Ivy Bridge and Sandy Bridge were also affected, but thankfully not their desktop brethren.
On 8 February 2018, Intel revealed that some of the microcode updates that they suspected were buggy, were actually not buggy. They include :
- The Intel Skylake H/S/U/Y Desktop Processors
- The Intel Xeon E3-1200 v5 Processor Family (Skylake)
We prepared the full list of CPUs affected by the buggy Intel Spectre 2 patches, but it is a VERY LONG LIST with 801 CPUs, so we split them into three sections.
- Server / Workstation CPUs With Buggy Intel Spectre Patches (487 SKUs)
- Desktop CPUs With Buggy Intel Spectre Patches (135 SKUs)
- Mobile CPUs With Buggy Intel Spectre Patches (179 SKUs)
As you can see, many more server and workstation CPUs are affected than desktop and mobile CPUs combined. That’s because Intel prioritised the patching of their server and workstation CPUs, over desktop and mobile CPUs.
What Is Being Done About The Buggy Intel Spectre 2 Patches?
When he first posted on the spontaneous reboot issue, Navin said that Intel was working to “understand, diagnose and address this reboot issue“.
In his latest update, he shared that Intel had already issued an early version of the new microcode updates to their partners for tests, and will release them “once that testing has been completed“.
These new microcode updates basically have Spectre 2 mitigations removed. This will restore stability to the affected Intel CPUs, while Intel fixes the problems in those mitigations.
The Intel Spectre Microcode Update Schedule Updated!
On 7 February 2018, Navin Shenoy announced that Intel has released “production microcode updates for several Skylake-based platforms” to their OEM customers and industry partners, with more platform updates “in coming days“.
The schedule was updated on 12, 20, 26 February and 1 March with more details, including production (final), pre-beta and beta versions of the new Intel Spectre microcode updates.
What Should YOU Do?
While Intel initially advised end-users to “apply updates” from system and operating system providers, they have now changed their guidance, as of 22 January 2018 :
- We recommend that OEMs, Cloud service providers, system manufacturers, software vendors and end users stop deployment of current versions on the below platforms, as they may introduce higher than expected reboots and other unpredictable system behavior.
- We also ask that our industry partners focus efforts on testing early versions of the updated solution for Broadwell and Haswell we started rolling out this weekend, so we can accelerate its release. We expect to share more details on timing later this week.
- For those concerned about system stability while we finalize the updated solutions, we are also working with our OEM partners on the option to utilize a previous version of microcode that does not display these issues, but removes the Variant 2 (Spectre) mitigations. This would be delivered via a BIOS update, and would not impact mitigations for Variant 1 (Spectre) and Variant 3 (Meltdown).
Please note that there has been no actual recorded threat or attack using the Meltdown or Spectre exploits. The damage, or risk of damage, every time your system or server spontaneously reboot is FAR WORSE than the (currently) non-existent threat of a Meltdown or Spectre exploit.
Therefore, we recommend that you DO NOT apply any microcode update for your Intel system, if you are using any Intel processor manufactured since 2011.
If you have already applied the latest Intel Spectre microcode update, and are affected by spontaneous reboots; you should upgrade to the new firmware (if they are available), or revert to the older firmware.
Meltdown + Spectre Reading Suggestions
- Everything On The Meltdown + Spectre CPU Flaws!
- The Complete List Of CPUs Vulnerable To Meltdown / Spectre
- The Microsoft Spectre + Meltdown Patch Schedule
- Intel Penryn CPUs Also Vulnerable To Meltdown + Spectre
- The Complete AMD Spectre Mitigation Strategy Guide
- The Apple Spectre + Meltdown Patches Detailed
- These Windows 10 Updates Are Bricking AMD PCs!
- Pre-2016 Intel CPUs Hit Worst By Meltdown + Spectre Fix
- Yes, AMD CPUs Are Also Vulnerable To Spectre 2 Exploit
- AMD K10 And K8 Processors Also Vulnerable To Spectre
- KB3078130 : Emergency Windows Update To Disable Intel Spectre Patches!