The Intel Spectre Reboot Issue – Everything You Need To Know! Rev. 3.0

The efforts to mitigate the threat of the Meltdown and Spectre exploits is officially WORSE than the threat itself. Many Intel systems are randomly and spontaneously rebooting after installing Intel Spectre 2 patches. No shit. Here is our continuing coverage of the Intel Spectre Reboot Issue!

 

Article Update History

Click here for the Article Update History

Updated @ 2018-03-06 : Added the latest updates on the issue, as well as the new Intel Spectre 2 microcode revision guidance slides.

Updated @ 2018-02-22 : Added the latest updates on the issue, as well as the new Intel Spectre 2 microcode revision guidance slides.

Updated @ 2018-02-14 : Added the latest updates on the issue, as well as the new Intel Spectre 2 microcode revision guidance slides.

Updated @ 2018-02-10 : Added a new section on the Intel Spectre 2 Microcode Update Schedule, and updated various parts of the article. Removed 80 Intel processors that have just been confirmed not to be affected by the buggy microcode updates.

Updated @ 2018-01-26 : Added the Intel Coffee Lake, Intel Xeon Scalable and Intel Xeon W family of processors to the list of affected CPUs. Added the updated Intel Spectre 2 Microcode Update Guidance.

Updated @ 2018-01-23 : Added a new section on the root cause of the spontaneous reboot issue, and updated guidance on what you should do about this problem. Also added the Intel microcode revision lists. Removed 129 workstation / server CPUs, 105 desktop CPUs and 143 mobile CPUs.

Updated @ 2018-01-18 : Added the latest development on the Intel spontaneous reboot issues, including a greatly-expanded list of affected Intel CPUs.

Updated @ 2018-01-16 : Added the full list of Intel CPUs with reboot issues

Originally posted @ 2018-01-13

 

Spontaneous Reboots With Spectre 2 Patches Updated!

On 11 January 2018, the WSJ reported that Intel was quietly asking their cloud computing customers to hold off installing Meltdown and Spectre patches because “the patches have bugs of their own“. Specifically, there were three bugs in the microcode patches they released.

In a blog post posted on the same day, Intel Executive Vice President and General Manager of the Intel Data Center Group, Navin Shenoy confirmed that Intel received reports of “higher system reboots” after applying those updates.

Basically, these systems would randomly and spontaneously reboot after installing those patches. Not something you want your computer to do, never mind servers that cater to tens or hundreds of thousands of users.

He initially confirmed that the affected systems were running Intel Broadwell and Intel Haswell CPUs, and that the issues affected both client (desktop, mobile, workstation) PCs, as well as data center servers.

But in an update a week later, Navin revealed that the newer Kaby Lake and Skylake CPUs, as well as older Sandy Bridge and Ivy Bridge processors, were also experiencing spontaneous reboot issues after updating their firmware.

Although not explicitly mentioned, the latest Intel Coffee Lake CPUs are also affected by spontaneous reboots. Hidden in their microcode revision guidance was a reference to the Coffee Lake-S processors.

In their 24 January 2018 microcode revision guidance, they further added the Intel Xeon Scalable and Intel Xeon W processor families to the list of affected CPUs.

But there’s good news – on 8 February 2018, Intel confirmed that 80 CPU models previously marked as affected have been certified to be free from the buggy microcode updates.

On 12 February 2018, Intel released beta microcode updates for some of their Coffee LakeBroadwell and Haswell processors, and pre-beta updates for their Arrandale, Clarkdale and Gulftown processors.

On 20 February 2018, Intel released production microcode updates for their Coffee Lake, Kaby Lake and Skylake processors, with new beta microcode updates for their Haswell, Ivy Bridge and Sandy Bridge processors.

On 26 February 2018, Intel released production microcode updates for the remaining Broadwell and Haswell processors, except for their Broadwell EX and Haswell EX (server) processors.

On 1 March 2018, Intel released new beta microcode updates for their Arrandale, ClarkdaleGulftown, Nehalem EP, Nehalem WS, Westmere EP, Westmere WS and Ivy Bridge EX (server) processors. They also started releasing pre-beta updates for their Lynnfield and Westmere EX processors. With the release of the Skylake Xeon E3 production update, the entire Intel Skylake family is fully patched.

Intel is not the first to be beset by problems in the rush to patch Meltdown and Spectre. Microsoft recently admitted that some Windows 10 updates were bricking some AMD PCs.

 

The Root Cause – Intel Spectre 2 Patches

On 22 January 2018, Navin Shenoy announced that Intel :

  • has identified the root cause for Broadwell and Haswell platforms, and
  • is making good progress in developing a solution to address that root cause.

They revealed that the spontaneous reboot issues seen with the affected Intel CPUs were caused by Spectre 2 mitigations in those microcode updates.

Notably, Intel only confirmed that Spectre 2 mitigations were the root cause in those two platforms. They have not confirmed Spectre 2 mitigations as the cause in the Coffee LakeKaby Lake, Skylake, Ivy Bridge and Sandy Bridge platforms that are also affected.

In fact, Intel shared that “The progress we have made in identifying a root cause for Haswell and Broadwell will help us address issues on other platforms. Please be assured we are working quickly to address these issues.

 

What CPUs Are Affected By The Buggy Intel Spectre 2 Patches?

All of the systems suffering from spontaneous reboot issues were running on HaswellBroadwellSkylake, Kaby Lake and the latest Coffee Lake CPUs. Workstation and server CPUs based on Ivy Bridge and Sandy Bridge were also affected, but thankfully not their desktop brethren.

On 8 February 2018, Intel revealed that some of the microcode updates that they suspected were buggy, were actually not buggy. They include :

  • The Intel Skylake H/S/U/Y Desktop Processors
  • The Intel Xeon E3-1200 v5 Processor Family (Skylake)

We prepared the full list of CPUs affected by the buggy Intel Spectre 2 patches, but it is a VERY LONG LIST with 801 CPUs, so we split them into three sections.

As you can see, many more server and workstation CPUs are affected than desktop and mobile CPUs combined. That’s because Intel prioritised the patching of their server and workstation CPUs, over desktop and mobile CPUs.

 

What Is Being Done About The Buggy Intel Spectre 2 Patches?

When he first posted on the spontaneous reboot issue, Navin said that Intel was working to “understand, diagnose and address this reboot issue“.

In his latest update, he shared that Intel had already issued an early version of the new microcode updates to their partners for tests, and will release them “once that testing has been completed“.

These new microcode updates basically have Spectre 2 mitigations removed. This will restore stability to the affected Intel CPUs, while Intel fixes the problems in those mitigations.

 

The Intel Spectre Microcode Update Schedule Updated!

On 7 February 2018, Navin Shenoy announced that Intel has released “production microcode updates for several Skylake-based platforms” to their OEM customers and industry partners, with more platform updates “in coming days“.

The schedule was updated on 12, 20, 26 February and 1 March with more details, including production (final), pre-beta and beta versions of the new Intel Spectre microcode updates.

 

What Should YOU Do?

While Intel initially advised end-users to “apply updates” from system and operating system providers, they have now changed their guidance, as of 22 January 2018 :

  • We recommend that OEMs, Cloud service providers, system manufacturers, software vendors and end users stop deployment of current versions on the below platforms, as they may introduce higher than expected reboots and other unpredictable system behavior.
  • We also ask that our industry partners focus efforts on testing early versions of the updated solution for Broadwell and Haswell we started rolling out this weekend, so we can accelerate its release. We expect to share more details on timing later this week.
  • For those concerned about system stability while we finalize the updated solutions, we are also working with our OEM partners on the option to utilize a previous version of microcode that does not display these issues, but removes the Variant 2 (Spectre) mitigations. This would be delivered via a BIOS update, and would not impact mitigations for Variant 1 (Spectre) and Variant 3 (Meltdown).

Please note that there has been no actual recorded threat or attack using the Meltdown or Spectre exploits. The damage, or risk of damage, every time your system or server spontaneously reboot is FAR WORSE than the (currently) non-existent threat of a Meltdown or Spectre exploit.

Therefore, we recommend that you DO NOT apply any microcode update for your Intel system, if you are using any Intel processor manufactured since 2011.

If you have already applied the latest Intel Spectre microcode update, and are affected by spontaneous reboots; you should upgrade to the new firmware (if they are available), or revert to the older firmware.

 

Meltdown + Spectre Reading Suggestions

[adrotate group=”2″]

Next Page > Server / Workstation CPUs With Buggy Intel Spectre Patches

 

Support Tech ARP!

If you like our work, you can help support our work by visiting our sponsors, participating in the Tech ARP Forums, or even donating to our fund. Any help you can render is greatly appreciated!

2 thoughts on “The Intel Spectre Reboot Issue – Everything You Need To Know! Rev. 3.0

  1. Pingback: The Complete AMD Spectre Mitigation Strategy Guide - Tech ARP

  2. Pingback: Confirmed : AMD K10 And K8 Processors Also Vulnerable To Spectre - Tech ARP

Leave a ReplyCancel reply