Firmware Bugs and Firmware Updates

My co-author and partner in various things UEFI, Vincent Zimmer, has penned some wise words about how firmware bugs are perceived on his blog (here). He quotes the first chapter of Embedded Firmware Solutions wherein an anonymous manager states, "If you can fix a hardware bug in firmware, it’s not a bug but a documentation issue."

What Vincent said about hardware used to be the same for operating systems. That is, it was often hard to (a) convince an OS company that they had a bug, (b) get them to fix that bug and (c) get that fix out to customers. But now, Patch Tuesday is a weekly event, monitored by websites everywhere. The OS images used by OEMs can have hot-fixes applied. So, now the situation is fixed. Hardware is the hardest to fix, followed by firmware, followed by the OS, followed by applications.

That is why firmware update has been a major focus of the recent UEFI specification updates, standardizing how 3rd party components can produce and process updates (c.f. capsules and the Firmware Management protocol, ESRT). These updates are not only for the system firmware's flash device, but also for the embedded flash on smaller chips, as well as attached USB and PCI devices. Security concerns, in particular, are driving the need for reliable and timely updates of all of these.

The next frontier is delivery of these firmware updates via the OS. While there has been some progress here by the OS vendors (Redhat, Microsoft, Canonical, see older summary here), there seems to be reluctance on the part of some OEMs. Part of this is that some of their unique value(if you can call the little tray icon apps "value") is getting sucked into the OS. Part of this is relying on the process by a 3rd party (or more than one 3rd parties) to deliver updates. Part of this is: older, out of production platforms aren't interesting any more. But highly publicized hacks and bugs are putting pressure on the industry to solve the distribution problem.

Whatever the case, platform stability rests solidly on firmware stability because of its unique capabilities to fix or mitigate hardware and OS issues, as testified to by Marvel's Agents of SHIELD.

UEFI At IDF13, Part 3: Using UEFI for Secure Firmware Update of Expansion Cards

The third presentation at IDF related to UEFI was titled: "Using UEFI for Secure Firmware Update of Expansion Cards" presented by Brian Richardson, Jeff Bobzin and Terry Kirch.

With the UEFI 2.3 specification, the contributors recognized that there are lots of devices in the platform that might need their firmware image updated. Add-in cards, embedded controllers, BMCs, and sensor hubs have flash. If they have flash, then they probably want to update the contents of that flash. So the specification allowed each device driver to publish the Firmware Management protocol (chapter 32) which gave the driver a way to publish its ability to handle firmware updates and validate their version and contents. But there was still something lacking: how can an OS application update these flash images in a standard, secure way?

This presentation highlighted the security issues with firmware update of the motherboard or the add-in devices. If you leave the flash device unprotected while the OS is running, malware can attack it. So there must be security. And, if you want updated firmware images to be distributed in such a way that they can be queued for update, there has to be a standard, especially while the OS is running.

If this is true, then there needs to be some way for the OS application to pass back the new firmware image to the pre-OS environment, and then a way for these firmware images to be passed to the correct instance of the Firmware Management protocol.

The UEFI 2.4 specification introduced the Firmware Management capsule. Capsules are buckets of data associated with a type that the OS or OS application can pass back to the system firmware using the UpdateCapsule() runtime service. After a reset (when the flash protection is disabled and control is handed back to the firmware), the capsules are passed to the correct instance of the Firmware Management protocol.

Jeff, Terry and Brian went over this and then highlighted the best practices for using this new capsule type. They also identified three scenarios for capsule update which were all easy to use but which offered varying degrees of security, from good to best.

• Good - In the first scenario, the capsule contains both a UEFI driver and a firmware image. The BIOS validates the UEFI driver and then launches it, expecting it will produce the Firmware Management protocol. The system firmware then passes that instance of the protocol the rest of the capsule payload. The driver's instance of the protocol validates the firmware image and updates the flash image. The add-in card vendor doesn't put the flash and validation logic on the add-in card itself, saving precious ROM space on the card. The system firmware validates the UEFI driver using the normal secure boot mechanism, assuming it is turned on. That is, the card has to depend on the system firmware. 
• Better - In the second scenario, the capsule only contains the firmware update payload, but not a UEFI driver. Instead, the add-in card provides the instance of the Firmware Management protocol, which is installed by the UEFI driver that resides on the add-in card. That driver is also validated by the system firmware using the same secure boot criteria, and the UEFI driver on the add-in card performs the validation of the payload before flashing it. This requires more space on the flash device on the add-in card to support this validation and flash code, but it means that the security of the card's flash image is all in the hands of the card without any other help from the BIOS. 
• Best - In the third scenario, we have the same situation as the second scenario except that the validation and flash update are done by hardware on the add-in card rather than by the UEFI driver. 

This session highlighted this new feature of the UEFI 2.4 specification: secure, standard update of other firmware in the system other than system BIOS. This other firmware could be a security hole if facilities weren't available to ensure the untampered delivery and launch of flash image verification and update. This presentation highlighted the best practices with UEFI.

Disclaimer: Jeff Bobzin is my co-worker and vice president at Insyde Software.