🚀 Industrial IoT Security: Designing Resilient OTA Updates Against Failure

[Hasan S. Cemkan]

⚙️ Update Challenges in Industrial Devices​

Industrial Linux devices are challenging to update because they are installed in factory environments or remote locations. While a simple `apt upgrade` over SSH might work in a lab setting, it's insufficient for industrial devices. These devices often have difficult access, unreliable network connections, and are expected to operate continuously. A failed update, a power outage, or an application not working after a reboot can turn a routine software release into a costly service call.

🛡️ Fundamentals of Secure OTA Updates​

A robust over-the-air (OTA) update system must do more than just copy files. It should securely install the system, verify its operation, and automatically recover in case of failure. A practical architecture includes:

• A/B system updates
  
• Hardware-backed rollback
  
• Signed update packages
  
• Software Bills of Materials (SBOMs)
  
• Reproducible builds
  

This article explores these concepts using Sfera Labs' Strato Pi Max and Silitics' Rugix as examples. This duo offers an effective combination of industrial hardware with multiple physical storage options and a software stack designed for robust, reproducible updates.

💡 Why Industrial OTA Updates Are Different​

An industrial Linux device is not finished when it's unboxed. Throughout its lifespan, it will require operating system security patches, dependency updates, application releases, and vulnerability remediation. These updates often must be performed without physical access to the device. An update system must therefore:

• Not affect the running system while the update is being installed.
  
• Withstand power loss during the update.
  
• Verify that the new system is bootable and functional.
  
• Automatically roll back to the previous version if it fails.
  
• Prevent unauthorized software from being installed.
  
• Provide traceability for each release.
  

This is also becoming increasingly important in terms of regulations such as the EU Cyber Resilience Act (CRA). A reliable OTA mechanism is a crucial part of the infrastructure needed to ensure products remain connected throughout their lifecycle.

🔄 A/B Updates: Don't Overwrite the Running System​

The core idea of robust OTA updates is simple: don't modify the running operating system in place. Instead, maintain two bootable system instances, typically called A and B. While A is running, the update is installed to B. The device then reboots and attempts to boot B. If B boots successfully and passes appropriate health checks, it becomes active. If it fails, the device reverts to A.

This prevents one of the biggest risks of in-place updates: a power loss rendering the system unusable. With A/B updates, the previous system remains available while the new system is being installed.

However, there's a significant distinction between A/B implementations. While two partitions on the same disk provide software redundancy, two physically independent storage devices offer an additional layer of resilience.

💾 Hardware Redundancy Matters Too​

The Strato Pi Max is a modular industrial platform based on the Raspberry Pi Compute Module. Its dual SD card configuration allows for two complete systems to reside on two independent SD cards. The Strato Pi Max microcontroller controls storage routing and can switch between them. This means the A/B concept isn't limited to two partitions on a single storage device. Each side can reside on separate physical media, ensuring a recovery system remains available even if one device fails.

🛠️ A Hardware/Software Stack Designed for the Problem​

Sfera Labs and Silitics have created a particularly effective hardware and software stack by combining the Strato Pi Max's hardware foundation, which provides independent storage and hardware supervision, with Rugix's software layer for reproducible Linux images, updates, rollback, and controlled system state.

Rugix consists of RugixBakery, which creates system images and update packages, and RugixCtrl, which runs on the target device and manages updates, A/B slots, and rollback.

🚀 Sending the First Update​

To demonstrate an update, the LED interval is changed from 500ms to 100ms, and the image is rebuilt. The resulting .rugixb package is transferred to the device and installed using RugixCtrl. While skipping signature verification is acceptable for this demonstration, production systems should always use cryptographically signed update packages for installation. Once installed, the device reboots into the updated system, and the LED's blink rate visibly changes.

🚨 What Happens When an Update Fails?​

The true value of the architecture emerges when things go wrong. With the dual SD configuration, Rugix prepares the inactive card while the active system continues to operate:

• The update is written to the inactive SD card.
  
• A watchdog-based rollback mechanism is engaged.
  
• The system reboots.
  

If the new system fails to boot successfully or pass health checks, the hardware-backed rollback mechanism kicks in, automatically reverting the device to the previous, known good state. This ensures uninterrupted operation in industrial environments.