Building a Security-Hardened VxWorks Kernel

Introduction: Why Security Hardening Matters

#

In critical environments—such as defense systems, healthcare devices, and industrial automation—security and reliability are non-negotiable. Just as hospitals follow strict safety protocols before surgery, developers of real-time operating systems (RTOS) must follow strict standards to reduce risks of cyberattacks and human error.

For VxWorks, Wind River provides a security-hardened kernel profile that maps directly to NIST 800-53 controls. This approach enables compliance, improves resilience, and ensures mission-critical embedded systems are protected from vulnerabilities.

Prerequisites for Building a Hardened VxWorks Kernel

#

To follow this guide, you will need:

• Intel target hardware with TPM and Secure Boot support (e.g., Dell Latitude E6540).
• Two USB flash drives (minimum 4 GB each).
• A Windows workstation with:
  • Wind River VxWorks 7, SR21.07

Key References and Documentation

#

• Wind River

  • VxWorks Hardening Guide Approach
  • VxWorks Hardening Guide

• NIST

  • SP 800-53 Rev 4: Security and Privacy Controls
  • Cybersecurity Framework

Understanding NIST 800-53 and Its Role in RTOS Security

#

NIST 800-53 defines a structured set of outcomes for secure system design. Instead of prescribing strict steps, it outlines security goals that organizations must achieve:

1. Prepare the Organization – align processes and technology to produce secure software.
2. Protect the Software – prevent unauthorized access to software components.
3. Produce Well-Secured Software – minimize vulnerabilities in design and implementation.
4. Respond to Vulnerabilities – detect and address risks quickly after release.

This framework underpins the VxWorks hardened profile.

VxWorks Hardened Profile: NIST Compliance in Action

#

Wind River has aligned VxWorks with NIST 800-53 via the hardened profile, which is structured around five cybersecurity functions:

• Identify
• Protect
• Detect
• Respond
• Recover

The profile defines three levels of requirements:

• Mandatory – required by both Wind River and developers.
• Discretionary – optional but recommended.
• Not Applicable – irrelevant for RTOS (desktop-specific).

VxWorks Hardened Profile Features

#

When applied, the hardened profile automatically includes:

• Disk encryption (protecting data at rest).
• Secrets repository (secure storage of keys and credentials).
• SSH support for secure access.
• Kernel hardening protections.
• Secure loader.
• Stack smashing protection in RTPs.

🔐 Developers must still implement hardware security features such as secure boot, anti-tampering protections, and patch management.

Hardened Kernel Configurations

#

You can build the hardened VxWorks kernel in three modes:

1. Required Controls – base set of NIST 800-53 controls.
2. Required + Discretionary Controls – adds networking protections.
3. Development (with shell) – for testing only; never deploy this in production.

Step 1: Create Hardened VxWorks Projects

#

1. Open Wind River Workbench.

2. Navigate to File → New → Example → VxWorks System Setup.

3. Select VxWorks Security Hardened System.

4. Set base name: hssHardenedVx1.

5. Select image: development (with shell).

6. Workbench generates:

   • hssHardenedVx1_develop_vsb
   • hssHardenedVx1_develop_vip
   • hssHardenedVx1_develop_rtp

7. Apply security-related changes:

// In rtpPartition.c – set default RTP path
#define EX_USE_ROMFS

// In scapVxWorks.c – disable SCAP mechanism
#define EX_CONFIG_CHECK_SKIP

8. Update DEFAULT_BOOT_LINE with a valid IP and build the projects.

Step 2: USB Setup for Hardened VxWorks Kernel

#

• Flashdrive1: copy secure boot keys (db.sig, KEK.sig, PK.sig) and bootloader files.
• Flashdrive2: create a 2 GB FAT partition for encrypted kernel storage.

Step 3: Secure Boot Configuration in BIOS

#

1. Insert flashdrive1 and boot into BIOS.
2. Add secure boot keys (db.sig, KEK.sig, PK.sig).
3. Reboot the system; VxWorks should start with the kernel shell.

Use the devs command to identify flash drive names:

-> devs
drv refs name
  4 [ 3] /ata0a
  ...
  8 [ 3] /romfs
  1 [ 3] /ttyS0

Example: flashdrive1 = /bd0a, flashdrive2 = /bd16a.

Step 4: Stage Two Hardened Projects

#

Recreate projects with updated device names:

• Trust store vault root → /bd0a
• Encrypted partition → /bd16a

Then repeat USB preparation and boot steps.

At this point, you will have a development configuration of the hardened VxWorks kernel.

Next Steps: Moving from Development to Production

#

Now that you have a hardened development environment, you can:

• Build and test secure RTP applications.
• Deploy .vxe binaries to /romfs.
• Debug using Wind River tools.

For production deployment:

• Rebuild projects using “required controls” or “required + discretionary controls”.
• This removes the kernel shell and ensures only hardened components are included.

Frequently Asked Questions (FAQ)

#

❓ What is NIST 800-53?

#

NIST 800-53 is a framework published by the U.S. National Institute of Standards and Technology. It provides security and privacy controls for federal information systems and is widely adopted in defense, aerospace, and industrial sectors.

❓ Can I use the VxWorks hardened profile on ARM-based targets?

#

Currently, the hardened profile is supported only on the itl_generic BSP (Intel targets). ARM targets may not support all secure boot and TPM features required for compliance.

❓ Why shouldn’t I deploy the “development (with shell)” kernel?

#

The development configuration includes the VxWorks kernel shell, which provides powerful debugging capabilities. However, it also exposes security risks and should never be used in production systems.

❓ What security features does the hardened profile enable automatically?

#

It enables disk encryption, secrets repository, SSH support, kernel hardening, secure loader, and stack smashing protection. Developers must add hardware-level protections such as anti-tampering measures and patch management.

❓ Is VxWorks hardened profile compliant with STIG requirements?

#

Yes. The VxWorks Hardening Guide is presented in STIG (Security Technical Implementation Guide) format and maps directly to NIST 800-53 controls.

Conclusion

#

Building a security-hardened VxWorks kernel ensures compliance with NIST 800-53, strengthens embedded system security, and prepares your application for deployment in critical environments.

By following the steps in this guide—covering secure boot, USB setup, BIOS configuration, and project rebuilding—you can create a hardened foundation for developing secure, reliable, and resilient RTOS-based applications.

✅ Best practice: Always use hardened profiles for production and integrate hardware-level protections early in your design.