Shakedown - Linux Hardware Stress Testing Application

disclaimer, I am not a developer. This was built for a UI to make the tests I run and system info I monitor for RMA's more accessible to others. This is not good code, but it gets the job done without hindering stress testing.

Intended method of use.

curl -sSL https://git.karner.dev/jdkarner/shakedown/raw/branch/main/scripts/release.sh | bash

Shakedown is a GUI application for running comprehensive hardware stress tests on Linux systems to detect hardware faults. It provides a user-friendly interface for configuring and monitoring stress-ng tests while tracking system temperatures and hardware errors.

Features

• GUI-based test selection - Select which subsystems to stress test across three backend tools:

  • stress-ng:
    • CPU - Computation-heavy workloads, FPU, vector/matrix & cache stress (~30m, 100% --verify)
    • Memory - RAM integrity, bit-flips, and memory subsystem testing (~30m, 100% --verify)
    • I/O - Buffer data integrity across pipes, VM transfers & sockets (~30m, 100% --verify)
    • Thermal Stress-ng - Maximum CPU/FPU/cache thermal load with verification
    • Disk (RMA Only) - Storage read/write/seek verification for suspected failing drives
    • 24 Hour (RMA Only) - 24-hour full system stress test without storage wear
  • GPU Burn:
    • GPU Burn - CUDA GPU compute stress (gpu_burn.job)
    • Thermal GPU Burn - CUDA GPU thermal stress (thermal_gpu_burn.job)
  • P6:
    • P6 Compute & Tensor - Vulkan GPU compute & tensor stress (1h_compute_tensor.toml)
    • P6 Full System Burn - Vulkan full GPU subsystem stress (1h_full_system_burn.toml)
    • P6 Memory Torture - Vulkan VRAM torture stress (1h_memory_torture.toml)

• Verification & Sticky Result Banners:

  • Parses stress-ng .Log and .yaml output files after each run to verify that all expected stressors ran and passed with 0 failures.
  • Displays a sticky ALL TESTS PASSED or TEST FAILED banner with diagnostic details and bogo-ops metrics.

• Execution modes:

  • Sequential - Run tests one after another (easier to identify which test causes issues)
  • Parallel - Run all tests simultaneously (maximum system stress)
  • Pre-configured jobfiles - Solid defaults in jobfiles/ that can be customized if needed

Jobfiles are Sequential by default; Parallel here runs all selected test categories at the same time.

• Real-time monitoring:
  • Temperature sensors via Tricorder
  • Hardware error logging via Tricorder
  • Test progress and elapsed time

Building

scripts/build-dist.sh This builds shakedown, stress-ng, and tricorder. It does not currently include building gpu_burn, The forgejo workflow does build gpu_burn.

Usage

1. Launch the application: (The intended method of use is to use the curl command above.)

   cd $shakedown_dir
   ./shakedown
   

2. Tricorder Monitor: The Tricorder system monitor will launch automatically in monitor mode. If closed accidentally, click "🖖 Launch Tricorder" in the top bar to relaunch it.

3. Select Tests: Check the boxes for the subsystems you want to stress test

4. Choose Execution Mode:

   • Sequential: Tests run job at a time
   • Parallel: All selected jobs run simultaneously

5. Start Tests: Click "Start Tests" to begin

6. Monitor:

   • Use Tricorder for comprehensive real-time monitoring (CPU, GPU, temperatures, fans, logs)
   • Track progress in the status bar

7. Stop Tests: Click "Stop Tests" to terminate running tests

Jobfiles

Pre-configured stress-ng jobfiles are located in the jobfiles/ directory:

Customizing Jobfiles

The jobfiles use stress-ng's jobfile format. You can edit them to:

• Adjust test duration (timeout)
• Enable/disable specific stressors
• Change resource limits

Example jobfile directive:

timeout 300s     # Run for 5 minutes
cpu 0            # Use all CPU cores
verify           # Verify computations for error detection

Future Improvements / Roadmap

• P6 JSON & GPU Burn Log Parsing for Status Breakdown:
  • Currently, stress-ng .Log and .yaml files are parsed to populate per-stressor verification breakdowns and bogo-ops metrics in the sticky Passed/Failed banner and Status section, while gpu_burn and p6 rely on exit codes and fault marker detection (FAULTY / [FAULT] DATA CORRUPTION DETECTED).
  • Future implementation: Parse p6's --json report (<run_dir>/<stem>.json) and gpu_burn's .out logs (Gflop/s, error counts, per-GPU temperatures, and per-workload completion status) so that CategoryTestResult and the sticky Passed/Failed banner display detailed per-GPU and per-job metrics across all three tools.

License

GPLv2

Acknowledgments

• stress-ng - The underlying CPU, memory, I/O, and disk stress testing tool
• egui - Immediate mode GUI library for Rust
• gpu-burn - CUDA GPU stress testing tool
• p6 - Vulkan GPU stress testing tool
• tricorder - Companion system monitor