deepseek/wrobot
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v1.2版本改为通过xbox系列手柄进行遥控,并增加智能模式和手动模式

Browse Source
This commit is contained in:
myperworld committed 2025-10-10 21:58:44 +08:00
1 parent 76603d3c31
commit 1a97bc32c3
1839 files changed
+478852 -3
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

v1.2/Shibo-main-TOMMY/lib/FastLED/ci/AGENTS.md
+246
View File
@@ -0,0 +1,246 @@
# FastLED CI/Build System Agent Guidelines
## 🔧 Python Build System
### Core Architecture
FastLED uses a high-performance Python-based build system:
**Key Components:**
- `ci/compiler/clang_compiler.py` - Core compiler with PCH and fingerprint cache
- `ci/ci/fingerprint_cache.py` - 99% faster PCH rebuilds through intelligent caching
- `ci/util/build_flags.py` - TOML-based build configuration management
- `ci/compiler/test_example_compilation.py` - Example compilation system
**Performance Features:**
- **PCH (Precompiled Headers)** - Dramatically faster compilation
- **Fingerprint Cache** - 99% time reduction for unchanged dependencies
- **Parallel Compilation** - Automatic CPU-optimized parallel builds
- **Smart Rebuilds** - Only rebuild when source files actually change
### 🚨 Command Execution Rules
#### FOREGROUND AGENTS (Interactive Development)
**Python Code Execution:**
- ✅ **Create/modify tmp.py** in project root (NOT subdirectories)
- ✅ **Always run: `uv run python tmp.py`** (never just `python`)
- ✅ **Correct import paths:** `from ci.ci.module` for ci.ci modules, `from ci.module` for top-level ci modules
- ❌ **NEVER cd to subdirectories** - stay in project root
**Example:**
```python
# tmp.py (created in project root)
from ci.ci.clang_compiler import Compiler # CORRECT import path
import subprocess
result = subprocess.run(['git', 'status'], capture_output=True, text=True)
print(result.stdout)
```
Then run: `uv run python tmp.py`
#### BACKGROUND AGENTS (Automated/CI Environments)
**Restrictions:**
- ❌ **NO tmp.py files** (forbidden for background agents)
- ✅ **Use existing scripts:** `uv run python ci/ci-compile.py uno --examples Blink`
- ✅ **Use MCP server tools** for programmatic operations
- ✅ **Always use `uv run python`** with existing scripts
## 🤖 MCP Server Configuration
**Start server:** `uv run mcp_server.py`
**Available Tools:**
- Running tests with various options
- Compiling examples for different platforms
- Code fingerprinting and change detection
- Linting and formatting
- **Build info analysis** for platform-specific defines, compiler flags, toolchain information
- **Symbol analysis** for binary optimization (all platforms)
- Stack trace setup for enhanced debugging
- **🌐 FastLED Web Compiler** with Playwright (FOREGROUND AGENTS ONLY)
- **🚨 `validate_completion` tool** (MANDATORY for background agents)
### FastLED Web Compiler (FOREGROUND AGENTS ONLY)
**Prerequisites:**
- `pip install fastled` - FastLED web compiler
- `pip install playwright` - Browser automation
- Docker (optional, for faster compilation)
**Usage via MCP Server:**
```python
# Basic usage
use run_fastled_web_compiler tool with:
- example_path: "examples/Audio"
- capture_duration: 30
- headless: false
- save_screenshot: true
```
**Key Features:**
- Compiles Arduino sketches to WASM for browser execution
- Captures console.log output with playwright automation
- Takes screenshots of running visualizations
- Monitors FastLED_onFrame calls to verify proper initialization
**🚫 BACKGROUND AGENT RESTRICTION:** This tool is COMPLETELY DISABLED for background agents and CI environments.
## 🤖 Linting Guidelines
### FOREGROUND AGENTS
**🚨 ALWAYS USE `bash lint` - DO NOT RUN INDIVIDUAL LINTING SCRIPTS**
- ✅ **CORRECT:** `bash lint`
- ❌ **FORBIDDEN:** `./lint-js`, `./check-js`, `python3 scripts/enhance-js-typing.py`
- ❌ **FORBIDDEN:** `uv run ruff check`, `uv run black`, individual tools
**WHY:** `bash lint` provides comprehensive coverage of Python, C++, JavaScript, and enhancement analysis.
### BACKGROUND AGENTS
**CAN USE FINE-GRAINED LINTING FOR SPECIFIC NEEDS**
Background agents may use individual linting scripts when needed:
- `./lint-js` - JavaScript-only linting
- `./check-js` - JavaScript type checking
- `python3 scripts/enhance-js-typing.py` - JavaScript enhancement analysis
- `uv run ruff check` - Python linting only
- MCP server `lint_code` tool - Programmatic access
**BUT STILL PREFER `bash lint` FOR COMPREHENSIVE CHECKING**
## Build System Commands
**Unit Test Compilation:**
```bash
# Fast Python API (default)
bash test --unit --verbose # Uses PCH optimization
# Disable PCH if needed
bash test --unit --no-pch --verbose
# Legacy CMake system (8x slower)
bash test --unit --legacy --verbose
```
**Available Build Options:**
- `--no-pch` - Disable precompiled headers
- `--clang` - Use Clang compiler (recommended for speed)
- `--clean` - Force full rebuild
- `--verbose` - Show detailed compilation output
- `--legacy` - Use old CMake system (discouraged)
**Platform Compilation:**
```bash
# Compile examples for specific platforms
uv run ci/ci-compile.py uno --examples Blink
uv run ci/ci-compile.py esp32dev --examples Blink
uv run ci/ci-compile.py teensy31 --examples Blink
```
**WASM Compilation:**
```bash
# Compile any sketch directory to WASM
uv run ci/wasm_compile.py path/to/your/sketch
# Quick compile test (compile only, no browser)
uv run ci/wasm_compile.py examples/Blink --just-compile
# Compile and open browser
uv run ci/wasm_compile.py examples/Blink -b --open
```
## Build Troubleshooting
**For Linker Issues:**
1. Check `tests/cmake/LinkerCompatibility.cmake` first
2. Look for lld-link detection and compatibility functions
3. Check GNU→MSVC flag translation logic
4. Verify warning suppression settings
**For Compiler Issues:**
1. Check `tests/cmake/CompilerDetection.cmake` for detection logic
2. Review `tests/cmake/CompilerFlags.cmake` for flag conflicts
3. Verify optimization settings in `OptimizationSettings.cmake`
**For Build Performance:**
1. Check `tests/cmake/ParallelBuild.cmake` for parallel settings
2. Review LTO configuration in `OptimizationSettings.cmake`
3. Verify linker selection (mold, lld, default)
## 🚨 Critical User Feedback Corrections
**Always Use `uv run python` - NEVER just `python` or `uv run`:**
- ❌ **WRONG**: `python -c "..."`
- ❌ **WRONG**: `uv run -c "..."`
- ✅ **CORRECT**: `uv run python -c "..."`
**Correct Import Paths:**
- ✅ **CORRECT**: `from ci.ci.clang_compiler import ...` (for ci.ci modules)
- ✅ **CORRECT**: `from ci.clang_compiler import ...` (for top-level ci modules, if they exist)
**Never Change Directory:**
- ❌ **WRONG**: `cd ci && python ...`
- ❌ **WRONG**: `cd ci/ci && python ...`
- ✅ **CORRECT**: Stay in project root, use full paths: `uv run python ci/script.py`
**Git-Bash Terminal Compatibility:**
Use leading space for git-bash compatibility:
- ✅ **CORRECT**: ` bash test` (note the leading space)
- ❌ **INCORRECT**: `bash test` (may get truncated to `ash test`)
## Platform Build Information Analysis
**Generate Build Info:**
```bash
# Compile a platform to generate build_info.json
uv run ci/ci-compile.py uno --examples Blink
uv run ci/ci-compile.py esp32dev --examples Blink
```
**Analyze Platform Defines:**
```bash
# Command line tool
python3 ci/ci/build_info_analyzer.py --board uno --show-defines
# Via MCP server
uv run mcp_server.py
# Use build_info_analysis tool with: board="uno", show_defines=true
```
**Compare Platforms:**
```bash
# Command line
python3 ci/ci/build_info_analyzer.py --compare uno esp32dev
# Via MCP
# Use build_info_analysis tool with: board="uno", compare_with="esp32dev"
```
## Symbol Analysis for Binary Optimization
**Analyze specific platform:**
```bash
uv run ci/ci/symbol_analysis.py --board uno
uv run ci/ci/symbol_analysis.py --board esp32dev
uv run ci/ci/symbol_analysis.py --board teensy31
```
**Analyze all available platforms:**
```bash
uv run ci/demo_symbol_analysis.py
```
**Using MCP Server:**
```bash
# Use MCP server tools
uv run mcp_server.py
# Then use symbol_analysis tool with board: "uno" or "auto"
# Or use symbol_analysis tool with run_all_platforms: true
```
**JSON Output:**
```bash
uv run symbol_analysis.py --board esp32dev --output-json
# Results saved to: .build/esp32dev_symbol_analysis.json
```
## Memory Refresh Rule
**🚨 ALL AGENTS: Read ci/AGENTS.md before concluding CI/build work to refresh memory about current Python build system rules and MCP server requirements.**
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/BUILD_SYSTEM_MIGRATION.md
+228
View File
@@ -0,0 +1,228 @@
# FastLED Python Build System - Implementation Summary
## Overview
Successfully implemented a new Python-based build system to replace the complex CMake approach for FastLED example compilation testing. The new system delivers on all design goals: **Simple, Fast, and Transparent**.
## Key Achievements
### ⚡ Performance Results
- **4.3 examples/second** compilation rate
- **90%+ faster** than CMake-based approach
- **Sub-second builds** for incremental changes
- **Intelligent caching** with 30%+ hit rates on repeated builds
### 🏗️ Architecture Implemented
```
FastLED Python Build System
├── ci/build_system/
│ ├── __init__.py # Package initialization
│ ├── build_config.py # Configuration management
│ ├── example_scanner.py # .ino file discovery & analysis
│ ├── change_detector.py # Fast file change detection
│ ├── compiler.py # Direct Clang compilation
│ ├── cache_manager.py # Build cache management
│ └── fastled_build.py # Main build engine
├── ci/fastled_build # CLI entry point
├── ci/test_example_compilation_python.py # New test script
└── ci/test_example_compilation.py # Updated original (with --use-python-build flag)
```
### 🎯 Core Features Delivered
#### 1. **Fast Example Discovery**
- Scans 80+ examples in milliseconds
- Categorizes by FastLED usage and PCH compatibility
- Intelligent filtering of problematic examples
#### 2. **Content-Based Change Detection**
- SHA-256 fingerprinting of source files
- Dependency tracking for headers
- Smart cache invalidation
#### 3. **Precompiled Header Optimization**
- Automatic PCH generation for FastLED headers
- PCH reuse across compatible examples
- 50%+ compilation speed improvement
#### 4. **Intelligent Caching**
- Content-addressable build cache
- sccache integration for compiler-level caching
- Automatic cleanup of stale entries
#### 5. **Parallel Compilation**
- Multi-threaded compilation (16 cores utilized)
- ThreadPoolExecutor-based job management
- Optimal resource utilization
## Usage Examples
### CLI Tool
```bash
# Quick status check
./fastled_build status
# Build all examples
./fastled_build build
# Build specific examples
./fastled_build build Blink DemoReel100
# Clean build
./fastled_build clean
./fastled_build build
# System diagnostics
./fastled_build diagnostics
```
### Test Scripts
```bash
# Default Python build system (recommended)
python test_example_compilation.py
# Alternative Python build system script
python test_example_compilation_python.py
# Both scripts now use Python build system by default
python test_example_compilation.py Blink DemoReel100 # Python system
```
## Technical Implementation Details
### Example Categorization
- **FastLED PCH Compatible**: Examples that can use precompiled headers
- **FastLED Full Build**: Examples requiring full compilation
- **Basic Examples**: Non-FastLED examples
- **Platform Specific**: Hardware-dependent examples
- **Problematic**: Known issues (excluded by default)
### Build Configuration
- **Compiler**: Auto-detection (Clang preferred)
- **Platform**: STUB implementation for testing
- **Flags**: C++17, -O0 for fast compilation
- **Arduino Compatibility**: WASM Arduino.h integration
### Performance Optimizations
- **Fast Paths**: "No changes" detection in < 1 second
- **PCH Reuse**: Share PCH across compatible examples
- **Content Hashing**: Only hash changed files
- **Parallel Processing**: All I/O and compilation parallelized
## Migration Status
### ✅ Phase 1: Parallel Implementation - COMPLETE
- ✅ Implemented Python build system alongside CMake
- ✅ Added `--use-python-build` flag to test script
- ✅ Validated equivalent functionality
- ✅ Performance testing shows 4x+ improvement
### 🚧 Phase 2: Feature Parity - IN PROGRESS
- ✅ All CMake features replicated
- ✅ Performance optimization completed
- ✅ CLI and API documentation
- ⏳ Extended platform testing
### ✅ Phase 3: Migration - COMPLETE
- ✅ Switched default to Python build system
- ✅ Removed CMake implementation completely
- ✅ Updated `test_example_compilation.py` to use Python build system only
- ✅ Maintained backward compatibility with same CLI interface
### 🧹 Phase 4: Cleanup - ⚠️ **INCOMPLETE**
- ❌ Remove misleading "CMake" references in function names and error messages
- ❌ Clean up dead code paths that reference non-existent CMake directories
- ❌ Fix confusing system names (both systems are Python-based)
- ❌ Update all documentation to reflect accurate system architecture
- ⚠️ **Status**: Misleading naming caused confusion and bugs; cleanup required
## Performance Comparison
| Metric | CMake System | Python System | Improvement |
|--------|--------------|---------------|-------------|
| Cold start | 15-30s | 1-2s | **15x faster** |
| Incremental (5 examples) | 8-15s | 1-3s | **5x faster** |
| Full build (80 examples) | 45-60s | 10-15s | **4x faster** |
| Cache hit rate | ~60% | ~90% | **50% better** |
| Memory usage | 2-4GB | 200-500MB | **80% less** |
## Error Handling & Diagnostics
### Comprehensive Error Reporting
- Clear, actionable error messages
- Compilation error aggregation
- Performance timing breakdown
- Cache statistics and health
### Diagnostic Tools
```bash
./fastled_build diagnostics
```
Shows:
- Compiler information and availability
- Cache statistics and hit rates
- sccache integration status
- File change detection stats
## Integration with Existing Workflow
### Backward Compatibility
- ✅ Original `test_example_compilation.py` still works with same interface
- ✅ Same command-line arguments and exit codes
- ✅ Same behavior and output format
- ✅ Complete drop-in replacement (CMake removed)
- ✅ Performance improvement with no API changes
### Testing Integration
```python
# Easy integration in existing test scripts
from build_system import BuildConfig, FastLEDBuildEngine
config = BuildConfig.from_environment()
engine = FastLEDBuildEngine(config)
result = engine.build(examples=["Blink", "DemoReel100"])
assert result.success
```
## Benefits Realized
### 1. **Simplicity**
- ✅ Direct compiler control (no CMake abstraction)
- ✅ Python stack traces for easy debugging
- ✅ Transparent, understandable build logic
### 2. **Performance**
- ✅ 4x+ faster than CMake approach
- ✅ Intelligent caching tailored to FastLED
- ✅ Efficient incremental builds
### 3. **Maintainability**
- ✅ Python codebase (team expertise)
- ✅ Modular, testable design
- ✅ Clear separation of concerns
### 4. **Flexibility**
- ✅ Easy FastLED-specific optimizations
- ✅ Simple tool integration
- ✅ Consistent cross-platform behavior
## Next Steps
1. **Extended Testing**: Test with more examples and edge cases
2. **Documentation**: Update development docs to recommend Python system
3. **CI Integration**: Consider switching CI to use Python build system
4. **Feature Additions**: Add requested features (linking, size analysis, etc.)
## Conclusion
⚠️ **MIGRATION STATUS**: The Python build system has successfully replaced the CMake approach, but **Phase 4 cleanup was never completed**, leaving misleading function names and error messages that reference "CMake" when both systems are actually Python-based.
### Final Results
- **🚀 13.3 examples/second** compilation rate (in cached scenarios)
- **⚡ Ultra-fast builds** with intelligent caching
- **🔄 100% backward compatibility** with existing workflows
- **🧹 Clean codebase** with CMake complexity completely removed
- **📊 Superior performance** across all metrics
The system delivers on all original design goals: **Simple, Fast, and Transparent**. However, **incomplete cleanup has left confusing naming** that makes the system less transparent than intended. The naming issues have been identified and are being addressed.
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/README.md
+11
View File
@@ -0,0 +1,11 @@
This is the Continuous Integration tool that builds the product in various configurations.
Running
Install the python `uv` tool.
`pip install uv`
Now run the python in this directory
`uv run ci-compile.py`
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/SYMBOL_ANALYSIS.md
+313
View File
@@ -0,0 +1,313 @@
# FastLED Symbol Analysis
This document describes the new generic symbol analysis functionality that works across all supported platforms including UNO, ESP32, and others.
## Overview
The symbol analysis tool provides comprehensive analysis of compiled ELF files to help identify optimization opportunities and understand binary size allocation. Unlike the previous ESP32-specific implementation, this tool:
- ✅ **Works on ALL platforms** (UNO, ESP32, Teensy, STM32, etc.)
- ✅ **Pulls ALL symbols** without filtering or classification
- ✅ **No FastLED-specific filtering** - analyzes everything
- ✅ **Generic and extensible** for any platform with build_info.json
## Quick Start
### Analyze a specific platform
```bash
cd ci
uv run symbol_analysis.py --board uno
uv run symbol_analysis.py --board esp32dev
```
### Run analysis on all available platforms
```bash
uv run demo_symbol_analysis.py
```
### Use in GitHub Actions
```bash
uv run symbol_analysis_runner.py --board uno --example Blink
```
## Files
- **`symbol_analysis.py`** - Main generic symbol analysis tool
- **`symbol_analysis_runner.py`** - GitHub Actions wrapper
- **`demo_symbol_analysis.py`** - Demonstration script for multiple platforms
- **`SYMBOL_ANALYSIS.md`** - This documentation file
## Example Output
### UNO Platform Results
```
================================================================================
UNO SYMBOL ANALYSIS REPORT
================================================================================
SUMMARY:
Total symbols: 51
Total symbol size: 3767 bytes (3.7 KB)
LARGEST SYMBOLS (all symbols, sorted by size):
1. 1204 bytes - ClocklessController<...>::showPixels(...)
2. 572 bytes - CFastLED::show(unsigned char) [clone .constprop.18]
3. 460 bytes - main
4. 204 bytes - CPixelLEDController<...>::show(...)
5. 168 bytes - CPixelLEDController<...>::showColor(...)
SYMBOL TYPE BREAKDOWN:
t: 19 symbols, 2710 bytes (2.6 KB)
T: 14 symbols, 904 bytes (0.9 KB)
b: 15 symbols, 89 bytes (0.1 KB)
d: 3 symbols, 64 bytes (0.1 KB)
```
### ESP32 Platform Results
```
================================================================================
ESP32DEV SYMBOL ANALYSIS REPORT
================================================================================
SUMMARY:
Total symbols: 2503
Total symbol size: 237092 bytes (231.5 KB)
LARGEST SYMBOLS (all symbols, sorted by size):
1. 12009 bytes - _vfprintf_r
2. 11813 bytes - _svfprintf_r
3. 8010 bytes - _vfiprintf_r
4. 7854 bytes - _svfiprintf_r
5. 4192 bytes - port_IntStack
SYMBOL TYPE BREAKDOWN:
T: 1240 symbols, 156230 bytes (152.6 KB)
t: 371 symbols, 45841 bytes (44.8 KB)
d: 475 symbols, 13032 bytes (12.7 KB)
D: 60 symbols, 9421 bytes (9.2 KB)
W: 126 symbols, 5209 bytes (5.1 KB)
```
### ESP32S3 Platform Results
```
================================================================================
ESP32S3 SYMBOL ANALYSIS REPORT
================================================================================
SUMMARY:
Total symbols: 2831
Total symbol size: 260710 bytes (254.6 KB)
LARGEST SYMBOLS (all symbols, sorted by size):
1. 11309 bytes - _vfprintf_r
2. 11118 bytes - _svfprintf_r
3. 7337 bytes - _svfiprintf_r
4. 7265 bytes - _vfiprintf_r
5. 4192 bytes - port_IntStack
6. 1164 bytes - printBeforeSetupInfo()
7. 828 bytes - fl::RmtController5::loadPixelData(PixelIterator&)
SYMBOL TYPE BREAKDOWN:
T: 1416 symbols, 172591 bytes (168.5 KB)
t: 429 symbols, 48854 bytes (47.7 KB)
d: 529 symbols, 14189 bytes (13.9 KB)
D: 72 symbols, 9831 bytes (9.6 KB)
W: 122 symbols, 5457 bytes (5.3 KB)
```
## Output Files
The tool generates JSON files with complete symbol data:
- `.build/{board}_symbol_analysis.json` - Complete analysis results
### JSON Structure
```json
{
"summary": {
"board": "uno",
"total_symbols": 51,
"total_size": 3767,
"largest_symbols": [...],
"type_breakdown": {...},
"dependencies": {...}
},
"all_symbols_sorted_by_size": [
{
"address": "00001240",
"size": 1204,
"type": "t",
"name": "_ZN19ClocklessController...",
"demangled_name": "ClocklessController<...>::showPixels(...)"
},
...
],
"dependencies": {
"module.cpp.o": ["symbol1", "symbol2", ...]
}
}
```
## Supported Platforms
The tool automatically detects and works with any platform that has:
- A `build_info.json` file in `.build/{platform}/`
- Standard toolchain tools (nm, c++filt) in the build info aliases
Successfully tested platforms:
- ✅ **UNO** (AVR) - 51 symbols, 3.7 KB
- ✅ **ESP32DEV** (Xtensa) - 2503 symbols, 231.5 KB
- ✅ **ESP32S3** (Xtensa) - 2831 symbols, 254.6 KB
- ✅ **TEENSY31** (ARM Cortex-M4) - 238 symbols, 10.9 KB
- ✅ **TEENSYLC** (ARM Cortex-M0+) - 241 symbols, 9.1 KB
- ✅ **DIGIX** (ARM Cortex-M3) - 298 symbols, 17.9 KB
- ✅ **STM32** (ARM Cortex-M3) - 311 symbols, 15.6 KB
## Key Improvements
### From ESP32-specific to Generic
**Before (ESP32 only):**
```python
# Hard-coded ESP32 board detection
esp32_boards = ["esp32dev", "esp32", "esp32s2", ...]
# FastLED-specific filtering
if any(keyword in search_text for keyword in [
"fastled", "cfastled", "crgb", "hsv", ...]):
fastled_symbols.append(symbol_info)
```
**After (Generic):**
```python
# Auto-detect any platform with build_info.json
for item in build_dir.iterdir():
if item.is_dir() and (item / "build_info.json").exists():
available_boards.append((build_info_file, item.name))
# No filtering - analyze ALL symbols
symbols.append(symbol_info) # Everything included
```
### Symbol Analysis Improvements
1. **No Classification** - All symbols are included without filtering
2. **Complete Demangling** - All C++ symbols are properly demangled
3. **Platform Agnostic** - Works with any toolchain (AVR, ARM, Xtensa, etc.)
4. **Size-based Sorting** - All results sorted by symbol size for easy optimization targeting
## GitHub Actions Integration
The `symbol_analysis_runner.py` script provides GitHub Actions compatibility:
```yaml
- name: Run Symbol Analysis
run: python ci/symbol_analysis_runner.py --board uno --example Blink
```
This runner:
- Validates build_info.json exists
- Handles errors gracefully with `--skip-on-failure`
- Provides proper exit codes for CI/CD
## Migration from ESP32 Analysis
If you were using the old ESP32-specific analysis:
**Old command:**
```bash
cd ci/ci && python esp32_symbol_analysis.py
```
**New command:**
```bash
cd ci && uv run symbol_analysis.py --board esp32dev
```
The new tool provides all the same information plus:
- Support for more platforms
- ALL symbols instead of just FastLED ones
- Better error handling
- Cleaner output format
## Advanced Usage
### Analyzing Custom Platforms
To add support for a new platform:
1. Ensure the platform builds correctly with PlatformIO
2. Verify `build_info.json` is generated in `.build/{platform}/`
3. Run: `uv run symbol_analysis.py --board {platform}`
### Batch Analysis
```bash
# Analyze all available platforms
uv run demo_symbol_analysis.py
# Analyze specific platforms in sequence
for board in uno esp32dev teensy31; do
uv run symbol_analysis.py --board $board
done
```
### Integration with Build Systems
```python
# In your Python build script
from ci.ci.symbol_analysis import main as analyze_symbols
import sys
# Override arguments
sys.argv = ["symbol_analysis.py", "--board", "uno"]
analyze_symbols()
```
## Troubleshooting
### Common Issues
1. **"build_info.json not found"**
- Solution: Compile the platform first: `uv run ci/ci-compile.py {board} --examples Blink`
2. **"KeyError: 'board_name'"**
- Solution: The board name in the directory doesn't match the key in build_info.json
- Check: `cat .build/{board}/build_info.json | jq 'keys'`
3. **"Tool not found"**
- Solution: The toolchain isn't installed or path is incorrect in build_info.json
### Debug Mode
To debug issues, run with verbose Python output:
```bash
uv run python -v symbol_analysis.py --board uno
```
## Contributing
When adding new features:
1. Maintain platform independence
2. Don't add filtering logic
3. Test on multiple platforms
4. Update this documentation
5. Add tests to verify functionality
## Performance
Platform analysis times (approximate):
- **UNO**: < 1 second (51 symbols)
- **Teensy**: < 2 seconds (238-241 symbols)
- **STM32**: < 3 seconds (311 symbols)
- **ESP32**: < 10 seconds (2503 symbols)
The tool scales well with symbol count and provides progress feedback during demangling.
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/__init__.py
+13
View File
@@ -0,0 +1,13 @@
"""FastLED CI tools and utilities
Module-level initialization to ensure consistent runtime behavior across tools.
"""
# Configure UTF-8 console output on Windows globally for CI tools
try:
from ci.util.console_utf8 import configure_utf8_console
configure_utf8_console()
except Exception:
# Never fail import due to console configuration
pass
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/adafruit_bridge/adafruit_bridge.ino.cpp
+45
View File
@@ -0,0 +1,45 @@
#define FASTLED_USE_ADAFRUIT_NEOPIXEL
#include "FastLED.h"
// Test the non-template Adafruit bridge
#define DATA_PIN 2
#define NUM_LEDS 144
// Test LED arrays with different RGB orders
CRGB leds_grb[NUM_LEDS];
CRGB leds_rgb[NUM_LEDS];
CRGB leds_bgr[NUM_LEDS];
void setup() {
Serial.begin(115200);
// Test template controllers
FastLED.addLeds<WS2812, DATA_PIN, GRB>(leds_grb, NUM_LEDS);
FastLED.addLeds<WS2812, DATA_PIN+1, RGB>(leds_rgb, NUM_LEDS);
FastLED.addLeds<WS2812, DATA_PIN+2, BGR>(leds_bgr, NUM_LEDS);
// Fill with test patterns
fill_rainbow(leds_grb, NUM_LEDS, 0, 255/NUM_LEDS);
fill_solid(leds_rgb, NUM_LEDS, CRGB::Red);
fill_solid(leds_bgr, NUM_LEDS, CRGB::Blue);
FastLED.show();
Serial.println("Adafruit bridge test initialized");
}
void loop() {
static uint8_t hue = 0;
// Update colors
fill_rainbow(leds_grb, NUM_LEDS, hue, 255/NUM_LEDS);
fill_solid(leds_rgb, NUM_LEDS, CHSV(hue, 255, 255));
fill_solid(leds_bgr, NUM_LEDS, CHSV(hue + 128, 255, 255));
FastLED.show();
hue += 2;
delay(50);
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/adafruit_bridge/platformio.ini
+13
View File
@@ -0,0 +1,13 @@
[platformio]
src_dir = symlink://../../../src
[env:dev]
platform = https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip
board = esp32-s3-devkitc-1
framework = arduino
lib_deps =
adafruit/Adafruit NeoPixel
lib_ldf_mode=deep
build_flags=-DFASTLED_STUB_MAIN_INCLUDE_INO="../ci/adafruit_bridge/adafruit_bridge.ino.cpp"
-DFASTLED_USE_ADAFRUIT_NEOPIXEL=1
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards.py
+769
View File
@@ -0,0 +1,769 @@
# dataclasses
import json
from copy import deepcopy
from dataclasses import dataclass
from typing import Any
# An open source version of the esp-idf 5.1 platform for the ESP32 that
# gives esp32 boards the same build environment as the Arduino 2.3.1+.
# Set to a specific release, we may want to update this in the future.
ESP32_IDF_5_1_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip"
# TODO: Upgrade toolkit to 5.3
ESP32_IDF_5_3_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/53.03.10/platform-espressif32.zip"
ESP32_IDF_5_4_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/54.03.20/platform-espressif32.zip"
ESP32_IDF_5_5_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32/releases/download/55.03.30-2/platform-espressif32.zip"
ESP32_IDF_LATEST_PIOARDUINO = ESP32_IDF_5_5_PIOARDUINO
ESP32_IDF_LATEST_PIOARDUINO = (
"https://github.com/pioarduino/platform-espressif32.git#develop"
)
ESP32_IDF_4_4_LATEST = (
"https://github.com/platformio/platform-espressif32/archive/refs/tags/v4.4.0.zip"
)
APOLLO3_2_2_0 = "https://github.com/nigelb/platform-apollo3blue"
# Top of trunk.
# ESP32_IDF_5_1_PIOARDUINO = "https://github.com/pioarduino/platform-espressif32"
# Old fork that we were using
# ESP32_IDF_5_1_PIOARDUINO = "https://github.com/zackees/platform-espressif32#Arduino/IDF5"
# ALL will be auto populated in the Board constructor whenever a
# board is defined.
ALL: list["Board"] = []
class Auto:
pass
@dataclass
class Board:
board_name: str
real_board_name: str | None = None
platform: str | None = None
platform_needs_install: bool = False
use_pio_run: bool = (
False # some platforms like esp32-c2-devkitm-1 will only work with pio run
)
platform_packages: str | None = None
framework: str | None = None
board_build_mcu: str | None = None
board_build_core: str | None = None
board_build_filesystem_size: str | None = None
build_flags: list[str] | None = None # Reserved for future use.
build_unflags: list[str] | None = None # New: unflag options
defines: list[str] | None = None
customsdk: str | None = None
board_partitions: str | None = None # Reserved for future use.
no_board_spec: bool = (
False # For platforms like 'native' that don't need a board specification
)
add_board_to_all: bool = True
lib_compat_mode: str | None = (
None # Library compatibility mode (e.g., 'off' for native platform)
)
lib_ldf_mode: str | None = (
None # Library Dependency Finder mode (e.g., 'chain+' for enhanced dependency finding)
)
lib_ignore: list[str] | None = (
None # Libraries to ignore during compilation (e.g., ['I2S'] for UNO R4 WiFi)
)
def __post_init__(self) -> None:
# Check if framework is set, warn and auto-set to arduino if missing (except for native/stub platforms)
if self.framework is None and not self._is_native_or_stub_platform():
self.framework = "arduino"
# Auto-detect and exclude problematic libraries based on environmental signals
self._auto_detect_library_exclusions()
ALL.append(self)
def _is_native_or_stub_platform(self) -> bool:
"""Check if this is a native or stub platform that doesn't need a framework"""
# Check for native platform
if self.platform and (
"native" in self.platform.lower() or "stub" in self.platform.lower()
):
return True
# Check for stub build flags
if self.build_flags:
for flag in self.build_flags:
if "FASTLED_STUB" in flag or "PLATFORM_NATIVE" in flag:
return True
# Check no_board_spec flag (typically used for native platforms)
if self.no_board_spec:
return True
return False
def _auto_detect_library_exclusions(self) -> None:
"""Automatically detect and exclude libraries known to be broken on this platform."""
# Check for I2S library problems based on environmental signals
if self._should_ignore_i2s_library():
if not self.lib_ignore:
self.lib_ignore = []
if "I2S" not in self.lib_ignore:
self.lib_ignore.append("I2S")
def _should_ignore_i2s_library(self) -> bool:
"""Detect if I2S library should be ignored based on platform characteristics."""
# Known problematic platforms - Renesas RA family
if self.platform == "renesas-ra":
return True
# Known problematic board names
problematic_boards = ["uno_r4_wifi", "uno_r4_minima"]
if self.board_name in problematic_boards:
return True
# Check for specific defines that indicate I2S problems
if self.defines:
problematic_defines = [
"ARDUINO_UNOR4_WIFI",
"ARDUINO_UNOR4_MINIMA",
"ARDUINO_ARCH_RENESAS",
"ARDUINO_ARCH_RENESAS_UNO",
"_RENESAS_RA_",
"ARDUINO_FSP",
]
for define in self.defines:
# Check if define contains any problematic patterns
if any(prob_define in define for prob_define in problematic_defines):
return True
return False
def clone(self) -> "Board":
out = Board(
board_name=self.board_name,
add_board_to_all=False,
)
for field_name, field_info in self.__dataclass_fields__.items():
field_value: Any = getattr(self, field_name)
# Create deep copy for mutable types to avoid shared references
if isinstance(field_value, (list, dict)):
field_value = deepcopy(field_value) # type: ignore[misc]
setattr(out, field_name, field_value)
return out
def get_real_board_name(self) -> str:
return self.real_board_name if self.real_board_name else self.board_name
def to_dictionary(self) -> dict[str, list[str]]:
out: dict[str, list[str]] = {}
if self.real_board_name:
out[self.board_name] = [f"board={self.real_board_name}"]
options = out.setdefault(self.board_name, [])
if self.platform:
options.append(f"platform={self.platform}")
if self.platform_needs_install:
options.append("platform_needs_install=true")
if self.platform_packages:
options.append(f"platform_packages={self.platform_packages}")
if self.framework:
options.append(f"framework={self.framework}")
if self.board_build_core:
options.append(f"board_build.core={self.board_build_core}")
if self.board_build_mcu:
options.append(f"board_build.mcu={self.board_build_mcu}")
if self.board_build_filesystem_size:
options.append(
f"board_build.filesystem_size={self.board_build_filesystem_size}"
)
if self.defines:
for define in self.defines:
options.append(f"build_flags=-D{define}")
# Handle build_unflags
if self.build_unflags:
for uf in self.build_unflags:
options.append(f"build_unflags={uf}")
# Handle explicit build_flags (added for native host compilation and other special cases)
if self.build_flags:
for bf in self.build_flags:
options.append(f"build_flags={bf}")
if self.customsdk:
options.append(f"custom_sdkconfig={self.customsdk}")
# Add board-specific build cache directory pointing via symlink directive
# here = Path(__file__).parent
# project_root = here.parent.parent # Move from ci/util/ to project root
# cache_dir = project_root / ".pio_cache" / self.board_name
# absolute_cache_dir = cache_dir.resolve()
# options.append(f"build_cache_dir=symlink://{absolute_cache_dir}")
return out
def __repr__(self) -> str:
json_str = json.dumps(self.to_dictionary(), indent=4, sort_keys=True)
return json_str
def __hash__(self) -> int:
data_str = self.__repr__()
return hash(data_str)
def to_platformio_ini(
self,
additional_defines: list[str] | None = None,
additional_include_dirs: list[str] | None = None,
additional_libs: list[str] | None = None,
include_platformio_section: bool = False,
core_dir: str | None = None,
packages_dir: str | None = None,
project_root: str | None = None,
build_cache_dir: str | None = None,
extra_scripts: list[str] | None = None,
) -> str:
"""Return a `platformio.ini` snippet representing this board.
The output is suitable for directly appending to a *platformio.ini* file
and follows the same semantics used by the PlatformIO CLI. Only
parameters understood by PlatformIO are emitted – internal helper
fields like ``platform_needs_install`` and ``use_pio_run`` are **not**
included because they are consumed exclusively by the build helpers in
the *ci/* folder and would be ignored (or flagged as errors) by
PlatformIO itself.
Args:
example: Example name for dynamic build flags (currently unused)
additional_defines: Additional defines to merge with board defines
additional_include_dirs: Additional include directories to merge with build flags
include_platformio_section: Whether to include [platformio] section with core_dir/packages_dir
core_dir: PlatformIO core directory path
packages_dir: PlatformIO packages directory path
project_root: FastLED project root for lib_deps symlink
"""
lines: list[str] = []
# Optional [platformio] section
if include_platformio_section:
lines.append("[platformio]")
if build_cache_dir:
lines.append(f"build_cache_dir = {build_cache_dir}")
if core_dir:
lines.append(f"core_dir = {core_dir}")
if packages_dir:
lines.append(f"packages_dir = {packages_dir}")
lines.append("")
# Section header
lines.append(f"[env:{self.board_name}]")
if extra_scripts:
lines.append(f"extra_scripts = {' '.join(extra_scripts)}")
# Board identifier (skip for platforms that don't need board specification)
if not self.no_board_spec:
lines.append(f"board = {self.get_real_board_name()}")
# Optional parameters -------------------------------------------------
if self.platform:
lines.append(f"platform = {self.platform}")
if self.platform_packages:
lines.append(f"platform_packages = {self.platform_packages}")
if self.framework:
lines.append(f"framework = {self.framework}")
if self.board_build_core:
lines.append(f"board_build.core = {self.board_build_core}")
if self.board_build_mcu:
lines.append(f"board_build.mcu = {self.board_build_mcu}")
if self.board_build_filesystem_size:
lines.append(
f"board_build.filesystem_size = {self.board_build_filesystem_size}"
)
if self.board_partitions:
lines.append(f"board_partitions = {self.board_partitions}")
# Build-time flags and unflags ---------------------------------------
build_flags_elements: list[str] = []
if self.defines:
build_flags_elements.extend(f"-D{define}" for define in self.defines)
# Add additional defines
if additional_defines:
build_flags_elements.extend(f"-D{define}" for define in additional_defines)
# Use static build flags
if self.build_flags:
build_flags_elements.extend(self.build_flags)
# Add additional include directories
if additional_include_dirs:
build_flags_elements.extend(
f"-I{include_dir}" for include_dir in additional_include_dirs
)
if build_flags_elements:
# Put each build flag on a separate line for better readability and debugging
lines.append("build_flags =")
for flag in build_flags_elements:
lines.append(f" {flag}")
if self.build_unflags:
# PlatformIO accepts multiple *build_unflags* separated by spaces.
# Emit a single line for readability.
lines.append(f"build_unflags = {' '.join(self.build_unflags)}")
# Custom ESP-IDF sdkconfig override (ESP32-family boards)
if self.customsdk:
lines.append(f"custom_sdkconfig = {self.customsdk}")
# Library compatibility mode (for platforms like native that need special handling)
if self.lib_compat_mode:
lines.append(f"lib_compat_mode = {self.lib_compat_mode}")
# Library Dependency Finder mode (for enhanced dependency finding)
if self.lib_ldf_mode:
lines.append(f"lib_ldf_mode = {self.lib_ldf_mode}")
# Libraries to ignore during compilation
if self.lib_ignore:
if len(self.lib_ignore) == 1:
lines.append(f"lib_ignore = {self.lib_ignore[0]}")
else:
lines.append("lib_ignore =")
for lib in self.lib_ignore:
lines.append(f" {lib}")
# Add FastLED-specific configurations if project_root is provided
if project_root:
# Only add default lib_ldf_mode if board doesn't specify its own
current_config = "\n".join(lines)
if "lib_ldf_mode" not in current_config:
lines.append("lib_ldf_mode = chain")
lines.append("lib_archive = true")
# Build lib_deps with additional libs only (FastLED is copied to lib/FastLED)
# PlatformIO supports multiple lib_deps entries on separate lines
if additional_libs:
# Format as multi-line lib_deps for proper PlatformIO parsing
if len(additional_libs) == 1:
lines.append(f"lib_deps = {additional_libs[0]}")
else:
lines.append("lib_deps =")
for entry in additional_libs:
lines.append(f" {entry}")
return "\n".join(lines) + "\n"
# [env:sparkfun_xrp_controller]
# platform = https://github.com/maxgerhardt/platform-raspberrypi
# board = sparkfun_xrp_controller
# framework = arduino
# lib_deps = fastled/FastLED @ ^3.9.16
WEBTARGET = Board(
board_name="web",
)
# Native host compilation target using PlatformIO's "native" platform.
# This allows compiling FastLED for the host machine (Linux/macOS/Windows)
# which is useful for CI compile-tests and static analysis. We replicate
# the build flags present in ci/native/platformio.ini so that the same
# stub implementation and main-file inclusion are used.
NATIVE = Board(
board_name="native",
platform="platformio/native",
no_board_spec=True, # Native platform doesn't need a board specification
lib_compat_mode="off", # Disable library compatibility checking for native platform
build_flags=[
"-DFASTLED_STUB_IMPL", # Enable stub platform implementations
"-DFASTLED_USE_STUB_ARDUINO", # Enable Arduino stub implementations
"-DPLATFORM_NATIVE", # Enable native platform stub compilation
"-std=c++17",
"-I../../../src/platforms/stub", # Include path for Arduino.h and other stub headers (relative to project dir)
"-I../../../src", # Include path for FastLED.h and other source headers (relative to project dir)
],
)
DUE = Board(
board_name="due",
platform="atmelsam",
)
SPARKFUN_XRP_CONTROLLER_2350B = Board(
board_name="sparkfun_xrp_controller",
platform="https://github.com/maxgerhardt/platform-raspberrypi",
platform_needs_install=True,
)
APOLLO3_RED_BOARD = Board(
board_name="apollo3_red",
real_board_name="SparkFun_RedBoard_Artemis_ATP",
platform=APOLLO3_2_2_0,
platform_packages="framework-arduinoapollo3@https://github.com/sparkfun/Arduino_Apollo3#v2.2.0",
platform_needs_install=True,
)
APOLLO3_SPARKFUN_THING_PLUS_EXPLORABLE = Board(
board_name="apollo3_thing_explorable",
real_board_name="SparkFun_Thing_Plus_expLoRaBLE",
platform=APOLLO3_2_2_0,
platform_packages="framework-arduinoapollo3@https://github.com/sparkfun/Arduino_Apollo3#v2.2.0",
platform_needs_install=True,
)
ESP32DEV = Board(
board_name="esp32dev",
platform=ESP32_IDF_5_3_PIOARDUINO,
)
ESP32DEV_IDF3_3 = Board(
board_name="esp32dev_idf33",
real_board_name="esp32dev",
platform="espressif32@1.11.2",
)
ESP32DEV_IDF4_4 = Board(
board_name="esp32dev_idf44",
real_board_name="esp32dev",
platform=ESP32_IDF_4_4_LATEST,
)
GIGA_R1 = Board(
board_name="giga_r1",
platform="ststm32",
framework="arduino",
real_board_name="giga_r1_m7",
)
# ESP01 = Board(
# board_name="esp01",
# platform=ESP32_IDF_5_1_PIOARDUINO,
# )
ESP32_C2_DEVKITM_1 = Board(
board_name="esp32c2",
real_board_name="esp32-c2-devkitm-1",
platform="https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip",
framework="arduino, espidf",
)
ESP32_C3_DEVKITM_1 = Board(
board_name="esp32c3",
real_board_name="esp32-c3-devkitm-1",
platform=ESP32_IDF_5_3_PIOARDUINO,
)
ESP32_C5_DEVKITC_1 = Board(
board_name="esp32c5",
real_board_name="esp32-c5-devkitc-1",
platform=ESP32_IDF_5_5_PIOARDUINO,
)
ESP32_C6_DEVKITC_1 = Board(
board_name="esp32c6",
real_board_name="esp32-c6-devkitc-1",
platform=ESP32_IDF_5_3_PIOARDUINO,
)
ESP32_S3_DEVKITC_1 = Board(
board_name="esp32s3",
real_board_name="esp32-s3-devkitc-1",
platform=ESP32_IDF_5_4_PIOARDUINO,
framework="arduino",
board_partitions="huge_app.csv",
build_unflags=["-DFASTLED_RMT5=0", "-DFASTLED_RMT5"],
)
ESP32_S2_DEVKITM_1 = Board(
board_name="esp32s2",
real_board_name="lolin_s2_mini ",
board_build_mcu="esp32s2",
platform=ESP32_IDF_5_3_PIOARDUINO,
)
ESP32_UPESY_WROOM = Board(
board_name="upesy_wroom",
real_board_name="upesy_wroom",
platform="espressif32",
)
ESP32H2 = Board(
board_name="esp32h2",
real_board_name="esp32-h2-devkitm-1",
platform_needs_install=True, # Install platform package to get the boards
platform=ESP32_IDF_5_3_PIOARDUINO,
)
ESP32_P4 = Board(
board_name="esp32p4",
real_board_name="esp32-p4-evboard",
platform_needs_install=True, # Install platform package to get the boards
platform="https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip",
)
ADA_FEATHER_NRF52840_SENSE = Board(
board_name="adafruit_feather_nrf52840_sense",
platform="nordicnrf52",
)
XIAOBLESENSE_ADAFRUIT_NRF52 = Board(
board_name="xiaoblesense_adafruit",
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
platform_needs_install=True, # Install platform package to get the boards
)
# Alias: handle common misspelling without the trailing 't'
XIAOBLESENSE_ADAFRUI_ALIAS = Board(
board_name="xiaoblesense_adafrui", # missing 't'
real_board_name="xiaoblesense_adafruit", # map to the correct board name
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
platform_needs_install=True,
)
XIAOBLESENSE_NRF52 = Board(
board_name="xiaoblesense",
real_board_name="xiaoble_adafruit",
platform="https://github.com/maxgerhardt/platform-nordicnrf52",
platform_needs_install=True,
)
# Correct nRF52840 DK board definition
# The Nordic nRF52840 DK is directly supported by the default PlatformIO
# `nordicnrf52` platform under the board name `nrf52840_dk`, so we don't
# need a custom platform package or extra installation steps. Point the
# Board definition at the stock platform and use the canonical board name.
# This fixes compilation failures introduced during the build-system
# migration where the board was temporarily mapped to the XIAO variant.
NRF52840 = Board(
board_name="nrf52840_dk",
real_board_name="nrf52840_dk_adafruit", # Use Adafruit BSP variant which includes full Nordic SDK headers
platform="nordicnrf52",
framework="arduino",
platform_needs_install=False,
platform_packages="framework-arduinoadafruitnrf52@^1.10601.0",
defines=[
"FASTLED_USE_COMPILE_TESTS=0",
],
board_build_core="nRF5", # Ensure correct core directory
)
RPI_PICO = Board(
board_name="rpipico",
platform="https://github.com/maxgerhardt/platform-raspberrypi.git",
platform_needs_install=True, # Install platform package to get the boards
platform_packages="framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git",
framework="arduino",
board_build_core="earlephilhower",
board_build_filesystem_size="0.5m",
)
RPI_PICO2 = Board(
board_name="rpipico2",
real_board_name="rpipico", # Use the existing Pico board definition until PlatformIO adds native Pico 2 support
platform="https://github.com/maxgerhardt/platform-raspberrypi.git",
platform_needs_install=True, # Install platform package to get the boards
platform_packages="framework-arduinopico@https://github.com/earlephilhower/arduino-pico.git",
framework="arduino",
board_build_core="earlephilhower",
board_build_filesystem_size="0.5m",
)
BLUEPILL = Board(
board_name="bluepill",
real_board_name="bluepill_f103c8",
platform="ststm32",
)
# maple_mini_b20
MAPLE_MINI = Board(
board_name="maple_mini",
real_board_name="maple_mini_b20",
platform="ststm32",
)
HY_TINYSTM103TB = Board(
board_name="hy_tinystm103tb",
platform="ststm32",
)
ATTINY88 = Board(
board_name="attiny88",
platform="atmelavr",
)
# ATtiny1604
ATTINY1604 = Board(
board_name="ATtiny1604",
platform="atmelmegaavr",
)
# attiny4313
ATTINY4313 = Board(
board_name="attiny4313",
platform="atmelavr",
)
ATTINY1616 = Board(
board_name="ATtiny1616",
platform="atmelmegaavr",
)
UNO_R4_WIFI = Board(
board_name="uno_r4_wifi",
platform="renesas-ra",
# I2S library auto-excluded due to missing r_i2s_api.h header in Arduino Renesas framework
)
NANO_EVERY = Board(
board_name="nano_every",
platform="atmelmegaavr",
)
ESP32DEV_I2S = Board(
board_name="esp32dev_i2s",
real_board_name="esp32dev",
platform=ESP32_IDF_4_4_LATEST,
)
ESP32S3_RMT51 = Board(
board_name="esp32rmt_51",
real_board_name="esp32-s3-devkitc-1",
platform_needs_install=True,
platform=ESP32_IDF_5_3_PIOARDUINO,
defines=[
"FASTLED_RMT5=1",
],
)
# Teensy boards
TEENSY_LC = Board(
board_name="teensylc",
platform="teensy",
framework="arduino",
)
TEENSY30 = Board(
board_name="teensy30",
platform="teensy",
framework="arduino",
)
TEENSY31 = Board(
board_name="teensy31",
platform="teensy",
framework="arduino",
)
TEENSY40 = Board(
board_name="teensy40",
platform="teensy",
framework="arduino",
)
TEENSY41 = Board(
board_name="teensy41",
platform="teensy",
framework="arduino",
)
# Basic Arduino boards
UNO = Board(
board_name="uno",
platform="atmelavr",
framework="arduino",
)
YUN = Board(
board_name="yun",
platform="atmelavr",
framework="arduino",
)
DIGIX = Board(
board_name="digix",
real_board_name="due", # Digix is Arduino Due compatible
platform="atmelsam",
framework="arduino",
)
# ESP8266 boards
ESP01 = Board(
board_name="esp01",
platform="espressif8266",
framework="arduino",
)
# ATtiny boards
ATTINY85 = Board(
board_name="attiny85",
platform="atmelavr",
framework="arduino",
)
# Seeed XIAO ESP32S3 board – same platform, needs FASTLED_RMT5 macro removal
XIAO_ESP32S3 = Board(
board_name="seeed_xiao_esp32s3",
real_board_name="seeed_xiao_esp32s3",
platform=ESP32_IDF_5_4_PIOARDUINO,
board_partitions="huge_app.csv",
defines=None,
build_unflags=["-DFASTLED_RMT5=0", "-DFASTLED_RMT5"],
)
# STM32F4 Black Pill board - addresses GitHub issue #726
BLACKPILL = Board(
board_name="blackpill",
real_board_name="blackpill_f411ce",
platform="ststm32",
)
# Silicon Labs MGM240S boards (Arduino Nano Matter, SparkFun Thing Plus Matter)
# Uses Silicon Labs EFM32 platform with Arduino framework support
# Based on EFR32MG24 SoC with ARM Cortex-M33 @ 78MHz
MGM240S = Board(
board_name="mgm240",
real_board_name="sparkfun_thingplusmatter",
platform="https://github.com/maxgerhardt/platform-siliconlabsefm32/archive/refs/heads/silabs-arduino.zip",
platform_needs_install=True,
framework="arduino",
)
def _make_board_map(boards: list[Board]) -> dict[str, Board]:
# make board map, but assert on duplicate board names
board_map: dict[str, Board] = {}
for board in boards:
assert board.board_name not in board_map, (
f"Duplicate board name: {board.board_name}"
)
board_map[board.board_name] = board
return board_map
_BOARD_MAP: dict[str, Board] = _make_board_map(ALL)
def create_board(board_name: str, no_project_options: bool = False) -> Board:
board: Board
if no_project_options:
board = Board(board_name=board_name, add_board_to_all=False)
if board_name not in _BOARD_MAP:
# empty board without any special overrides, assume platformio will know what to do with it.
board = Board(board_name=board_name, add_board_to_all=False)
else:
board = _BOARD_MAP[board_name]
return board.clone()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/bluepill.json
+47
View File
@@ -0,0 +1,47 @@
{
"build": {
"core": "maple",
"cpu": "cortex-m3",
"extra_flags": "-DSTM32F103C8 -DSTM32F1 -DARDUINO_GENERIC_STM32F103C8 -DARDUINO_ARCH_STM32",
"f_cpu": "72000000L",
"hwids": [
[
"0x1EAF",
"0x0003"
]
],
"ldscript": "jtag_c8.ld",
"mcu": "stm32f103c8",
"product_line": "STM32F103xx",
"variant": "STM32F103C8"
},
"connectivity": [
"can"
],
"debug": {
"jlink_device": "STM32F103C8",
"openocd_board": "stm32f103c8_blue_pill",
"openocd_target": "stm32f1x",
"svd_path": "STM32F103.svd"
},
"frameworks": [
"arduino",
"stm32cube",
"libopencm3"
],
"name": "Blue Pill STM32F103C8",
"upload": {
"maximum_ram_size": 20480,
"maximum_size": 65536,
"protocol": "stlink",
"protocols": [
"stlink",
"dfu",
"serial",
"blackmagic",
"jlink"
]
},
"url": "https://stm32-base.org/boards/STM32F103C8T6-Blue-Pill",
"vendor": "Generic"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/esp32-c2-devkitm-1.json
+35
View File
@@ -0,0 +1,35 @@
{
"build": {
"core": "esp32",
"f_cpu": "120000000L",
"f_flash": "60000000L",
"flash_mode": "qio",
"mcu": "esp32c2",
"variant": "esp32c2"
},
"connectivity": [
"wifi"
],
"debug": {
"openocd_target": "esp32c2.cfg"
},
"frameworks": [
"arduino",
"espidf"
],
"name": "Espressif ESP32-C2-DevKitM-1",
"upload": {
"flash_size": "4MB",
"maximum_ram_size": 278528,
"maximum_size": 4194304,
"require_upload_port": true,
"speed": 460800
},
"espidf": {
"custom_sdkconfig": [
"CONFIG_IDF_TARGET=\"esp32c2\""
]
},
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp8684/esp8684-devkitm-1/user_guide.html",
"vendor": "Espressif"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/esp32-c6-devkitc-1.json
+31
View File
@@ -0,0 +1,31 @@
{
"build": {
"core": "esp32",
"f_cpu": "160000000L",
"f_flash": "80000000L",
"flash_mode": "qio",
"mcu": "esp32c6",
"variant": "esp32c6"
},
"connectivity": [
"bluetooth",
"wifi"
],
"debug": {
"openocd_target": "esp32c6.cfg"
},
"frameworks": [
"arduino",
"espidf"
],
"name": "Espressif ESP32-C6-DevKitC-1",
"upload": {
"flash_size": "8MB",
"maximum_ram_size": 327680,
"maximum_size": 8388608,
"require_upload_port": true,
"speed": 460800
},
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32c6/esp32-c6-devkitc-1/index.html",
"vendor": "Espressif"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/esp32-h2-devkitm-1.json
+31
View File
@@ -0,0 +1,31 @@
{
"build": {
"core": "esp32",
"f_cpu": "96000000L",
"f_flash": "64000000L",
"f_image": "48000000L",
"flash_mode": "qio",
"mcu": "esp32h2",
"variant": "esp32h2"
},
"connectivity": [
"bluetooth"
],
"debug": {
"openocd_target": "esp32h2.cfg"
},
"frameworks": [
"arduino",
"espidf"
],
"name": "Espressif ESP32-H2-DevKit",
"upload": {
"flash_size": "4MB",
"maximum_ram_size": 327680,
"maximum_size": 4194304,
"require_upload_port": true,
"speed": 460800
},
"url": "https://docs.espressif.com/projects/espressif-esp-dev-kits/en/latest/esp32h2/esp32-h2-devkitm-1/index.html",
"vendor": "Espressif"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/giga.json
+61
View File
@@ -0,0 +1,61 @@
{
"build": {
"arduino": {
"ldscript": "linker_script.ld",
"flash_layout": "100_0"
},
"extra_flags": "-DARDUINO_GIGA -DARDUINO_GIGA_PINS -DGIGA_PINS -DGIGA -DCM4_BINARY_START=0x60000000 -DCM4_BINARY_END=0x60040000 -DCM4_RAM_END=0x60080000",
"core": "arduino",
"cpu": "cortex-m7",
"f_cpu": "480000000L",
"mcu": "stm32h747xih6",
"variant": "GIGA",
"product_line": "STM32H747xx",
"hwids": [
[
"0x2341",
"0x0266"
],
[
"0x2341",
"0x0366"
],
[
"0x2341",
"0x0466"
]
]
},
"connectivity": [
"bluetooth",
"wifi"
],
"debug": {
"jlink_device": "STM32H747XI_M7",
"openocd_target": "stm32h7x_dual_bank"
},
"frameworks": [
"arduino"
],
"name": "Arduino Giga R1 Wifi",
"upload": {
"_maximum_ram_size": 294248,
"_maximum_size": 1048576,
"maximum_ram_size": 523624,
"maximum_size": 1966080,
"protocol": "dfu",
"protocols": [
"cmsis-dap",
"dfu",
"jlink",
"stlink",
"mbed"
],
"require_upload_port": true,
"use_1200bps_touch": true,
"wait_for_upload_port": true,
"offset_address": "0x08100000"
},
"url": "https://docs.arduino.cc/hardware/giga-r1-wifi",
"vendor": "Arduino"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/nrf52840_dk.json
+51
View File
@@ -0,0 +1,51 @@
{
"build": {
"arduino":{
"ldscript": "nrf52_xxaa.ld"
},
"core": "nRF5",
"cpu": "cortex-m4",
"extra_flags": "-DARDUINO_NRF52_DK",
"f_cpu": "64000000L",
"mcu": "nrf52840",
"variant": "nRF52DK",
"zephyr": {
"variant": "nrf52840dk_nrf52840"
}
},
"connectivity": [
"bluetooth"
],
"debug": {
"default_tools": [
"jlink"
],
"jlink_device": "nRF52840_xxAA",
"onboard_tools": [
"cmsis-dap",
"jlink"
],
"svd_path": "nrf52840.svd"
},
"frameworks": [
"arduino",
"mbed",
"zephyr"
],
"name": "Nordic nRF52840-DK",
"upload": {
"maximum_ram_size": 262144,
"maximum_size": 1048576,
"protocol": "jlink",
"protocols": [
"jlink",
"nrfjprog",
"stlink",
"blackmagic",
"cmsis-dap",
"mbed"
]
},
"url": "https://os.mbed.com/platforms/Nordic-nRF52840-DK/",
"vendor": "Nordic"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/rpipico.json
+56
View File
@@ -0,0 +1,56 @@
{
"build": {
"arduino": {
"earlephilhower": {
"boot2_source": "boot2_w25q080_2_padded_checksum.S",
"usb_vid": "0x2E8A",
"usb_pid": "0x000A"
}
},
"core": "earlephilhower",
"cpu": "cortex-m0plus",
"extra_flags": "-DARDUINO_RASPBERRY_PI_PICO -DARDUINO_ARCH_RP2040 -DUSBD_MAX_POWER_MA=250 ",
"f_cpu": "133000000L",
"hwids": [
[
"0x2E8A",
"0x00C0"
],
[
"0x2E8A",
"0x000A"
]
],
"mcu": "rp2040",
"variant": "rpipico"
},
"debug": {
"jlink_device": "RP2040_M0_0",
"openocd_target": "rp2040.cfg",
"svd_path": "rp2040.svd"
},
"frameworks": [
"arduino"
],
"name": "Pico",
"upload": {
"maximum_ram_size": 262144,
"maximum_size": 2097152,
"require_upload_port": true,
"native_usb": true,
"use_1200bps_touch": true,
"wait_for_upload_port": false,
"protocol": "picotool",
"protocols": [
"blackmagic",
"cmsis-dap",
"jlink",
"raspberrypi-swd",
"picotool",
"picoprobe",
"pico-debug"
]
},
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
"vendor": "Raspberry Pi"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/rpipico2.json
+56
View File
@@ -0,0 +1,56 @@
{
"build": {
"arduino": {
"earlephilhower": {
"boot2_source": "boot2_generic_03h_2_padded_checksum.S",
"usb_vid": "0x2E8A",
"usb_pid": "0x000B"
}
},
"core": "earlephilhower",
"cpu": "cortex-m33",
"extra_flags": "-D ARDUINO_RASPBERRY_PI_PICO_2 -DARDUINO_ARCH_RP2350 -DUSBD_MAX_POWER_MA=250",
"f_cpu": "133000000L",
"hwids": [
[
"0x2E8A",
"0x00C0"
],
[
"0x2E8A",
"0x000B"
]
],
"mcu": "rp2350",
"variant": "rpipico2"
},
"debug": {
"jlink_device": "RP2350_0",
"openocd_target": "rp2350.cfg",
"svd_path": "rp2350.svd"
},
"frameworks": [
"arduino"
],
"name": "Pico 2",
"upload": {
"maximum_ram_size": 524288,
"maximum_size": 2097152,
"require_upload_port": true,
"native_usb": true,
"use_1200bps_touch": true,
"wait_for_upload_port": false,
"protocol": "picotool",
"protocols": [
"blackmagic",
"cmsis-dap",
"jlink",
"raspberrypi-swd",
"picotool",
"picoprobe",
"pico-debug"
]
},
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
"vendor": "Raspberry Pi"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/boards/rpipicow.json
+56
View File
@@ -0,0 +1,56 @@
{
"build": {
"arduino": {
"earlephilhower": {
"boot2_source": "boot2_w25q080_2_padded_checksum.S",
"usb_vid": "0x2E8A",
"usb_pid": "0xF00A"
}
},
"core": "earlephilhower",
"cpu": "cortex-m0plus",
"extra_flags": "-DARDUINO_RASPBERRY_PI_PICO_W -DARDUINO_ARCH_RP2040 -DUSBD_MAX_POWER_MA=250 ",
"f_cpu": "133000000L",
"hwids": [
[
"0x2E8A",
"0x00C0"
],
[
"0x2E8A",
"0xF00A"
]
],
"mcu": "rp2040",
"variant": "rpipicow"
},
"debug": {
"jlink_device": "RP2040_M0_0",
"openocd_target": "rp2040.cfg",
"svd_path": "rp2040.svd"
},
"frameworks": [
"arduino"
],
"name": "Pico W",
"upload": {
"maximum_ram_size": 262144,
"maximum_size": 2097152,
"require_upload_port": true,
"native_usb": true,
"use_1200bps_touch": true,
"wait_for_upload_port": false,
"protocol": "picotool",
"protocols": [
"blackmagic",
"cmsis-dap",
"jlink",
"raspberrypi-swd",
"picotool",
"picoprobe",
"pico-debug"
]
},
"url": "https://www.raspberrypi.org/products/raspberry-pi-pico/",
"vendor": "Raspberry Pi"
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/build_api.py
+643
View File
@@ -0,0 +1,643 @@
#!/usr/bin/env python3
"""
FastLED Unified Build API
This module provides a unified build system for both unit tests and examples,
centralizing PCH and library building while providing a simple, elegant API.
Key Features:
- Unified API for both unit tests and examples
- Automatic PCH header building and reuse
- Automatic libfastled.a building and reuse
- Organized build directories under .build/
- Efficient multiple target building
- Simple, elegant interface
Usage:
# Unit test building
unit_builder = BuildAPI(
config_file="ci/build_unit.toml",
build_dir=".build/unit",
build_type=BuildType.UNIT_TEST
)
unit_builder.build_targets(["test_json.cpp", "test_color.cpp"])
# Example building
example_builder = BuildAPI(
config_file="ci/build_example.toml",
build_dir=".build/examples",
build_type=BuildType.EXAMPLE
)
example_builder.build_targets(["Blink.ino", "DemoReel100.ino"])
"""
import os
import shutil
import subprocess
import tempfile
from dataclasses import dataclass
from enum import Enum
from pathlib import Path
from typing import Any, Dict, List, Optional, Sequence, Set, Tuple
from ci.ci.fingerprint_cache import FingerprintCache
from ci.compiler.clang_compiler import (
BuildFlags,
Compiler,
CompilerOptions,
LibarchiveOptions,
)
from ci.compiler.test_example_compilation import get_fastled_core_sources
from ci.util.paths import PROJECT_ROOT
from ci.util.running_process import subprocess_run
class BuildType(Enum):
"""Build type enumeration"""
UNIT_TEST = "unit_test"
EXAMPLE = "example"
@dataclass
class BuildTarget:
"""Represents a single build target (test file or example)"""
source_file: Path
name: str
output_dir: Path
def get_output_executable(self) -> Path:
"""Get the expected output executable path"""
if self.source_file.suffix == ".cpp":
# Unit test: test_name -> test_name.exe
base_name = self.source_file.stem
return self.output_dir / f"{base_name}.exe"
else:
# Example: Blink.ino -> Blink.exe
base_name = self.source_file.stem
return self.output_dir / f"{base_name}.exe"
@dataclass
class BuildResult:
"""Result of a build operation"""
target: BuildTarget
success: bool
message: str
build_time: float
class BuildAPI:
"""
Unified build API for both unit tests and examples.
This class centralizes all build operations including PCH building,
library creation, and target compilation. It provides a simple,
elegant interface while handling the complexity internally.
"""
def __init__(
self,
build_flags_toml: str | Path,
build_dir: str | Path,
build_type: BuildType,
use_pch: bool = True,
parallel: bool = True,
clean: bool = False,
):
"""
Initialize the build API.
Args:
build_flags_toml: Path to build_flags.toml file (build_unit.toml or build_example.toml)
build_dir: Directory for build outputs (e.g., .build/unit or .build/examples)
build_type: Type of build (UNIT_TEST or EXAMPLE)
use_pch: Whether to use precompiled headers
parallel: Whether to enable parallel compilation
clean: Whether to force a clean build
"""
self.build_flags_toml = Path(build_flags_toml)
self.build_dir = Path(build_dir)
self.build_type = build_type
self.use_pch = use_pch
self.parallel = parallel
self.clean = clean
# Ensure build directory exists
self.build_dir.mkdir(parents=True, exist_ok=True)
# Set up subdirectories for organization
self.artifacts_dir = self.build_dir / "artifacts" # PCH, lib files
self.targets_dir = self.build_dir / "targets" # Individual target builds
self.cache_dir = self.build_dir / "cache" # Fingerprint caches
for directory in [self.artifacts_dir, self.targets_dir, self.cache_dir]:
directory.mkdir(parents=True, exist_ok=True)
# Initialize internal state
self._compiler: Optional[Compiler] = None
self._pch_file: Optional[Path] = None
self._library_file: Optional[Path] = None
self._fingerprint_cache: Optional[FingerprintCache] = None
# Parse build flags from TOML file
self._parse_build_flags()
# Create compiler from TOML configuration
self._create_compiler()
def _parse_build_flags(self) -> None:
"""Parse build flags from TOML file"""
if not self.build_flags_toml.exists():
raise RuntimeError(
f"Build flags TOML file not found: {self.build_flags_toml}"
)
print(f"[BUILD API] Parsing build flags from {self.build_flags_toml}")
# Parse build flags from TOML
self.build_flags = BuildFlags.from_toml_file(
self.build_flags_toml,
quick_build=True, # Use quick build mode for faster iteration
strict_mode=False,
)
# Initialize fingerprint cache
cache_file = self.cache_dir / "fingerprint_cache.json"
self._fingerprint_cache = FingerprintCache(cache_file)
def _create_compiler(self) -> None:
"""Create compiler from TOML configuration ONLY - no inline settings allowed"""
from ci.compiler.clang_compiler import create_compiler_options_from_toml
# ALL settings must come from the TOML file - NO hardcoded/inline settings allowed
print(f"[BUILD API] Creating compiler from TOML configuration")
# Set up PCH output path
pch_output_path: Optional[str] = None
if self.use_pch:
pch_output_path = str(
self.artifacts_dir / f"{self.build_type.value}_pch.hpp.pch"
)
# Use the proper function to create CompilerOptions from TOML - NO inline settings
self.compiler_settings = create_compiler_options_from_toml(
toml_path=self.build_flags_toml,
include_path=str(PROJECT_ROOT / "src"), # Base include path only
quick_build=True, # Use quick build mode for faster iteration
strict_mode=False,
# Pass PCH and parallel settings from constructor
use_pch=self.use_pch,
pch_output_path=pch_output_path,
parallel=self.parallel,
)
# Create compiler instance with TOML-loaded settings
self._compiler = Compiler(self.compiler_settings, self.build_flags)
def _ensure_pch_built(self) -> bool:
"""
Ensure PCH header is built and ready.
Returns:
bool: True if PCH is ready, False otherwise
"""
if not self.use_pch or not self._compiler:
return True
print(f"[BUILD API] Building PCH header for {self.build_type.value}")
# Check if we need to rebuild PCH
if not self.clean and self._pch_file and self._pch_file.exists():
# TODO: Add proper dependency checking here
print(f"[BUILD API] Reusing existing PCH: {self._pch_file}")
return True
# Build PCH
success = self._compiler.create_pch_file()
if success and self.compiler_settings.pch_output_path:
self._pch_file = Path(self.compiler_settings.pch_output_path)
print(f"[BUILD API] PCH built successfully: {self._pch_file}")
else:
print(f"[BUILD API] PCH build failed")
return success
def _ensure_library_built(self) -> bool:
"""
Ensure libfastled.a is built and ready.
Returns:
bool: True if library is ready, False otherwise
"""
if not self._compiler:
return False
lib_name = f"libfastled_{self.build_type.value}.a"
self._library_file = self.artifacts_dir / lib_name
print(f"[BUILD API] Building FastLED library for {self.build_type.value}")
# Check if we need to rebuild library
if not self.clean and self._library_file.exists():
# TODO: Add proper dependency checking here
print(f"[BUILD API] Reusing existing library: {self._library_file}")
return True
# Prefer UNITY chunk build for faster library creation
try:
unity_dir = self.artifacts_dir / "unity"
unity_dir.mkdir(parents=True, exist_ok=True)
# Collect FastLED core sources (consistent with example compilation)
all_sources = [
p for p in get_fastled_core_sources() if p.name != "stub_main.cpp"
]
if not all_sources:
print("[BUILD API] No FastLED sources found for unity build")
return False
# Deterministic sort relative to project root
project_root = PROJECT_ROOT
def sort_key(p: Path) -> str:
try:
rel = p.relative_to(project_root)
except Exception:
rel = p
return rel.as_posix()
all_sources = sorted(all_sources, key=sort_key)
# Partition into chunks (1..4 typical)
total = len(all_sources)
max_chunks = min(4, max(1, (os.cpu_count() or 2) // 2))
chunks = max_chunks if total >= max_chunks else max(1, total)
# Prepare compile options (PCH off for unity)
compile_opts = CompilerOptions(
include_path=self._compiler.settings.include_path,
compiler=self._compiler.settings.compiler,
defines=self._compiler.settings.defines,
std_version=self._compiler.settings.std_version,
compiler_args=self._compiler.settings.compiler_args,
use_pch=False,
additional_flags=["-c"],
parallel=self.parallel,
)
# Compute chunk sizes
base = total // chunks
rem = total % chunks
start = 0
unity_objects: list[Path] = []
for i in range(chunks):
size = base + (1 if i < rem else 0)
end = start + size
group = all_sources[start:end]
start = end
if not group:
continue
unity_cpp = unity_dir / f"unity{i + 1}.cpp"
# Use synchronous path to simplify error handling here
result = self._compiler._compile_unity_sync(
compile_opts,
group,
unity_output_path=unity_cpp,
)
if not result.ok:
print(f"[BUILD API] Unity chunk {i + 1} failed: {result.stderr}")
return False
unity_objects.append(unity_cpp.with_suffix(".o"))
if not unity_objects:
print("[BUILD API] No unity objects produced")
return False
# Archive unity objects into the final library
archive_result = self._compiler.create_archive(
unity_objects, self._library_file, LibarchiveOptions()
).result()
if not archive_result.ok:
print(f"[BUILD API] Archive creation failed: {archive_result.stderr}")
return False
print(
f"[BUILD API] Library built successfully (UNITY): {self._library_file}"
)
return True
except Exception as e:
print(f"[BUILD API] UNITY library build failed with exception: {e}")
return False
def _create_static_library(
self, object_files: List[Path], output_lib: Path
) -> bool:
"""
Create static library from object files.
Args:
object_files: List of object files to archive
output_lib: Output library file path
Returns:
bool: True if successful, False otherwise
"""
if not object_files:
return False
# Use archiver from build flags
archiver_cmd = self.build_flags.tools.archiver
archive_flags = self.build_flags.archive.flags
# Build archiver command
cmd = (
archiver_cmd
+ [archive_flags]
+ [str(output_lib)]
+ [str(f) for f in object_files]
)
print(f"[BUILD API] Creating static library: {' '.join(cmd)}")
# Execute archiver command using stdout-pumping wrapper
try:
_completed = subprocess_run(
command=cmd,
cwd=None,
check=True,
timeout=900,
enable_stack_trace=False,
)
return True
except subprocess.CalledProcessError as e:
print(f"[BUILD API] Archiver failed: {e}")
print(f"[BUILD API] Archiver stdout: {e.stdout}")
print(f"[BUILD API] Archiver stderr: {e.stderr}")
return False
def build_targets(self, target_files: Sequence[str | Path]) -> List[BuildResult]:
"""
Build multiple targets efficiently.
Args:
target_files: List of source files to build
Returns:
List[BuildResult]: Results for each target
"""
print(
f"[BUILD API] Building {len(target_files)} targets for {self.build_type.value}"
)
# Ensure shared artifacts are built first
if not self._ensure_pch_built():
return [
BuildResult(
target=BuildTarget(Path(f), Path(f).stem, self.targets_dir),
success=False,
message="PCH build failed",
build_time=0.0,
)
for f in target_files
]
if not self._ensure_library_built():
return [
BuildResult(
target=BuildTarget(Path(f), Path(f).stem, self.targets_dir),
success=False,
message="Library build failed",
build_time=0.0,
)
for f in target_files
]
# Build individual targets
results: List[BuildResult] = []
for target_file in target_files:
result = self._build_single_target(Path(target_file))
results.append(result)
return results
def _build_single_target(self, target_file: Path) -> BuildResult:
"""
Build a single target.
Args:
target_file: Source file to build
Returns:
BuildResult: Result of the build
"""
import time
start_time = time.time()
# Create build target
target_dir = self.targets_dir / target_file.stem
target_dir.mkdir(parents=True, exist_ok=True)
target = BuildTarget(
source_file=target_file, name=target_file.stem, output_dir=target_dir
)
print(f"[BUILD API] Building target: {target.name}")
try:
# Compile and link the target
success = self._compile_and_link_target(target)
build_time = time.time() - start_time
message = "Build successful" if success else "Build failed"
return BuildResult(
target=target, success=success, message=message, build_time=build_time
)
except Exception as e:
build_time = time.time() - start_time
return BuildResult(
target=target,
success=False,
message=f"Build error: {e}",
build_time=build_time,
)
def _compile_and_link_target(self, target: BuildTarget) -> bool:
"""
Compile and link a single target.
Args:
target: Target to build
Returns:
bool: True if successful, False otherwise
"""
if not self._compiler:
return False
# Compile target source file
obj_file = target.output_dir / f"{target.name}.o"
if target.source_file.suffix == ".cpp":
success = self._compiler.compile_cpp_file(target.source_file, obj_file)
else:
# For .ino files
success = self._compiler.compile_ino_file(target.source_file, obj_file)
if not success:
print(f"[BUILD API] Failed to compile {target.source_file}")
return False
# Link with library to create executable
exe_file = target.get_output_executable()
success = self._link_executable([obj_file], exe_file)
if not success:
print(f"[BUILD API] Failed to link {exe_file}")
return False
print(f"[BUILD API] Successfully built {exe_file}")
return True
def _link_executable(self, object_files: List[Path], output_exe: Path) -> bool:
"""
Link object files into an executable.
Args:
object_files: Object files to link
output_exe: Output executable path
Returns:
bool: True if successful, False otherwise
"""
if not self._library_file or not self._library_file.exists():
print(f"[BUILD API] Library not available for linking")
return False
# Build linker command
linker_cmd = self.build_flags.tools.linker
link_flags = self.build_flags.link_flags
cmd = (
linker_cmd
+ link_flags
+ [str(f) for f in object_files]
+ [str(self._library_file)]
+ ["-o", str(output_exe)]
)
cmd_str = subprocess.list2cmdline(cmd)
print(f"[BUILD API] Linking: {cmd_str}")
# Execute linker using stdout-pumping wrapper
try:
_completed = subprocess_run(
command=cmd,
cwd=None,
check=True,
timeout=900,
enable_stack_trace=False,
)
return True
except subprocess.CalledProcessError as e:
print(f"[BUILD API] Linker failed: {e}")
print(f"[BUILD API] Linker stdout: {e.stdout}")
print(f"[BUILD API] Linker stderr: {e.stderr}")
return False
def get_build_info(self) -> Dict[str, Any]:
"""Get information about the current build configuration"""
return {
"build_flags_toml": str(self.build_flags_toml),
"build_dir": str(self.build_dir),
"build_type": self.build_type.value,
"use_pch": self.use_pch,
"parallel": self.parallel,
"pch_file": str(self._pch_file) if self._pch_file else None,
"library_file": str(self._library_file) if self._library_file else None,
}
def clean_build_artifacts(self) -> None:
"""Clean all build artifacts"""
print(f"[BUILD API] Cleaning build artifacts in {self.build_dir}")
if self.build_dir.exists():
shutil.rmtree(self.build_dir)
self.build_dir.mkdir(parents=True, exist_ok=True)
# Recreate subdirectories
for directory in [self.artifacts_dir, self.targets_dir, self.cache_dir]:
directory.mkdir(parents=True, exist_ok=True)
# Reset internal state
self._pch_file = None
self._library_file = None
if self._fingerprint_cache:
# Clear the cache by resetting it
cache_file = self.cache_dir / "fingerprint_cache.json"
self._fingerprint_cache = FingerprintCache(cache_file)
def create_unit_test_builder(
build_dir: str | Path = ".build/unit",
use_pch: bool = True,
parallel: bool = True,
clean: bool = False,
) -> BuildAPI:
"""
Create a BuildAPI instance for unit tests.
Args:
build_dir: Build directory for unit test outputs
use_pch: Whether to use precompiled headers
parallel: Whether to enable parallel compilation
clean: Whether to force a clean build
Returns:
BuildAPI: Configured build API for unit tests
"""
build_flags_toml = PROJECT_ROOT / "ci" / "build_unit.toml"
return BuildAPI(
build_flags_toml=build_flags_toml,
build_dir=build_dir,
build_type=BuildType.UNIT_TEST,
use_pch=use_pch,
parallel=parallel,
clean=clean,
)
def create_example_builder(
build_dir: str | Path = ".build/examples",
use_pch: bool = True,
parallel: bool = True,
clean: bool = False,
) -> BuildAPI:
"""
Create a BuildAPI instance for examples.
Args:
build_dir: Build directory for example outputs
use_pch: Whether to use precompiled headers
parallel: Whether to enable parallel compilation
clean: Whether to force a clean build
Returns:
BuildAPI: Configured build API for examples
"""
build_flags_toml = PROJECT_ROOT / "ci" / "build_example.toml"
return BuildAPI(
build_flags_toml=build_flags_toml,
build_dir=build_dir,
build_type=BuildType.EXAMPLE,
use_pch=use_pch,
parallel=parallel,
clean=clean,
)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/build_commands.toml
+32
View File
@@ -0,0 +1,32 @@
[all]
# Universal defines and compiler flags for compile-commands generation
defines = [
"STUB_PLATFORM",
"__EMSCRIPTEN__"
]
compiler_flags = [
"-std=gnu++17",
"-fpermissive",
"-Wall",
"-Wextra"
]
# Include flags are passed verbatim; use -I and -isystem as needed
include_flags = [
"-Isrc",
"-isystemsrc/platforms/stub"
]
[tools]
# Tool configuration required by BuildFlags.parse
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
linker = ["uv", "run", "python", "-m", "ziglang", "c++"]
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"]
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
[archive]
# Required by BuildFlags.parse; not used for compile-commands generation
flags = "rcsD"
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/build_example.toml
+181
View File
@@ -0,0 +1,181 @@
# ================================================================================================
# FastLED Example Build Flags Configuration
# ================================================================================================
#
# This file centralizes ALL compilation flags used by example compilation to ensure
# they remain synchronized and compatible across the entire build system.
#
# Examples use the STUB platform for Arduino-compatible functionality without hardware
# dependencies, but do NOT need unit test specific flags like FASTLED_FORCE_NAMESPACE.
[tools]
# Build tool configuration - Full compiler command with arguments
# CRITICAL: Must use 'uv run python' for subprocess execution (subprocess doesn't inherit uv environment)
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
linker = ["uv", "run", "python", "-m", "ziglang", "c++"] # Separate linker config for consistency
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"] # Ziglang archiver for unified toolchain
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
[all]
# Universal compilation flags (compiler command handled separately)
compiler_flags = [
"-std=gnu++17",
"-fpermissive",
"-Wall",
"-Wextra",
"-Wno-deprecated-register", # Suppress 'register' keyword warnings for C++17 compatibility
"-Wno-backslash-newline-escape", # Suppress backslash newline escape warnings
"-fno-exceptions",
"-fno-rtti",
]
defines = [
"STUB_PLATFORM",
# NOTE: Examples do NOT get FASTLED_FORCE_NAMESPACE=1 - they use global namespace
]
include_flags = [
"-I.",
"-Isrc",
]
[windows]
# ================================================================================================
# WINDOWS-SPECIFIC BUILD FLAGS
# ================================================================================================
# Comprehensive Windows build configuration using Zig's bundled Clang with GNU target
# C compiler flags for Windows
cc_flags = [
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
"-fuse-ld=lld", # Use Zig's bundled LLD linker
"-std=c17", # C17 standard
"-Wall",
"-Wextra",
]
# C++ compiler flags for Windows
cpp_flags = [
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
"-fuse-ld=lld-link", # Use lld-link (Windows MSVC-compatible linker)
"-std=c++17", # C++17 standard
"-Wall",
"-Wextra",
"-fno-exceptions",
"-fno-rtti",
# Fix Arduino INPUT macro conflict with Windows headers
"-DNOMINMAX", # Prevent Windows min/max macros
"-DWIN32_LEAN_AND_MEAN", # Reduce Windows header conflicts
]
# Linker flags for Windows
link_flags = [
"-mconsole", # Console application
# Prevent automatic library inclusion to avoid CRT collisions
"-nodefaultlibs", # Don't automatically include default libraries
# Fix CRT collision between Zig/Clang and MSYS2/MinGW runtimes
# This prevents "atexit was replaced" errors from conflicting CRT implementations
# "-rtlib=compiler-rt", # Removed - was causing conflicts
"-unwindlib=libunwind", # Use Clang's libunwind (avoid MinGW unwind)
"-nostdlib++", # Don't auto-link standard C++ library
"-lc++", # Manually link libc++ (Clang's C++ standard library)
"-lkernel32", # Windows kernel functions
"-luser32", # Windows user interface
"-lgdi32", # Graphics device interface
"-ladvapi32", # Advanced Windows API
]
[linking.base]
# Base linking flags (Unix-like systems)
flags = [
"-pthread", # POSIX threads (not used on Windows)
]
[build_modes.quick]
flags = [
"-O1",
"-g0",
"-fno-inline-functions",
"-fno-vectorize",
"-fno-unroll-loops",
"-fno-strict-aliasing",
]
link_flags = []
[strict_mode]
flags = [
"-Werror",
"-Wextra",
"-Wconversion",
"-Wsign-conversion",
"-Werror=unused-variable", # Force error on unused variables
"-Werror=unused-parameter", # Force error on unused parameters
"-Werror=unused-function", # Force error on unused functions
"-Werror=unused-but-set-variable", # Force error on variables that are set but never used
"-Wuninitialized",
"-Wdouble-promotion",
"-Wformat=2",
"-Wcast-align",
"-Wcast-qual",
"-Werror=return-type",
]
[stub_platform]
# ================================================================================================
# STUB PLATFORM DEFINES FOR EXAMPLE COMPILATION
# ================================================================================================
# These defines are used for compiling examples on the STUB platform
# which provides Arduino-compatible functionality without hardware dependencies.
defines = [
"STUB_PLATFORM", # Enable STUB platform
"ARDUINO=10808", # Arduino version compatibility
"FASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
"FASTLED_STUB_IMPL", # Enable STUB platform Arduino function implementations
"SKETCH_HAS_LOTS_OF_MEMORY=1", # Enable memory-intensive features for STUB platform
"FASTLED_HAS_ENGINE_EVENTS=1", # Enable EngineEvents for UI components
]
# Include paths required for stub platform compilation
include_paths = [
"src/platforms/stub", # Arduino.h stub for examples
]
# Additional compiler arguments for stub platform
compiler_args = [
# No additional args currently needed - all flags come from [all] and [build_modes.quick] sections
]
[archive]
# ================================================================================================
# ARCHIVE CREATION FLAGS
# ================================================================================================
# Flags for creating static libraries (.a files) with deterministic builds.
# Using Zig's unified toolchain (ziglang ar) which supports deterministic builds on all platforms.
flags = "rcsD" # r=insert, c=create if needed, s=write symbol table, D=deterministic (zero timestamps)
[linking.unix]
# ================================================================================================
# UNIX-SPECIFIC LINKING FLAGS
# ================================================================================================
# Linking flags specific to Unix-like systems (Linux, macOS).
flags = [
# Core Libraries
"-pthread", # POSIX threads support
"-lm", # Math library
"-ldl", # Dynamic loading library
# Platform-specific libraries (added as needed)
# Note: Additional platform-specific flags can be added here
]
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/build_example_usage.py
+220
View File
@@ -0,0 +1,220 @@
#!/usr/bin/env python3
"""
FastLED Unified Build API - Example Usage
This script demonstrates how to use the new unified build API to replace
the old scattered build logic for both unit tests and examples.
Before (scattered logic):
- PCH building logic in multiple places
- libfastled.a building in callers
- Build flags mixed between unit tests and examples
- Manual directory management
After (unified API):
- Single API for both unit tests and examples
- Automatic PCH and library building
- Clean separation of build configurations
- Organized build directory structure
"""
from pathlib import Path
from ci.build_api import BuildType, create_example_builder, create_unit_test_builder
def demo_unit_test_building():
"""Demonstrate unit test building with the unified API"""
print("=" * 60)
print("UNIT TEST BUILDING DEMO")
print("=" * 60)
# Create unit test builder - automatically uses ci/build_unit.toml
unit_builder = create_unit_test_builder(
build_dir=".build/unit", use_pch=True, parallel=True, clean=False
)
# Get build info
info = unit_builder.get_build_info()
print(f"Build configuration: {info}")
# Build multiple unit test targets efficiently
# The API automatically handles:
# - PCH building (once, reused by all targets)
# - libfastled.a building (once, reused by all targets)
# - Individual target compilation and linking
test_files = [
Path("tests/test_json.cpp"),
Path("tests/test_color.cpp"),
Path("tests/test_math.cpp"),
]
print(f"\nBuilding {len(test_files)} unit test targets...")
results = unit_builder.build_targets(test_files)
# Report results
for result in results:
status = "SUCCESS" if result.success else "FAILED"
print(f" {result.target.name}: {status} ({result.build_time:.2f}s)")
if not result.success:
print(f" Error: {result.message}")
print(f"\nBuild artifacts organized in: {unit_builder.build_dir}")
print(f" - PCH file: {unit_builder.artifacts_dir}")
print(f" - Library: {unit_builder.artifacts_dir}")
print(f" - Targets: {unit_builder.targets_dir}")
print(f" - Cache: {unit_builder.cache_dir}")
def demo_example_building():
"""Demonstrate example building with the unified API"""
print("\n" + "=" * 60)
print("EXAMPLE BUILDING DEMO")
print("=" * 60)
# Create example builder - automatically uses ci/build_example.toml
example_builder = create_example_builder(
build_dir=".build/examples", use_pch=True, parallel=True, clean=False
)
# Get build info
info = example_builder.get_build_info()
print(f"Build configuration: {info}")
# Build multiple example targets efficiently
# The API automatically handles:
# - PCH building (once, reused by all targets)
# - libfastled.a building (once, with example-specific flags)
# - Individual target compilation and linking
example_files = [
Path("examples/Blink/Blink.ino"),
Path("examples/DemoReel100/DemoReel100.ino"),
Path("examples/ColorPalette/ColorPalette.ino"),
]
print(f"\nBuilding {len(example_files)} example targets...")
results = example_builder.build_targets(example_files)
# Report results
for result in results:
status = "SUCCESS" if result.success else "FAILED"
print(f" {result.target.name}: {status} ({result.build_time:.2f}s)")
if not result.success:
print(f" Error: {result.message}")
print(f"\nBuild artifacts organized in: {example_builder.build_dir}")
print(f" - PCH file: {example_builder.artifacts_dir}")
print(f" - Library: {example_builder.artifacts_dir}")
print(f" - Targets: {example_builder.targets_dir}")
print(f" - Cache: {example_builder.cache_dir}")
def demo_build_configuration_differences():
"""Demonstrate the differences between unit test and example configurations"""
print("\n" + "=" * 60)
print("BUILD CONFIGURATION DIFFERENCES")
print("=" * 60)
unit_builder = create_unit_test_builder()
example_builder = create_example_builder()
unit_info = unit_builder.get_build_info()
example_info = example_builder.get_build_info()
print("Unit Test Configuration:")
print(f" - Build flags TOML: {unit_info['build_flags_toml']}")
print(f" - Build directory: {unit_info['build_dir']}")
print(f" - Build type: {unit_info['build_type']}")
print(f" - Includes FASTLED_FORCE_NAMESPACE=1")
print(f" - Includes FASTLED_TESTING=1")
print("\nExample Configuration:")
print(f" - Build flags TOML: {example_info['build_flags_toml']}")
print(f" - Build directory: {example_info['build_dir']}")
print(f" - Build type: {example_info['build_type']}")
print(f" - Uses global namespace (no FASTLED_FORCE_NAMESPACE)")
print(f" - Arduino compatibility defines")
def demo_clean_builds():
"""Demonstrate clean build functionality"""
print("\n" + "=" * 60)
print("CLEAN BUILD DEMO")
print("=" * 60)
# Create builders with clean=True to force rebuilding of all artifacts
unit_builder = create_unit_test_builder(clean=True)
example_builder = create_example_builder(clean=True)
print("Unit test builder:")
print(f" - Will clean and rebuild all artifacts in {unit_builder.build_dir}")
print("Example builder:")
print(f" - Will clean and rebuild all artifacts in {example_builder.build_dir}")
# Demonstrate manual cleaning
print("\nManual cleaning:")
unit_builder.clean_build_artifacts()
print(f" - Cleaned unit test artifacts")
example_builder.clean_build_artifacts()
print(f" - Cleaned example artifacts")
def demo_advanced_usage():
"""Demonstrate advanced usage patterns"""
print("\n" + "=" * 60)
print("ADVANCED USAGE DEMO")
print("=" * 60)
# Custom build directory structure
unit_builder = create_unit_test_builder(
build_dir=".build/my_custom_unit_tests",
use_pch=False, # Disable PCH for debugging
parallel=False, # Disable parallel for single-threaded debugging
)
print("Custom configuration:")
print(f" - Custom build directory: {unit_builder.build_dir}")
print(f" - PCH disabled for easier debugging")
print(f" - Parallel compilation disabled for sequential output")
# Show the organized directory structure
print(f"\nOrganized directory structure:")
print(f" {unit_builder.build_dir}/")
print(f" ├── artifacts/ # PCH files and libfastled.a")
print(f" ├── targets/ # Individual target builds")
print(f" │ ├── test_json/")
print(f" │ │ ├── test_json.o")
print(f" │ │ └── test_json.exe")
print(f" │ └── test_color/")
print(f" │ ├── test_color.o")
print(f" │ └── test_color.exe")
print(f" └── cache/ # Fingerprint caches")
if __name__ == "__main__":
print("FastLED Unified Build API - Example Usage")
print("This demonstrates the new unified build system")
# Run all demos (commented out to avoid actual compilation during demo)
# Uncomment these to run actual builds:
# demo_unit_test_building()
# demo_example_building()
# These can run without actual building:
demo_build_configuration_differences()
demo_clean_builds()
demo_advanced_usage()
print("\n" + "=" * 60)
print("DEMO COMPLETE")
print("=" * 60)
print("The unified build API provides:")
print("✓ Single API for both unit tests and examples")
print("✓ Automatic PCH and library building")
print("✓ Clean separation of build configurations")
print("✓ Organized build directory structure")
print("✓ Efficient multiple target building")
print("✓ Simple, elegant interface")
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/build_unit.toml
+276
View File
@@ -0,0 +1,276 @@
# ================================================================================================
# FastLED Unit Test Build Flags Configuration
# ================================================================================================
#
# This file centralizes ALL compilation flags used by unit test compilation to ensure
# they remain synchronized and compatible across the entire build system.
#
# Unit tests require special flags like FASTLED_FORCE_NAMESPACE=1 to put all symbols
# in the fl:: namespace that the tests expect.
[tools]
# Build tool configuration - Full compiler command with arguments
# CRITICAL: Must use 'uv run python' for subprocess execution (subprocess doesn't inherit uv environment)
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
linker = ["uv", "run", "python", "-m", "ziglang", "c++"] # Separate linker config for consistency
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"] # Ziglang archiver for unified toolchain
c_compiler = ["uv", "run", "python", "-m", "ziglang", "cc"]
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
[all]
# Universal compilation flags (compiler command handled separately)
compiler_flags = [
"-std=gnu++17",
"-fpermissive",
"-Wall",
"-Wextra",
"-Wno-deprecated-register", # Suppress 'register' keyword warnings for C++17 compatibility
"-Wno-backslash-newline-escape", # Suppress backslash newline escape warnings
"-fno-exceptions",
"-fno-rtti",
]
defines = [
"STUB_PLATFORM",
"FASTLED_UNIT_TEST=1",
"FASTLED_FORCE_NAMESPACE=1", # CRITICAL: Required for unit tests to access fl:: namespace
"FASTLED_TESTING=1", # Enable test-specific functionality like MockTimeProvider
]
include_flags = [
"-I.",
"-Isrc",
"-Itests",
]
[windows]
# ================================================================================================
# WINDOWS-SPECIFIC BUILD FLAGS
# ================================================================================================
# Comprehensive Windows build configuration using Zig's bundled Clang with GNU target
# C compiler flags for Windows
cc_flags = [
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
"-fuse-ld=lld", # Use Zig's bundled LLD linker
"-std=c17", # C17 standard
"-Wall",
"-Wextra",
]
# C++ compiler flags for Windows
cpp_flags = [
"--target=x86_64-windows-gnu", # Explicit target for MSYS2/MinGW compatibility
"-fuse-ld=lld-link", # Use lld-link (Windows MSVC-compatible linker)
"-std=c++17", # C++17 standard
"-Wall",
"-Wextra",
"-fno-exceptions",
"-fno-rtti",
# Fix Arduino INPUT macro conflict with Windows headers
"-DNOMINMAX", # Prevent Windows min/max macros
"-DWIN32_LEAN_AND_MEAN", # Reduce Windows header conflicts
]
# Linker flags for Windows
link_flags = [
"-mconsole", # Console application
# Prevent automatic library inclusion to avoid CRT collisions
"-nodefaultlibs", # Don't automatically include default libraries
# Fix CRT collision between Zig/Clang and MSYS2/MinGW runtimes
# This prevents "atexit was replaced" errors from conflicting CRT implementations
# "-rtlib=compiler-rt", # Removed - was causing conflicts
"-unwindlib=libunwind", # Use Clang's libunwind (avoid MinGW unwind)
"-nostdlib++", # Don't auto-link standard C++ library
"-lc++", # Manually link libc++ (Clang's C++ standard library)
"-lkernel32", # Windows kernel functions
"-luser32", # Windows user interface
"-lgdi32", # Graphics device interface
"-ladvapi32", # Advanced Windows API
]
[linking.base]
# Base linking flags (Unix-like systems)
flags = [
"-pthread", # POSIX threads (not used on Windows)
]
[linux]
# Linux-specific build configuration – libunwind disabled (fallback to execinfo)
link_flags = [
]
defines = [
]
[build_modes.quick]
flags = [
"-O1",
"-fno-inline-functions",
"-fno-vectorize",
"-fno-unroll-loops",
"-fno-strict-aliasing",
"-fno-omit-frame-pointer", # Preserve stack frames for debugging
]
link_flags = []
[build_modes.debug_asan]
# AddressSanitizer and UndefinedBehaviorSanitizer for debugging memory issues
flags = [
"-fsanitize=address", # AddressSanitizer for memory error detection
"-fsanitize=undefined", # UndefinedBehaviorSanitizer for undefined behavior
"-O1", # Light optimization for performance
"-fno-omit-frame-pointer", # Preserve stack frames for better debugging
"-fno-optimize-sibling-calls", # Better stack traces
]
link_flags = [
"-fsanitize=address",
"-fsanitize=undefined",
]
[strict_mode]
flags = [
"-Werror",
"-Wextra",
"-Wconversion",
"-Wsign-conversion",
"-Werror=unused-variable", # Force error on unused variables
"-Werror=unused-parameter", # Force error on unused parameters
"-Werror=unused-function", # Force error on unused functions
"-Werror=unused-but-set-variable", # Force error on variables that are set but never used
"-Wuninitialized",
"-Wdouble-promotion",
"-Wformat=2",
"-Wcast-align",
"-Wcast-qual",
"-Werror=return-type",
]
[test]
# Test-specific preprocessor defines
defines = [
# FastLED Test Configuration
"-DFASTLED_UNIT_TEST=1", # Identify as unit test build
"-DFASTLED_FORCE_NAMESPACE=1", # Force namespace usage to avoid conflicts
"-DFASTLED_USE_PROGMEM=0", # Disable PROGMEM usage in tests
"-DSTUB_PLATFORM", # Use stub platform for testing
"-DARDUINO=10808", # Arduino version compatibility
"-DFASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
"-DSKETCH_HAS_LOTS_OF_MEMORY=1", # Assume lots of memory available
"-DFASTLED_STUB_IMPL", # Use stub implementations
"-DFASTLED_USE_JSON_UI=1", # Enable JSON UI features
"-DFASTLED_TESTING", # Enable testing features
"-DFASTLED_NO_AUTO_NAMESPACE", # Disable automatic namespace
"-DFASTLED_NO_PINMAP", # Disable pin mapping
"-DHAS_HARDWARE_PIN_SUPPORT", # Enable hardware pin support
"-DFASTLED_DEBUG_LEVEL=1", # Set debug level
"-DFASTLED_NO_ATEXIT=1", # Disable atexit to fix Windows lld-link errors
"-DDOCTEST_CONFIG_NO_EXCEPTIONS_BUT_WITH_ALL_ASSERTS", # Doctest configuration
"-DENABLE_CRASH_HANDLER", # Enable global crash handler in tests
]
# Test-specific compiler flags
compiler_flags = [
# Language Standard and Compatibility
"-std=c++17", # Base C++17 standard
"-std=gnu++17", # Use GNU C++17 standard with extensions
"-fpermissive", # Allow some non-conforming code patterns
# Warning Configuration
"-Wall", # Enable all warnings
"-Wextra", # Enable extra warnings
"-Werror=return-type", # Error on missing return statements
# Debug and Crash Handling
"-fno-omit-frame-pointer", # Preserve stack frames for debugging
# AddressSanitizer and Memory Debugging
"-fsanitize=address", # AddressSanitizer for memory error detection
"-fsanitize=undefined", # UndefinedBehaviorSanitizer for undefined behavior
# Thread Safety and Runtime Behavior
"-fno-threadsafe-statics", # Disable thread-safe static initialization
"-fno-exceptions", # Disable C++ exceptions
"-fno-rtti", # Disable RTTI
# PCH Support
"-fpch-instantiate-templates", # Required for proper PCH template handling
]
# Test-specific include flags
include_flags = [
"-I.", # Current directory
"-Isrc", # FastLED source directory
"-Itests", # Test directory
]
# Test-specific linking flags for AddressSanitizer and libunwind
link_flags = [
"-fsanitize=address", # Link AddressSanitizer runtime
"-fsanitize=undefined", # Link UndefinedBehaviorSanitizer runtime
"-lunwind", # Link libunwind for stack traces
"-lunwind-x86_64", # Platform-specific libunwind functions
]
[archive]
# ================================================================================================
# ARCHIVE CREATION FLAGS
# ================================================================================================
# Flags for creating static libraries (.a files) with deterministic builds.
# Using Zig's unified toolchain (ziglang ar) which supports deterministic builds on all platforms.
flags = "rcsD" # r=insert, c=create if needed, s=write symbol table, D=deterministic (zero timestamps)
[stub_platform]
# ================================================================================================
# STUB PLATFORM DEFINES FOR UNIT TEST COMPILATION
# ================================================================================================
# These defines are used when unit tests need to compile example code on the STUB platform
# which provides Arduino-compatible functionality without hardware dependencies.
defines = [
"STUB_PLATFORM", # Enable STUB platform
"ARDUINO=10808", # Arduino version compatibility
"FASTLED_USE_STUB_ARDUINO", # Use stub Arduino implementation
"FASTLED_STUB_IMPL", # Enable STUB platform Arduino function implementations
"SKETCH_HAS_LOTS_OF_MEMORY=1", # Enable memory-intensive features for STUB platform
"FASTLED_HAS_ENGINE_EVENTS=1", # Enable EngineEvents for UI components
"FASTLED_UNIT_TEST=1", # Identify as unit test build
"FASTLED_FORCE_NAMESPACE=1", # Force namespace usage for unit tests
"FASTLED_TESTING=1", # Enable testing features
]
# Include paths required for stub platform compilation in unit tests
include_paths = [
"src/platforms/stub", # Arduino.h stub for examples
]
# Additional compiler arguments for stub platform
compiler_args = [
# No additional args currently needed - all flags come from [all] and other sections
]
[linking.unix]
# ================================================================================================
# UNIX-SPECIFIC LINKING FLAGS
# ================================================================================================
# Linking flags specific to Unix-like systems (Linux, macOS).
flags = [
# Core Libraries
"-pthread", # POSIX threads support
"-lm", # Math library
"-ldl", # Dynamic loading library
# Platform-specific libraries (added as needed)
# Note: Additional platform-specific flags can be added here
]
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/check-js-types.py
+61
View File
@@ -0,0 +1,61 @@
#!/usr/bin/env python3
"""
TypeScript checking for FastLED JavaScript/WASM compiler modules.
This script runs TypeScript compiler with JSDoc checking enabled
to catch type errors in JavaScript files that would be caught
at compile time in TypeScript.
"""
import subprocess
import sys
from pathlib import Path
def check_js_types():
"""Run TypeScript checking on JavaScript files."""
wasm_compiler_dir = (
Path(__file__).parent.parent / "src" / "platforms" / "wasm" / "compiler"
)
if not wasm_compiler_dir.exists():
print(f"❌ WASM compiler directory not found: {wasm_compiler_dir}")
return 1
jsconfig_path = wasm_compiler_dir / "jsconfig.json"
if not jsconfig_path.exists():
print(f"❌ jsconfig.json not found: {jsconfig_path}")
return 1
print(f"🔍 Checking JavaScript types in: {wasm_compiler_dir}")
try:
# Run TypeScript compiler with noEmit to only check types
result = subprocess.run(
["npx", "tsc", "--noEmit", "--project", str(jsconfig_path)],
cwd=wasm_compiler_dir,
capture_output=True,
text=True,
)
if result.returncode == 0:
print("✅ JavaScript type checking passed!")
return 0
else:
print("❌ JavaScript type checking failed:")
print(result.stdout)
print(result.stderr)
return 1
except FileNotFoundError:
print(
"❌ TypeScript (tsc) not found. Please install: npm install -g typescript"
)
return 1
except Exception as e:
print(f"❌ Error running type check: {e}")
return 1
if __name__ == "__main__":
sys.exit(check_js_types())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-check-compiled-size.py
+123
View File
@@ -0,0 +1,123 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
import argparse
import os
import re
import subprocess
import sys
from pathlib import Path
from typing import Any, List, Match, Optional
HERE = Path(__file__).resolve().parent
PROJECT_ROOT = HERE.parent
IS_GITHUB = "GITHUB_ACTIONS" in os.environ
def run_command(
cmd_list: List[str],
capture_output: bool = True,
shell: bool = False,
check: Optional[bool] = None,
) -> Optional[str]:
"""Run a command and return its output."""
actual_check = check if check is not None else False
cmd = cmd_list if not shell else subprocess.list2cmdline(cmd_list)
result: subprocess.CompletedProcess[str] = subprocess.run(
cmd, capture_output=capture_output, text=True, shell=shell, check=actual_check
)
if not capture_output:
return None
stdout: str = result.stdout
stdout = stdout.strip()
return stdout
def parse_args():
parser = argparse.ArgumentParser(
description="Check compiled program size for a board"
)
parser.add_argument("board", help="Board name")
parser.add_argument(
"--max-size", type=int, required=False, help="Maximum allowed size"
)
parser.add_argument(
"--no-build",
action="store_true",
help="Skip compilation and check existing build",
)
parser.add_argument(
"--example",
default="Blink",
help="Example to compile (default: Blink)",
)
# Parse known args first
args, unknown = parser.parse_known_args()
# Add remaining arguments as extra_args
args.extra_args = unknown
return args
def main():
os.chdir(str(PROJECT_ROOT))
args = parse_args()
if not args.no_build:
cmd_list = [
"uv",
"run",
"python",
"-m",
"ci.ci-compile",
args.board,
"--examples",
args.example,
] + args.extra_args
try:
run_command(cmd_list, shell=True, capture_output=IS_GITHUB, check=True)
except subprocess.CalledProcessError:
run_command(cmd_list, shell=True, capture_output=False, check=True)
output = run_command(
["uv", "run", "-m", "ci.compiled_size", "--board", args.board],
capture_output=True,
)
size_match = re.search(r": *(\d+)", output) # type: ignore
if not size_match:
print("Error: Unable to extract size from output")
print(f"Output: {output}")
sys.exit(1)
size_str = size_match.group(1) # type: ignore
if not size_str:
print("Error: Size group is empty")
sys.exit(1)
size = int(size_str) # type: ignore
if args.max_size is not None and args.max_size > 0:
max_size = args.max_size
if size > max_size:
print(f"{args.board} size {size} is greater than max size {max_size}")
print("::error::Compiled size exceeds maximum allowed size")
sys.exit(1)
else:
print(f"{args.board} size {size} is within the limit of {max_size}")
else:
if not args.max_size:
print("Warning: No max size specified")
elif args.max_size <= 0:
print("Warning: max size was <= 0 so no check was performed")
print(f"{args.board} size: {size}")
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-compile
+5
View File
@@ -0,0 +1,5 @@
#!/bin/bash
# cd to the directory of the script
cd "$(dirname "$0")"
uv run python ci-compile.py "$@"
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-compile-concurrent-backup.py
+424
View File
@@ -0,0 +1,424 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
"""
BACKUP: Original concurrent build system - REPLACED by pio ci approach
This file is a backup of the original ci-compile.py that used concurrent builds.
It has been replaced by a simpler pio ci approach in the main ci-compile.py.
The original system used ThreadPoolExecutor to run multiple board compilations
in parallel with build directory creation and artifact recycling.
The new system uses pio ci command directly for each board/example combination,
which is simpler and more straightforward.
This backup is kept for reference and potential future needs.
"""
import argparse
import concurrent.futures
import io
import multiprocessing
import os
import subprocess
import sys
import time
import warnings
from pathlib import Path
from typing import List, Set
from ci.boards import Board, create_board # type: ignore
from ci.util.concurrent_run import ConcurrentRunArgs, concurrent_run
from ci.util.locked_print import locked_print
HERE = Path(__file__).parent.resolve()
LIBS = ["src"]
EXTRA_LIBS = [
"https://github.com/me-no-dev/ESPAsyncWebServer.git",
"ArduinoOTA",
"SD",
"FS",
"ESPmDNS",
"WiFi",
"WebSockets",
]
BUILD_FLAGS = ["-Wl,-Map,firmware.map", "-fopt-info-all=optimization_report.txt"]
# Default boards to compile for. You can use boards not defined here but
# if the board isn't part of the officially supported platformio boards then
# you will need to add the board to the ~/.platformio/platforms directory.
# prior to running this script. This happens automatically as of 2024-08-20
# with the github workflow scripts.
DEFAULT_BOARDS_NAMES = [
"apollo3_red",
"apollo3_thing_explorable",
"web", # work in progress
"uno", # Build is faster if this is first, because it's used for global init.
"esp32dev",
"esp01", # ESP8266
"esp32c3",
"attiny85",
"ATtiny1616",
"esp32c6",
"esp32s3",
"esp32p4",
"yun",
"digix",
"teensylc",
"teensy30",
"teensy31",
"teensy41",
"adafruit_feather_nrf52840_sense",
"xiaoblesense_adafruit",
"rpipico",
"rpipico2",
"uno_r4_wifi",
"esp32rmt_51",
"esp32dev_idf44",
"bluepill",
"esp32rmt_51",
"giga_r1",
"sparkfun_xrp_controller",
]
OTHER_BOARDS_NAMES = [
"nano_every",
"esp32-c2-devkitm-1",
]
# Examples to compile.
DEFAULT_EXAMPLES = [
"Animartrix",
"Apa102",
"Apa102HD",
"Apa102HDOverride",
"Audio",
"Blink",
"Blur",
"Chromancer",
"ColorPalette",
"ColorTemperature",
"Corkscrew",
"Cylon",
"DemoReel100",
# "Downscale",
"FestivalStick",
"FirstLight",
"Fire2012",
"Multiple/MultipleStripsInOneArray",
"Multiple/ArrayOfLedArrays",
"Noise",
"NoisePlayground",
"NoisePlusPalette",
# "LuminescentGrand",
"Pacifica",
"Pride2015",
"RGBCalibrate",
"RGBSetDemo",
"RGBW",
"Overclock",
"RGBWEmulated",
"TwinkleFox",
"XYMatrix",
"FireMatrix",
"FireCylinder",
"FxGfx2Video",
"FxSdCard",
"FxCylon",
"FxDemoReel100",
"FxTwinkleFox",
"FxFire2012",
"FxNoisePlusPalette",
"FxPacifica",
"FxEngine",
"WS2816",
]
EXTRA_EXAMPLES: dict[Board, list[str]] = {
# ESP32DEV: ["EspI2SDemo"],
# ESP32_S3_DEVKITC_1: ["EspS3I2SDemo"],
}
def parse_args():
parser = argparse.ArgumentParser(
description="Compile FastLED examples for various boards."
)
# parser.add_argument(
# "--boards", type=str, help="Comma-separated list of boards to compile for"
# )
# needs to be a positional argument instead
parser.add_argument(
"boards",
type=str,
help="Comma-separated list of boards to compile for",
nargs="?",
)
parser.add_argument(
"positional_examples",
type=str,
help="Examples to compile (positional arguments after board name)",
nargs="*",
)
parser.add_argument(
"--examples", type=str, help="Comma-separated list of examples to compile"
)
parser.add_argument(
"--exclude-examples", type=str, help="Examples that should be excluded"
)
parser.add_argument(
"--skip-init", action="store_true", help="Skip the initialization step"
)
parser.add_argument(
"--defines", type=str, help="Comma-separated list of compiler definitions"
)
parser.add_argument("--customsdk", type=str, help="custom_sdkconfig project option")
parser.add_argument(
"--extra-packages",
type=str,
help="Comma-separated list of extra packages to install",
)
parser.add_argument(
"--add-extra-esp32-libs",
action="store_true",
help="Add extra libraries to the libraries list to check against compiler errors.",
)
parser.add_argument(
"--build-dir", type=str, help="Override the default build directory"
)
parser.add_argument(
"--no-project-options",
action="store_true",
help="Don't use custom project options",
)
parser.add_argument(
"--interactive",
action="store_true",
help="Enable interactive mode to choose a board",
)
# Passed by the github action to disable interactive mode.
parser.add_argument(
"--no-interactive", action="store_true", help="Disable interactive mode"
)
parser.add_argument(
"-v", "--verbose", action="store_true", help="Enable verbose output"
)
parser.add_argument(
"--supported-boards",
action="store_true",
help="Print the list of supported boards and exit",
)
parser.add_argument(
"--symbols",
action="store_true",
help="Run symbol analysis on compiled output",
)
parser.add_argument(
"--allsrc",
action="store_true",
help="Enable all-source build (adds FASTLED_ALL_SRC=1 define)",
)
parser.add_argument(
"--no-allsrc",
action="store_true",
help="Disable all-source build (adds FASTLED_ALL_SRC=0 define)",
)
try:
args = parser.parse_intermixed_args()
unknown = []
except SystemExit:
# If parse_intermixed_args fails, fall back to parse_known_args
args, unknown = parser.parse_known_args()
# Handle unknown arguments intelligently - treat non-flag arguments as examples
unknown_examples = []
unknown_flags = []
for arg in unknown:
if arg.startswith("-"):
unknown_flags.append(arg)
else:
unknown_examples.append(arg)
# Add unknown examples to positional_examples
if unknown_examples:
if not hasattr(args, "positional_examples") or args.positional_examples is None:
args.positional_examples = []
args.positional_examples.extend(unknown_examples)
# Only warn about actual unknown flags, not examples
if unknown_flags:
warnings.warn(f"Unknown arguments: {unknown_flags}")
# Check for FASTLED_CI_NO_INTERACTIVE environment variable
# This allows test.py and other scripts to force non-interactive mode
if os.environ.get("FASTLED_CI_NO_INTERACTIVE") == "true":
args.interactive = False
args.no_interactive = True
# if --interactive and --no-interative are both passed, --no-interactive takes precedence.
if args.interactive and args.no_interactive:
warnings.warn(
"Both --interactive and --no-interactive were passed, --no-interactive takes precedence."
)
args.interactive = False
# Validate that --allsrc and --no-allsrc are not both specified
if args.allsrc and args.no_allsrc:
warnings.warn(
"Both --allsrc and --no-allsrc were passed, this is contradictory. Please specify only one."
)
sys.exit(1) # Exit with error
return args
def remove_duplicates(items: List[str]) -> List[str]:
seen: Set[str] = set()
out: List[str] = []
for item in items:
if item not in seen:
seen.add(item)
out.append(item)
return out
def choose_board_interactively(boards: List[str]) -> List[str]:
print("Available boards:")
boards = remove_duplicates(sorted(boards))
for i, board in enumerate(boards):
print(f"[{i}]: {board}")
print("[all]: All boards")
out: List[str] = []
while True:
try:
# choice = int(input("Enter the number of the board(s) you want to compile to: "))
input_str = input(
"Enter the number of the board(s) you want to compile to, or it's name(s): "
)
if "all" in input_str:
return boards
for board in input_str.split(","):
if board == "":
continue
if not board.isdigit():
out.append(board) # Assume it's a board name.
else:
index = int(board) # Find the board from the index.
if 0 <= index < len(boards):
out.append(boards[index])
else:
warnings.warn(f"invalid board index: {index}, skipping")
if not out:
print("Please try again.")
continue
return out
except ValueError:
print("Invalid input. Please enter a number.")
def resolve_example_path(example: str) -> Path:
example_path = HERE.parent / "examples" / example
if not example_path.exists():
raise FileNotFoundError(f"Example '{example}' not found at '{example_path}'")
return example_path
def create_concurrent_run_args(args: argparse.Namespace) -> ConcurrentRunArgs:
skip_init = args.skip_init
if args.interactive:
boards = choose_board_interactively(DEFAULT_BOARDS_NAMES + OTHER_BOARDS_NAMES)
else:
boards = args.boards.split(",") if args.boards else DEFAULT_BOARDS_NAMES
projects: list[Board] = []
for board in boards:
projects.append(create_board(board, no_project_options=args.no_project_options))
extra_examples: dict[Board, list[Path]] = {}
# Handle both positional and named examples
if args.positional_examples:
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
examples = []
for example in args.positional_examples:
# Remove "examples/" prefix if present
if example.startswith("examples/"):
example = example[len("examples/") :]
examples.append(example)
elif args.examples:
examples = args.examples.split(",")
else:
examples = DEFAULT_EXAMPLES
# Only add extra examples when using defaults
for b, _examples in EXTRA_EXAMPLES.items():
resolved_examples = [resolve_example_path(example) for example in _examples]
extra_examples[b] = resolved_examples
examples_paths = [resolve_example_path(example) for example in examples]
# now process example exclusions.
if args.exclude_examples:
exclude_examples = args.exclude_examples.split(",")
examples_paths = [
example
for example in examples_paths
if example.name not in exclude_examples
]
for exclude in exclude_examples:
examples.remove(exclude)
customsdk = args.customsdk
defines: list[str] = []
if args.defines:
defines.extend(args.defines.split(","))
# Add FASTLED_ALL_SRC define when --allsrc or --no-allsrc flag is specified
if args.allsrc:
defines.append("FASTLED_ALL_SRC=1")
elif args.no_allsrc:
defines.append("FASTLED_ALL_SRC=0")
extra_packages: list[str] = []
if args.extra_packages:
extra_packages.extend(args.extra_packages.split(","))
build_dir = args.build_dir
extra_scripts = "pre:lib/ci/util-flags.py"
verbose = args.verbose
out: ConcurrentRunArgs = ConcurrentRunArgs(
projects=projects,
examples=examples_paths,
skip_init=skip_init,
defines=defines,
customsdk=customsdk,
extra_packages=extra_packages,
libs=LIBS,
build_dir=build_dir,
extra_scripts=extra_scripts,
cwd=str(HERE.parent),
board_dir=(HERE / "boards").absolute().as_posix(),
build_flags=BUILD_FLAGS,
verbose=verbose,
extra_examples=extra_examples,
symbols=args.symbols,
)
return out
def main() -> int:
"""Main function."""
args = parse_args()
if args.supported_boards:
print(",".join(DEFAULT_BOARDS_NAMES))
return 0
if args.add_extra_esp32_libs:
LIBS.extend(EXTRA_LIBS)
# Set the working directory to the script's parent directory.
run_args = create_concurrent_run_args(args)
start_time = time.time()
rtn = concurrent_run(args=run_args)
time_taken = time.strftime("%Mm:%Ss", time.gmtime(time.time() - start_time))
locked_print(f"Compilation finished in {time_taken}.")
return rtn
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
sys.exit(1)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-compile-native.py
+21
View File
@@ -0,0 +1,21 @@
#!/usr/bin/env python3
import os
import subprocess
import sys
def main():
# Change to the directory of the script
os.chdir(os.path.dirname(os.path.abspath(__file__)))
# Change to the 'native' directory and run 'pio run'
os.chdir("native")
result = subprocess.run(["pio", "run"], check=True)
# Exit with the same status as the pio run command
sys.exit(result.returncode)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-compile-pio-ci-working.py
+596
View File
@@ -0,0 +1,596 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
"""
Runs the compilation process for examples on boards using pio ci command.
This replaces the previous concurrent build system with a simpler pio ci approach.
"""
import argparse
import os
import subprocess
import sys
import time
import warnings
from pathlib import Path
from typing import List, Set
from ci.boards import Board, create_board # type: ignore
from ci.util.locked_print import locked_print
HERE = Path(__file__).parent.resolve()
# Default boards to compile for
DEFAULT_BOARDS_NAMES = [
"apollo3_red",
"apollo3_thing_explorable",
"web", # work in progress
"uno", # Build is faster if this is first, because it's used for global init.
"esp32dev",
"esp01", # ESP8266
"esp32c3",
"attiny85",
"ATtiny1616",
"esp32c6",
"esp32s3",
"esp32p4",
"yun",
"digix",
"teensylc",
"teensy30",
"teensy31",
"teensy41",
"adafruit_feather_nrf52840_sense",
"xiaoblesense_adafruit",
"rpipico",
"rpipico2",
"uno_r4_wifi",
"esp32rmt_51",
"esp32dev_idf44",
"bluepill",
"esp32rmt_51",
"giga_r1",
"sparkfun_xrp_controller",
]
OTHER_BOARDS_NAMES = [
"nano_every",
"esp32-c2-devkitm-1",
]
# Examples to compile.
DEFAULT_EXAMPLES = [
"Animartrix",
"Apa102",
"Apa102HD",
"Apa102HDOverride",
"Audio",
"Blink",
"Blur",
"Chromancer",
"ColorPalette",
"ColorTemperature",
"Corkscrew",
"Cylon",
"DemoReel100",
# "Downscale",
"FestivalStick",
"FirstLight",
"Fire2012",
"Multiple/MultipleStripsInOneArray",
"Multiple/ArrayOfLedArrays",
"Noise",
"NoisePlayground",
"NoisePlusPalette",
# "LuminescentGrand",
"Pacifica",
"Pride2015",
"RGBCalibrate",
"RGBSetDemo",
"RGBW",
"Overclock",
"RGBWEmulated",
"TwinkleFox",
"XYMatrix",
"FireMatrix",
"FireCylinder",
"FxGfx2Video",
"FxSdCard",
"FxCylon",
"FxDemoReel100",
"FxTwinkleFox",
"FxFire2012",
"FxNoisePlusPalette",
"FxPacifica",
"FxEngine",
"WS2816",
]
EXTRA_EXAMPLES: dict[Board, list[str]] = {
# ESP32DEV: ["EspI2SDemo"],
# ESP32_S3_DEVKITC_1: ["EspS3I2SDemo"],
}
def parse_args():
parser = argparse.ArgumentParser(
description="Compile FastLED examples for various boards using pio ci.util."
)
parser.add_argument(
"boards",
type=str,
help="Comma-separated list of boards to compile for",
nargs="?",
)
parser.add_argument(
"positional_examples",
type=str,
help="Examples to compile (positional arguments after board name)",
nargs="*",
)
parser.add_argument(
"--examples", type=str, help="Comma-separated list of examples to compile"
)
parser.add_argument(
"--exclude-examples", type=str, help="Examples that should be excluded"
)
parser.add_argument(
"--defines", type=str, help="Comma-separated list of compiler definitions"
)
parser.add_argument("--customsdk", type=str, help="custom_sdkconfig project option")
parser.add_argument(
"--extra-packages",
type=str,
help="Comma-separated list of extra packages to install",
)
parser.add_argument(
"--build-dir", type=str, help="Override the default build directory"
)
parser.add_argument(
"--interactive",
action="store_true",
help="Enable interactive mode to choose a board",
)
parser.add_argument(
"--no-interactive", action="store_true", help="Disable interactive mode"
)
parser.add_argument(
"-v", "--verbose", action="store_true", help="Enable verbose output"
)
parser.add_argument(
"--supported-boards",
action="store_true",
help="Print the list of supported boards and exit",
)
parser.add_argument(
"--symbols",
action="store_true",
help="Run symbol analysis on compiled output",
)
parser.add_argument(
"--allsrc",
action="store_true",
help="Enable all-source build (adds FASTLED_ALL_SRC=1 define)",
)
parser.add_argument(
"--no-allsrc",
action="store_true",
help="Disable all-source build (adds FASTLED_ALL_SRC=0 define)",
)
try:
args = parser.parse_intermixed_args()
unknown = []
except SystemExit:
# If parse_intermixed_args fails, fall back to parse_known_args
args, unknown = parser.parse_known_args()
# Handle unknown arguments intelligently - treat non-flag arguments as examples
unknown_examples = []
unknown_flags = []
for arg in unknown:
if arg.startswith("-"):
unknown_flags.append(arg)
else:
unknown_examples.append(arg)
# Add unknown examples to positional_examples
if unknown_examples:
if not hasattr(args, "positional_examples") or args.positional_examples is None:
args.positional_examples = []
args.positional_examples.extend(unknown_examples)
# Only warn about actual unknown flags, not examples
if unknown_flags:
warnings.warn(f"Unknown arguments: {unknown_flags}")
# Check for FASTLED_CI_NO_INTERACTIVE environment variable
if os.environ.get("FASTLED_CI_NO_INTERACTIVE") == "true":
args.interactive = False
args.no_interactive = True
# if --interactive and --no-interative are both passed, --no-interactive takes precedence.
if args.interactive and args.no_interactive:
warnings.warn(
"Both --interactive and --no-interactive were passed, --no-interactive takes precedence."
)
args.interactive = False
# Validate that --allsrc and --no-allsrc are not both specified
if args.allsrc and args.no_allsrc:
warnings.warn(
"Both --allsrc and --no-allsrc were passed, this is contradictory. Please specify only one."
)
sys.exit(1)
return args
def remove_duplicates(items: List[str]) -> List[str]:
seen: Set[str] = set()
out: List[str] = []
for item in items:
if item not in seen:
seen.add(item)
out.append(item)
return out
def choose_board_interactively(boards: List[str]) -> List[str]:
print("Available boards:")
boards = remove_duplicates(sorted(boards))
for i, board in enumerate(boards):
print(f"[{i}]: {board}")
print("[all]: All boards")
out: list[str] = []
while True:
try:
input_str = input(
"Enter the number of the board(s) you want to compile to, or it's name(s): "
)
if "all" in input_str:
return boards
for board in input_str.split(","):
if board == "":
continue
if not board.isdigit():
out.append(board) # Assume it's a board name.
else:
index = int(board) # Find the board from the index.
if 0 <= index < len(boards):
out.append(boards[index])
else:
warnings.warn(f"invalid board index: {index}, skipping")
if not out:
print("Please try again.")
continue
return out
except ValueError:
print("Invalid input. Please enter a number.")
def resolve_example_path(example: str) -> Path:
example_path = HERE.parent / "examples" / example
if not example_path.exists():
raise FileNotFoundError(f"Example '{example}' not found at '{example_path}'")
return example_path
def compile_with_pio_ci(
board: Board,
example_paths: list[Path],
build_dir: str | None,
defines: list[str],
verbose: bool,
) -> tuple[bool, str]:
"""Compile examples for a board using pio ci command."""
# Skip web boards
if board.board_name == "web":
locked_print(f"Skipping web target for board {board.board_name}")
return True, ""
board_name = board.board_name
real_board_name = board.get_real_board_name()
# Set up build directory
if build_dir:
board_build_dir = Path(build_dir) / board_name
else:
board_build_dir = Path(".build") / board_name
board_build_dir.mkdir(parents=True, exist_ok=True)
locked_print(f"*** Compiling examples for board {board_name} using pio ci ***")
errors: list[str] = []
for example_path in example_paths:
locked_print(
f"*** Building example {example_path.name} for board {board_name} ***"
)
# Find the .ino file in the example directory
ino_files = list(example_path.glob("*.ino"))
if not ino_files:
error_msg = f"No .ino file found in {example_path}"
locked_print(f"ERROR: {error_msg}")
errors.append(error_msg)
continue
# Use the first .ino file found
ino_file = ino_files[0]
# Build pio ci command
cmd_list = [
"pio",
"ci",
str(ino_file),
"--board",
real_board_name,
"--lib",
"src", # FastLED source directory
"--keep-build-dir",
"--build-dir",
str(board_build_dir / example_path.name),
]
# Add platform-specific options
if board.platform:
cmd_list.extend(["--project-option", f"platform={board.platform}"])
if board.platform_packages:
cmd_list.extend(
["--project-option", f"platform_packages={board.platform_packages}"]
)
if board.framework:
cmd_list.extend(["--project-option", f"framework={board.framework}"])
if board.board_build_core:
cmd_list.extend(
["--project-option", f"board_build.core={board.board_build_core}"]
)
if board.board_build_filesystem_size:
cmd_list.extend(
[
"--project-option",
f"board_build.filesystem_size={board.board_build_filesystem_size}",
]
)
# Add defines
all_defines = defines.copy()
if board.defines:
all_defines.extend(board.defines)
if all_defines:
build_flags = " ".join(f"-D{define}" for define in all_defines)
cmd_list.extend(["--project-option", f"build_flags={build_flags}"])
# Add custom SDK config if specified
if board.customsdk:
cmd_list.extend(["--project-option", f"custom_sdkconfig={board.customsdk}"])
# Add verbose flag if requested
if verbose:
cmd_list.append("--verbose")
# Execute the command
cmd_str = subprocess.list2cmdline(cmd_list)
locked_print(f"Running command: {cmd_str}")
start_time = time.time()
try:
result = subprocess.run(
cmd_list,
capture_output=True,
text=True,
cwd=str(HERE.parent),
timeout=300, # 5 minute timeout per example
)
elapsed_time = time.time() - start_time
if result.returncode == 0:
locked_print(
f"*** Successfully built {example_path.name} for {board_name} in {elapsed_time:.2f}s ***"
)
if verbose and result.stdout:
locked_print(f"Build output:\n{result.stdout}")
else:
error_msg = f"Failed to build {example_path.name} for {board_name}"
locked_print(f"ERROR: {error_msg}")
locked_print(f"Command: {cmd_str}")
if result.stdout:
locked_print(f"STDOUT:\n{result.stdout}")
if result.stderr:
locked_print(f"STDERR:\n{result.stderr}")
errors.append(f"{error_msg}: {result.stderr}")
except subprocess.TimeoutExpired:
error_msg = f"Timeout building {example_path.name} for {board_name}"
locked_print(f"ERROR: {error_msg}")
errors.append(error_msg)
except Exception as e:
error_msg = f"Exception building {example_path.name} for {board_name}: {e}"
locked_print(f"ERROR: {error_msg}")
errors.append(error_msg)
if errors:
return False, "\n".join(errors)
return True, f"Successfully compiled all examples for {board_name}"
def run_symbol_analysis(boards: list[Board]) -> None:
"""Run symbol analysis on compiled outputs if requested."""
locked_print("\nRunning symbol analysis on compiled outputs...")
for board in boards:
if board.board_name == "web":
continue
try:
locked_print(f"Running symbol analysis for board: {board.board_name}")
cmd = [
"uv",
"run",
"ci/util/symbol_analysis.py",
"--board",
board.board_name,
]
result = subprocess.run(
cmd, capture_output=True, text=True, cwd=str(HERE.parent)
)
if result.returncode != 0:
locked_print(
f"ERROR: Symbol analysis failed for board {board.board_name}: {result.stderr}"
)
else:
locked_print(f"Symbol analysis completed for board: {board.board_name}")
if result.stdout:
print(result.stdout)
except Exception as e:
locked_print(
f"ERROR: Exception during symbol analysis for board {board.board_name}: {e}"
)
def main() -> int:
"""Main function."""
args = parse_args()
if args.supported_boards:
print(",".join(DEFAULT_BOARDS_NAMES))
return 0
# Determine which boards to compile for
if args.interactive:
boards_names = choose_board_interactively(
DEFAULT_BOARDS_NAMES + OTHER_BOARDS_NAMES
)
else:
boards_names = args.boards.split(",") if args.boards else DEFAULT_BOARDS_NAMES
# Get board objects
boards: list[Board] = []
for board_name in boards_names:
try:
board = create_board(board_name, no_project_options=False)
boards.append(board)
except Exception as e:
locked_print(f"ERROR: Failed to get board '{board_name}': {e}")
return 1
# Determine which examples to compile
if args.positional_examples:
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
examples = []
for example in args.positional_examples:
# Remove "examples/" prefix if present
if example.startswith("examples/"):
example = example[len("examples/") :]
examples.append(example)
elif args.examples:
examples = args.examples.split(",")
else:
examples = DEFAULT_EXAMPLES
# Process example exclusions
if args.exclude_examples:
exclude_examples = args.exclude_examples.split(",")
examples = [ex for ex in examples if ex not in exclude_examples]
# Resolve example paths
example_paths: list[Path] = []
for example in examples:
try:
example_path = resolve_example_path(example)
example_paths.append(example_path)
except FileNotFoundError as e:
locked_print(f"ERROR: {e}")
return 1
# Add extra examples for specific boards
extra_examples: dict[Board, list[Path]] = {}
for board in boards:
if board in EXTRA_EXAMPLES:
board_examples = []
for example in EXTRA_EXAMPLES[board]:
try:
board_examples.append(resolve_example_path(example))
except FileNotFoundError as e:
locked_print(f"WARNING: {e}")
if board_examples:
extra_examples[board] = board_examples
# Set up defines
defines: list[str] = []
if args.defines:
defines.extend(args.defines.split(","))
# Add FASTLED_ALL_SRC define when --allsrc or --no-allsrc flag is specified
if args.allsrc:
defines.append("FASTLED_ALL_SRC=1")
elif args.no_allsrc:
defines.append("FASTLED_ALL_SRC=0")
# Start compilation
start_time = time.time()
locked_print(
f"Starting compilation for {len(boards)} boards with {len(example_paths)} examples"
)
compilation_errors: List[str] = []
# Compile for each board
for board in boards:
board_examples = example_paths.copy()
if board in extra_examples:
board_examples.extend(extra_examples[board])
success, message = compile_with_pio_ci(
board=board,
example_paths=board_examples,
build_dir=args.build_dir,
defines=defines,
verbose=args.verbose,
)
if not success:
compilation_errors.append(f"Board {board.board_name}: {message}")
locked_print(f"ERROR: Compilation failed for board {board.board_name}")
# Continue with other boards instead of stopping
# Run symbol analysis if requested
if args.symbols:
run_symbol_analysis(boards)
# Report results
elapsed_time = time.time() - start_time
time_str = time.strftime("%Mm:%Ss", time.gmtime(elapsed_time))
if compilation_errors:
locked_print(
f"\nCompilation finished in {time_str} with {len(compilation_errors)} error(s):"
)
for error in compilation_errors:
locked_print(f" - {error}")
return 1
else:
locked_print(f"\nAll compilations completed successfully in {time_str}")
return 0
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
locked_print("\nInterrupted by user")
sys.exit(1)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-compile.py
+657
View File
@@ -0,0 +1,657 @@
#!/usr/bin/env python3
"""
FastLED Example Compiler
Streamlined compiler that uses the PioCompiler to build FastLED examples for various boards.
This replaces the previous complex compilation system with a simpler approach using the Pio compiler.
"""
import argparse
import os
import sys
import threading
import time
from concurrent.futures import Future, as_completed
from dataclasses import dataclass
from pathlib import Path
from typing import List, Optional, cast
from typeguard import typechecked
from ci.boards import ALL, Board, create_board
from ci.compiler.compiler import CacheType, SketchResult
from ci.compiler.pio import PioCompiler
def green_text(text: str) -> str:
"""Return text in green color."""
return f"\033[32m{text}\033[0m"
def red_text(text: str) -> str:
"""Return text in red color."""
return f"\033[31m{text}\033[0m"
def get_default_boards() -> List[str]:
"""Get all board names from the ALL boards list, with preferred boards first."""
# These are the boards we want to compile first (preferred order)
# Order matters: UNO first because it's used for global init and builds faster
preferred_board_names = [
"uno", # Build is faster if this is first, because it's used for global init.
"esp32dev",
"esp01", # ESP8266
"esp32c3",
"esp32c6",
"esp32s3",
"teensylc",
"teensy31",
"teensy41",
"digix",
"rpipico",
"rpipico2",
]
# Get all available board names from the ALL list
available_board_names = {board.board_name for board in ALL}
# Start with preferred boards that exist, warn about missing ones
default_boards: List[str] = []
for board_name in preferred_board_names:
if board_name in available_board_names:
default_boards.append(board_name)
else:
print(
f"WARNING: Preferred board '{board_name}' not found in available boards"
)
# Add all remaining boards (sorted for consistency)
remaining_boards = sorted(available_board_names - set(default_boards))
default_boards.extend(remaining_boards)
return default_boards
def get_all_examples() -> List[str]:
"""Get all available example names from the examples directory."""
project_root = Path(__file__).parent.parent.resolve()
examples_dir = project_root / "examples"
if not examples_dir.exists():
return []
examples: List[str] = []
for item in examples_dir.iterdir():
if item.is_dir():
# Check if it contains a .ino file with the same name
ino_file = item / f"{item.name}.ino"
if ino_file.exists():
examples.append(item.name)
# Sort for consistent ordering
examples.sort()
return examples
def parse_args(args: Optional[list[str]] = None) -> argparse.Namespace:
"""Parse command line arguments."""
parser = argparse.ArgumentParser(
description="Compile FastLED examples for various boards using PioCompiler"
)
parser.add_argument(
"boards",
type=str,
help="Comma-separated list of boards to compile for",
nargs="?",
)
parser.add_argument(
"positional_examples",
type=str,
help="Examples to compile (positional arguments after board name)",
nargs="*",
)
parser.add_argument(
"--examples", type=str, help="Comma-separated list of examples to compile"
)
parser.add_argument(
"--exclude-examples", type=str, help="Examples that should be excluded"
)
parser.add_argument(
"--defines", type=str, help="Comma-separated list of compiler definitions"
)
parser.add_argument(
"-v", "--verbose", action="store_true", help="Enable verbose output"
)
parser.add_argument(
"--no-cache",
action="store_true",
help="Disable sccache for faster compilation (default is already disabled)",
)
parser.add_argument(
"--enable-cache",
action="store_true",
help="Enable sccache for faster compilation",
)
parser.add_argument(
"--cache",
action="store_true",
help="(Deprecated) Enable sccache for faster compilation - use --enable-cache instead",
)
parser.add_argument(
"--supported-boards",
action="store_true",
help="Print the list of supported boards and exit",
)
parser.add_argument(
"--no-interactive",
action="store_true",
help="Disables interactive mode (deprecated)",
)
parser.add_argument(
"--log-failures",
type=str,
help="Directory to write per-example failure logs (created on first failure)",
)
parser.add_argument(
"--global-cache",
type=str,
help="Override global PlatformIO cache directory path (for testing)",
)
parser.add_argument(
"-o",
"--out",
type=str,
help="Output path for build artifact. Requires exactly one sketch. "
"If path ends with '/', treated as directory. If has suffix, treated as file. "
"Use '-o .' to save in current directory with sketch name.",
)
try:
parsed_args = parser.parse_intermixed_args(args)
unknown = []
except SystemExit:
# If parse_intermixed_args fails, fall back to parse_known_args
parsed_args, unknown = parser.parse_known_args(args)
# Handle unknown arguments intelligently - treat non-flag arguments as examples
unknown_examples: List[str] = []
for arg in unknown:
if not arg.startswith("-"):
unknown_examples.append(arg)
# Add unknown examples to positional_examples
if unknown_examples:
if (
not hasattr(parsed_args, "positional_examples")
or parsed_args.positional_examples is None
):
parsed_args.positional_examples = []
# Type assertion to help the type checker - argparse returns Any but we know it's List[str]
positional_examples: List[str] = cast(
List[str], getattr(parsed_args, "positional_examples", [])
)
positional_examples.extend(unknown_examples)
parsed_args.positional_examples = positional_examples
return parsed_args
def resolve_example_path(example: str) -> str:
"""Resolve example name to path, ensuring it exists."""
project_root = Path(__file__).parent.parent.resolve()
examples_dir = project_root / "examples"
# Handle both "Blink" and "examples/Blink" formats
if example.startswith("examples/"):
example = example[len("examples/") :]
example_path = examples_dir / example
if not example_path.exists():
raise FileNotFoundError(f"Example not found: {example_path}")
return example
@typechecked
@dataclass
class BoardCompilationResult:
"""Aggregated result for compiling a set of examples on a single board."""
ok: bool
sketch_results: List[SketchResult]
def compile_board_examples(
board: Board,
examples: List[str],
defines: List[str],
verbose: bool,
enable_cache: bool,
global_cache_dir: Optional[Path] = None,
) -> BoardCompilationResult:
"""Compile examples for a single board using PioCompiler."""
# Resolve global cache directory immediately for display
resolved_cache_dir = None
if global_cache_dir is not None:
# User specified a path - use it exactly as provided
resolved_cache_dir = global_cache_dir.resolve()
else:
# Default path ends with 'global_cache'
resolved_cache_dir = Path.home() / ".fastled" / "global_cache"
print(f"\n{'=' * 60}")
print(f"COMPILING BOARD: {board.board_name}")
print(f"EXAMPLES: {', '.join(examples)}")
print(f"GLOBAL CACHE: {resolved_cache_dir}")
# Show other cache directories
from ci.compiler.pio import FastLEDPaths
paths = FastLEDPaths(board.board_name)
print(f"BUILD CACHE: {paths.build_cache_dir}")
print(f"CORE DIR: {paths.core_dir}")
print(f"PACKAGES DIR: {paths.packages_dir}")
print(f"{'=' * 60}")
try:
# Determine cache type based on flag and board frameworks
frameworks = [f.strip().lower() for f in (board.framework or "").split(",")]
mixed_frameworks = "arduino" in frameworks and "espidf" in frameworks
cache_type = (
CacheType.SCCACHE
if enable_cache and not mixed_frameworks
else CacheType.NO_CACHE
)
# Create PioCompiler instance
compiler = PioCompiler(
board=board,
verbose=verbose,
global_cache_dir=resolved_cache_dir,
additional_defines=defines,
cache_type=cache_type,
)
# Build all examples
futures: List[Future[SketchResult]] = compiler.build(examples)
# Wait for completion and collect results
results: List[SketchResult] = []
for future in futures:
try:
result = future.result()
results.append(result)
# SUCCESS/FAILED messages are printed by worker threads
except KeyboardInterrupt:
print("Keyboard interrupt detected, cancelling builds")
compiler.cancel_all()
import _thread
for future in futures:
future.cancel()
_thread.interrupt_main()
raise
except Exception as e:
# Represent unexpected exception as a failed SketchResult for consistency
from pathlib import Path as _Path
results.append(
SketchResult(
success=False,
output=f"Build exception: {str(e)}",
build_dir=_Path("."),
example="<exception>",
)
)
print(f"EXCEPTION during build: {e}")
# Cleanup
compiler.cancel_all()
# Show compiler statistics after all builds complete
try:
stats = compiler.get_cache_stats()
if stats:
print("\n" + "=" * 60)
print("COMPILER STATISTICS")
print("=" * 60)
print(stats)
print("=" * 60)
except Exception as e:
print(f"Warning: Could not retrieve compiler statistics: {e}")
any_failures = False
for r in results:
if not r.success:
any_failures = True
break
return BoardCompilationResult(ok=not any_failures, sketch_results=results)
except KeyboardInterrupt:
print("Keyboard interrupt detected, cancelling builds")
import _thread
_thread.interrupt_main()
raise
except Exception as e:
# Compiler could not be set up; return a single failed result to carry message
from pathlib import Path as _Path
return BoardCompilationResult(
ok=False,
sketch_results=[
SketchResult(
success=False,
output=f"Compiler setup failed: {str(e)}",
build_dir=_Path("."),
example="<setup>",
)
],
)
def get_board_artifact_extension(board: Board) -> str:
"""Get the primary artifact extension for a board."""
# ESP32/ESP8266 boards always produce .bin files
if board.board_name.startswith("esp"):
return ".bin"
# Most Arduino-based boards produce .hex files
if board.framework and "arduino" in board.framework.lower():
return ".hex"
# Default to .hex for most microcontroller boards
return ".hex"
def validate_output_path(
output_path: str, sketch_name: str, board: Board
) -> tuple[bool, str, str]:
"""Validate output path and return (is_valid, resolved_path, error_message).
Args:
output_path: The user-specified output path
sketch_name: Name of the sketch being built
board: Board configuration
Returns:
Tuple of (is_valid, resolved_output_path, error_message)
"""
import os
expected_ext = get_board_artifact_extension(board)
# Handle special case: -o .
if output_path == ".":
resolved_path = f"{sketch_name}{expected_ext}"
return True, resolved_path, ""
# If path ends with /, it's a directory
if output_path.endswith("/") or output_path.endswith("\\"):
resolved_path = os.path.join(output_path, f"{sketch_name}{expected_ext}")
return True, resolved_path, ""
# If path has an extension, it's a file - validate the extension
if "." in os.path.basename(output_path):
_, ext = os.path.splitext(output_path)
if ext != expected_ext:
return (
False,
"",
f"Output file extension '{ext}' doesn't match expected '{expected_ext}' for board '{board.board_name}'",
)
return True, output_path, ""
# Path doesn't end with / and has no extension - treat as directory
resolved_path = os.path.join(output_path, f"{sketch_name}{expected_ext}")
return True, resolved_path, ""
def copy_build_artifact(
build_dir: Path, board: Board, sketch_name: str, output_path: str
) -> bool:
"""Copy the build artifact to the specified output path.
Args:
build_dir: Build directory path
board: Board configuration
sketch_name: Name of the sketch
output_path: Target output path
Returns:
True if successful, False otherwise
"""
import os
import shutil
expected_ext = get_board_artifact_extension(board)
# Find the source artifact
# PlatformIO builds are in .build/pio/{board}/.pio/build/{board}/firmware.{ext}
artifact_dir = build_dir / ".pio" / "build" / board.board_name
source_artifact = artifact_dir / f"firmware{expected_ext}"
if not source_artifact.exists():
print(f"ERROR: Build artifact not found: {source_artifact}")
return False
# Ensure output directory exists
output_path_obj = Path(output_path)
output_path_obj.parent.mkdir(parents=True, exist_ok=True)
try:
print(f"Copying {source_artifact} to {output_path}")
shutil.copy2(source_artifact, output_path)
print(f"✅ Build artifact saved to: {output_path}")
return True
except Exception as e:
print(f"ERROR: Failed to copy build artifact: {e}")
return False
def main() -> int:
"""Main function."""
args = parse_args()
if args.verbose:
os.environ["VERBOSE"] = "1"
if args.supported_boards:
print(",".join(get_default_boards()))
return 0
# Determine which boards to compile for
if args.boards:
boards_names = args.boards.split(",")
else:
boards_names = get_default_boards()
# Get board objects
boards: List[Board] = []
for board_name in boards_names:
try:
board = create_board(board_name, no_project_options=False)
boards.append(board)
except Exception as e:
print(f"ERROR: Failed to get board '{board_name}': {e}")
return 1
# Determine which examples to compile
if args.positional_examples:
# Convert positional examples, handling both "examples/Blink" and "Blink" formats
examples: List[str] = []
for example in args.positional_examples:
# Remove "examples/" prefix if present
if example.startswith("examples/"):
example = example[len("examples/") :]
examples.append(example)
elif args.examples:
examples = args.examples.split(",")
else:
# Compile all available examples since builds are fast now!
examples = get_all_examples()
# Process example exclusions
if args.exclude_examples:
exclude_examples = args.exclude_examples.split(",")
examples = [ex for ex in examples if ex not in exclude_examples]
# Validate examples exist
for example in examples:
try:
resolve_example_path(example)
except FileNotFoundError as e:
print(f"ERROR: {e}")
return 1
# Validate -o/--out option requirements
if args.out:
if len(examples) != 1:
print(
f"ERROR: -o/--out option requires exactly one sketch, got {len(examples)}: {examples}"
)
return 1
if len(boards) != 1:
print(
f"ERROR: -o/--out option requires exactly one board, got {len(boards)}: {[b.board_name for b in boards]}"
)
return 1
# Validate the output path
sketch_name = examples[0]
board = boards[0]
is_valid, resolved_output_path, error_msg = validate_output_path(
args.out, sketch_name, board
)
if not is_valid:
print(f"ERROR: {error_msg}")
return 1
# Set up defines
defines: List[str] = []
if args.defines:
defines.extend(args.defines.split(","))
# Start compilation
start_time = time.time()
print(
f"Starting compilation for {len(boards)} boards with {len(examples)} examples"
)
compilation_errors: List[str] = []
failed_example_names: List[str] = []
failure_logs_dir: Optional[Path] = (
Path(args.log_failures) if getattr(args, "log_failures", None) else None
)
# Compile for each board
for board in boards:
# Parse global cache directory if provided
global_cache_dir = None
if args.global_cache:
global_cache_dir = Path(args.global_cache)
result = compile_board_examples(
board=board,
examples=examples,
defines=defines,
verbose=args.verbose,
enable_cache=(args.enable_cache or args.cache) and not args.no_cache,
global_cache_dir=global_cache_dir,
)
if not result.ok:
# Record board-level error
compilation_errors.append(f"Board {board.board_name} failed")
print(red_text(f"ERROR: Compilation failed for board {board.board_name}"))
# Print each failing sketch's stdout and collect names for summary
for sketch in result.sketch_results:
if not sketch.success:
if sketch.example and sketch.example not in failed_example_names:
failed_example_names.append(sketch.example)
# Write per-example failure logs when requested
if failure_logs_dir is not None:
try:
failure_logs_dir.mkdir(parents=True, exist_ok=True)
safe_name = f"{sketch.example}.log"
log_path = failure_logs_dir / safe_name
with log_path.open(
"a", encoding="utf-8", errors="ignore"
) as f:
f.write(sketch.output)
f.write("\n")
except KeyboardInterrupt:
print("Keyboard interrupt detected, cancelling builds")
import _thread
_thread.interrupt_main()
raise
except Exception as e:
print(
f"Warning: Could not write failure log for {sketch.example}: {e}"
)
print(f"\n{'-' * 60}")
print(f"Sketch: {sketch.example}")
print(f"{'-' * 60}")
# Print the collected output for this sketch
print(sketch.output)
# Continue with other boards instead of stopping
else:
# Compilation succeeded - handle -o/--out option if specified
if args.out:
sketch_name = examples[0] # We already validated there's exactly one
is_valid, resolved_output_path, _ = validate_output_path(
args.out, sketch_name, board
)
if is_valid:
# Find the build directory for this board
project_root = Path(__file__).parent.parent.resolve()
build_dir = project_root / ".build" / "pio" / board.board_name
if not copy_build_artifact(
build_dir, board, sketch_name, resolved_output_path
):
compilation_errors.append(
f"Failed to copy artifact for {board.board_name}"
)
print(
red_text(
f"ERROR: Failed to copy build artifact for {board.board_name}"
)
)
return 1
# Report results
elapsed_time = time.time() - start_time
time_str = time.strftime("%Mm:%Ss", time.gmtime(elapsed_time))
if compilation_errors:
print(
f"\nCompilation finished in {time_str} with {len(compilation_errors)} error(s):"
)
for error in compilation_errors:
print(f" - {error}")
if failed_example_names:
print("")
print(red_text(f"ERROR! There were {len(failed_example_names)} failures:"))
# Sort for stable output
for name in sorted(failed_example_names):
# print(f" {name}")
# same but in red
print(red_text(f" {name}"))
print("")
return 1
else:
print(f"\nAll compilations completed successfully in {time_str}")
return 0
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
print("\nInterrupted by user")
sys.exit(1)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-compile.save.py
+1032
View File
File diff suppressed because it is too large. Load diff
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-cppcheck.py
+134
View File
@@ -0,0 +1,134 @@
import argparse
import os
import subprocess
import sys
from pathlib import Path
MINIMUM_REPORT_SEVERTIY = "medium"
MINIMUM_FAIL_SEVERTIY = "high"
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Run cppcheck on the project")
parser.add_argument("board", nargs="?", help="Board to check, optional")
return parser.parse_args()
def find_platformio_project_dir(build_root: Path, board_name: str) -> Path | None:
"""Find a directory containing platformio.ini file for the specified board.
Handles both old and new build directory structures:
Old: .build/uno/Blink/platformio.ini
New: .build/fled/examples/uno/Blink/platformio.ini
Returns the directory containing platformio.ini, or None if not found.
"""
if not build_root.exists():
return None
# Search patterns for different build structures
search_patterns = [
# New structure: .build/fled/examples/{board}/**/*.ini
build_root / "fled" / "examples" / board_name,
# Old structure: .build/{board}/**/*.ini
build_root / board_name,
# Fallback: search all subdirectories for board name
]
# Try each search pattern
for base_dir in search_patterns:
if not base_dir.exists():
continue
# Look for platformio.ini in this directory and its subdirectories
project_dir = find_platformio_ini_in_tree(base_dir)
if project_dir:
return project_dir
# Last resort: search the entire build directory for board-related platformio.ini
for platformio_ini in build_root.rglob("platformio.ini"):
# Check if this platformio.ini is for our board by looking at the path
if board_name in str(platformio_ini):
print(f"Found platformio.ini in {platformio_ini.parent}")
return platformio_ini.parent
return None
def find_platformio_ini_in_tree(base_dir: Path) -> Path | None:
"""Recursively search for platformio.ini in a directory tree."""
if not base_dir.exists():
return None
# Check direct subdirectories first (most common case)
for subdir in base_dir.iterdir():
if subdir.is_dir():
platformio_ini = subdir / "platformio.ini"
if platformio_ini.exists():
print(f"Found platformio.ini in {subdir}")
return subdir
# Check if there's a platformio.ini directly in the base directory
if (base_dir / "platformio.ini").exists():
print(f"Found platformio.ini directly in {base_dir}")
return base_dir
return None
def main() -> int:
args = parse_args()
here = Path(__file__).parent
project_root = here.parent
build = project_root / ".build"
if not build.exists():
print(f"Build directory {build} not found")
return 1
if args.board:
# Search for the specified board
project_dir = find_platformio_project_dir(build, args.board)
if not project_dir:
print(f"No platformio.ini found for board '{args.board}'")
print("This usually means the board hasn't been compiled yet.")
print(f"Try running: uv run ci/ci-compile.py {args.board} --examples Blink")
return 1
else:
# Auto-detect available board by searching for any platformio.ini
project_dir = None
# Try to find any platformio.ini file in the build directory
for platformio_ini in build.rglob("platformio.ini"):
project_dir = platformio_ini.parent
print(f"Auto-detected project directory: {project_dir}")
break
if not project_dir:
print(f"No platformio.ini files found in {build}")
print("This usually means no boards have been compiled yet.")
print("Try running: uv run ci/ci-compile.py uno --examples Blink")
return 1
print(f"Running pio check in {project_dir}")
os.chdir(str(project_dir))
# Run pio check command
cp = subprocess.run(
[
"pio",
"check",
"--skip-packages",
"--src-filters=+<src/>",
f"--severity={MINIMUM_REPORT_SEVERTIY}",
f"--fail-on-defect={MINIMUM_FAIL_SEVERTIY}",
"--flags",
"--inline-suppr",
],
)
return cp.returncode
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-flags.py
+6
View File
@@ -0,0 +1,6 @@
# flake8: noqa
# ruff: skip
Import("env") # type: ignore
env.Append(CXXFLAGS=["-Wno-register"]) # type: ignore
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci-iwyu.py
+271
View File
@@ -0,0 +1,271 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
import argparse
import os
import subprocess
import sys
from pathlib import Path
from typing import List
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Run include-what-you-use on the project"
)
parser.add_argument("board", nargs="?", help="Board to check, optional")
parser.add_argument(
"--fix",
action="store_true",
help="Automatically apply suggested fixes using fix_includes",
)
parser.add_argument(
"--mapping-file",
action="append",
help="Additional mapping file to use (can be specified multiple times)",
)
parser.add_argument(
"--max-line-length",
type=int,
default=100,
help="Maximum line length for suggestions (default: 100)",
)
parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
return parser.parse_args()
def find_platformio_project_dir(board_dir: Path) -> Path | None:
"""Find a directory containing platformio.ini file in the board's build directory.
With the new pio ci build system, the structure is:
.build/uno/Blink/platformio.ini
.build/uno/SomeExample/platformio.ini
We need to find one of these example directories that has a platformio.ini file.
"""
if not board_dir.exists():
return None
# Look for subdirectories containing platformio.ini
for subdir in board_dir.iterdir():
if subdir.is_dir():
platformio_ini = subdir / "platformio.ini"
if platformio_ini.exists():
print(f"Found platformio.ini in {subdir}")
return subdir
# Fallback: check if there's a platformio.ini directly in the board directory (old system)
if (board_dir / "platformio.ini").exists():
print(f"Found platformio.ini directly in {board_dir}")
return board_dir
return None
def check_iwyu_available() -> bool:
"""Check if include-what-you-use is available in the system"""
try:
result = subprocess.run(
["include-what-you-use", "--version"],
capture_output=True,
text=True,
timeout=10,
)
return result.returncode == 0
except (
subprocess.CalledProcessError,
FileNotFoundError,
subprocess.TimeoutExpired,
):
return False
def run_iwyu_on_cpp_tests(args: argparse.Namespace) -> int:
"""Run IWYU on the C++ test suite using CMake integration"""
here = Path(__file__).parent
project_root = here.parent
print("Running include-what-you-use on C++ test suite...")
# Use the existing test infrastructure with --check flag
cmd = [
"uv",
"run",
"test.py",
"--cpp",
"--check", # This enables IWYU via CMake
"--clang",
"--no-interactive",
]
if args.verbose:
cmd.append("--verbose")
try:
result = subprocess.run(cmd, cwd=project_root)
return result.returncode
except subprocess.CalledProcessError as e:
print(f"IWYU analysis failed with return code {e.returncode}")
return e.returncode
def run_iwyu_on_platformio_project(project_dir: Path, args: argparse.Namespace) -> int:
"""Run IWYU on a PlatformIO project"""
print(f"Running include-what-you-use in {project_dir}")
os.chdir(str(project_dir))
# Build mapping file arguments
mapping_args: List[str] = []
# Add FastLED mapping files if they exist
project_root = project_dir
while project_root.parent != project_root: # Find project root
if (project_root / "ci" / "iwyu").exists():
break
project_root = project_root.parent
fastled_mapping = project_root / "ci" / "iwyu" / "fastled.imp"
stdlib_mapping = project_root / "ci" / "iwyu" / "stdlib.imp"
if fastled_mapping.exists():
mapping_args.extend(["--mapping_file", str(fastled_mapping)])
if stdlib_mapping.exists():
mapping_args.extend(["--mapping_file", str(stdlib_mapping)])
# Add user-specified mapping files
if args.mapping_file:
for mapping in args.mapping_file:
mapping_args.extend(["--mapping_file", mapping])
# Build IWYU command
iwyu_cmd = [
"include-what-you-use",
f"--max_line_length={args.max_line_length}",
"--quoted_includes_first",
"--no_comments",
]
if args.verbose:
iwyu_cmd.append("--verbose=3")
else:
iwyu_cmd.append("--verbose=1")
iwyu_cmd.extend(mapping_args)
# Run through PlatformIO's check system with IWYU
pio_cmd = [
"pio",
"check",
"--skip-packages",
"--src-filters=+<src/>",
"--tool=include-what-you-use",
"--flags",
] + iwyu_cmd[1:] # Skip the include-what-you-use binary name
try:
result = subprocess.run(pio_cmd)
return result.returncode
except subprocess.CalledProcessError as e:
print(f"PlatformIO IWYU check failed with return code {e.returncode}")
return e.returncode
def apply_iwyu_fixes(source_dir: Path) -> int:
"""Apply IWYU fixes using fix_includes tool"""
try:
# Check if fix_includes is available
subprocess.run(["fix_includes", "--help"], capture_output=True, check=True)
except (subprocess.CalledProcessError, FileNotFoundError):
print(
"Warning: fix_includes tool not found. Install IWYU tools to use --fix option."
)
return 1
# Find all .h and .cpp files
cpp_files: List[Path] = []
for pattern in ["**/*.cpp", "**/*.h", "**/*.hpp"]:
cpp_files.extend(source_dir.glob(pattern))
if not cpp_files:
print("No C++ files found to fix")
return 0
print(f"Applying IWYU fixes to {len(cpp_files)} files...")
# Run fix_includes on the source directory
cmd = ["fix_includes", "--update_comments", str(source_dir)]
try:
result = subprocess.run(cmd)
if result.returncode == 0:
print("IWYU fixes applied successfully")
else:
print(f"fix_includes failed with return code {result.returncode}")
return result.returncode
except subprocess.CalledProcessError as e:
print(f"Failed to apply IWYU fixes: {e}")
return e.returncode
def main() -> int:
args = parse_args()
here = Path(__file__).parent
project_root = here.parent
# Check if IWYU is available
if not check_iwyu_available():
print("Error: include-what-you-use not found in PATH")
print("Install it with:")
print(" Ubuntu/Debian: sudo apt install iwyu")
print(" macOS: brew install include-what-you-use")
print(" Or build from source: https://include-what-you-use.org/")
return 1
print("Found include-what-you-use")
# If no board specified, run on C++ test suite
if not args.board:
print("No board specified, running IWYU on C++ test suite")
return run_iwyu_on_cpp_tests(args)
# Run on specific board
build = project_root / ".build"
if not build.exists():
print(f"Build directory {build} not found")
print("Run a compilation first: ./compile [board] --examples [example]")
return 1
board_dir = build / args.board
if not board_dir.exists():
print(f"Board {args.board} not found in {build}")
print("Available boards:")
for d in build.iterdir():
if d.is_dir():
print(f" {d.name}")
return 1
project_dir = find_platformio_project_dir(board_dir)
if not project_dir:
print(f"No platformio.ini found in {board_dir} or its subdirectories")
print("This usually means the board hasn't been compiled yet.")
print(f"Try running: ./compile {args.board} --examples Blink")
return 1
# Run IWYU on the PlatformIO project
result = run_iwyu_on_platformio_project(project_dir, args)
# Apply fixes if requested and analysis succeeded
if args.fix and result == 0:
src_dir = project_dir / "src"
if src_dir.exists():
fix_result = apply_iwyu_fixes(src_dir)
if fix_result != 0:
result = fix_result
return result
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/ci/fingerprint_cache.py
+238
View File
@@ -0,0 +1,238 @@
"""
Fingerprint Cache Feature
A two-layer file change detection system that efficiently determines if source files
have been modified by combining fast modification time checks with slower but accurate
MD5 hash verification.
"""
import hashlib
import json
import os
from dataclasses import dataclass
from pathlib import Path
from typing import Dict, Optional
@dataclass
class CacheEntry:
"""Cache entry storing file modification time and content hash."""
modification_time: float
md5_hash: str
@dataclass
class FingerprintCacheConfig:
"""Configuration for fingerprint cache behavior."""
cache_file: Path
hash_algorithm: str
ignore_patterns: list[str]
max_cache_size: int
class FingerprintCache:
"""
Two-layer file change detection using modification time and content hashing.
Provides efficient change detection by:
1. Fast modification time comparison (microsecond performance)
2. Accurate content verification via MD5 hashing when needed
"""
def __init__(self, cache_file: Path, modtime_only: bool = False):
"""
Initialize fingerprint cache.
Args:
cache_file: Path to JSON cache file
modtime_only: When True, disable hashing and rely solely on modtime.
"""
self.cache_file = cache_file
self._modtime_only: bool = modtime_only
self.cache = self._load_cache()
def _load_cache(self) -> Dict[str, CacheEntry]:
"""
Load cache from JSON file, return empty dict if file doesn't exist.
Returns:
Dictionary mapping file paths to cache entries
"""
if not self.cache_file.exists():
return {}
try:
with open(self.cache_file, "r") as f:
data = json.load(f)
# Convert JSON dict to CacheEntry objects
cache: Dict[str, CacheEntry] = {}
for file_path, entry_data in data.items():
cache[file_path] = CacheEntry(
modification_time=entry_data["modification_time"],
md5_hash=entry_data["md5_hash"],
)
return cache
except (json.JSONDecodeError, KeyError, TypeError):
# Cache corrupted - start fresh
return {}
def _save_cache(self) -> None:
"""Save current cache state to JSON file."""
# Ensure cache directory exists
self.cache_file.parent.mkdir(parents=True, exist_ok=True)
# Convert CacheEntry objects to JSON-serializable dict
data = {}
for file_path, entry in self.cache.items():
data[file_path] = {
"modification_time": entry.modification_time,
"md5_hash": entry.md5_hash,
}
with open(self.cache_file, "w") as f:
json.dump(data, f, indent=2)
def _compute_md5(self, file_path: Path) -> str:
"""
Compute MD5 hash of file content.
Args:
file_path: Path to file to hash
Returns:
MD5 hash as hexadecimal string
Raises:
IOError: If file cannot be read
"""
try:
hasher = hashlib.md5()
with open(file_path, "rb") as f:
# Read in chunks to handle large files efficiently
for chunk in iter(lambda: f.read(8192), b""):
hasher.update(chunk)
return hasher.hexdigest()
except IOError as e:
raise IOError(f"Cannot read file {file_path}: {e}")
def has_changed(self, src_path: Path, previous_modtime: float) -> bool:
"""
Determine if a source file has changed since the last known modification time.
Two-layer verification process:
1. Fast modification time check - if times match, return False immediately
2. Content verification - compute/use cached MD5 hash for accurate comparison
Args:
src_path: Path to the source file to check
previous_modtime: Previously known modification time (Unix timestamp)
Returns:
True if the file has changed, False if unchanged
Raises:
FileNotFoundError: If source file doesn't exist
"""
if not src_path.exists():
raise FileNotFoundError(f"Source file not found: {src_path}")
current_modtime = os.path.getmtime(src_path)
# Optional mode: strictly use modification time only (no hashing)
# This is required for toolchains that invalidate PCH on any newer
# dependency regardless of content changes (e.g., Clang).
if self._modtime_only:
# Treat as changed only when file is newer than the reference time
# (keeps behavior stable when reference is an external artifact's mtime)
return current_modtime > previous_modtime
# Layer 1: Quick modification time check
if current_modtime == previous_modtime:
return False # No change detected
file_key = str(src_path.resolve()) # Use absolute path as key
# Layer 2: Content verification via hash comparison
if file_key in self.cache:
cached_entry = self.cache[file_key]
previous_hash = (
cached_entry.md5_hash
) # Store previous hash before potential update
if current_modtime == cached_entry.modification_time:
# File is cached with current modtime - use cached hash
current_hash = cached_entry.md5_hash
else:
# Cache is stale - compute new hash
current_hash = self._compute_md5(src_path)
self._update_cache_entry(src_path, current_modtime, current_hash)
# Compare current hash with previous cached hash
# If they match, content hasn't actually changed despite modtime difference
return current_hash != previous_hash
else:
# File not in cache - compute hash and cache it
current_hash = self._compute_md5(src_path)
self._update_cache_entry(src_path, current_modtime, current_hash)
return True # Assume changed since we have no previous state to compare
def _update_cache_entry(
self, file_path: Path, modification_time: float, md5_hash: str
) -> None:
"""
Update cache with new entry and save to disk.
Args:
file_path: Path to file being cached
modification_time: File modification time
md5_hash: File content hash
"""
file_key = str(file_path.resolve())
self.cache[file_key] = CacheEntry(
modification_time=modification_time, md5_hash=md5_hash
)
self._save_cache()
def get_cache_stats(self) -> dict[str, int]:
"""
Get cache statistics.
Returns:
Dictionary with cache size and other metrics
"""
return {
"total_entries": len(self.cache),
"cache_file_exists": self.cache_file.exists(),
"cache_file_size_bytes": self.cache_file.stat().st_size
if self.cache_file.exists()
else 0,
}
def clear_cache(self) -> None:
"""Clear all cache entries and remove cache file."""
self.cache.clear()
if self.cache_file.exists():
self.cache_file.unlink()
def has_changed(src_path: Path, previous_modtime: float, cache_file: Path) -> bool:
"""
Convenience function implementing the main API.
Args:
src_path: Path to the source file to check
previous_modtime: Previously known modification time (Unix timestamp)
cache_file: Path to the fingerprint cache file
Returns:
True if the file has changed, False if unchanged
Raises:
FileNotFoundError: If source file doesn't exist
"""
cache = FingerprintCache(cache_file)
return cache.has_changed(src_path, previous_modtime)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compile-commands.py
+290
View File
@@ -0,0 +1,290 @@
#!/usr/bin/env python
"""
Compile Commands Generator
Generates a compile_commands.json for clangd/clang-tidy based on explicit
tools/flags configuration. See FEATURE.md for usage and rationale.
Key behaviors:
- Reads configuration from ci/build_commands.toml
- Enumerates library source files under src/ (respecting optional filters.exclude_dirs)
- Emits one entry per translation unit with accurate arguments/command fields
Notes:
- This script intentionally avoids project-unit-test driven flags to keep
IntelliSense consistent with the core FastLED library build.
"""
from __future__ import annotations
import _thread
import argparse
import json
import os
import subprocess
import sys
# Require Python 3.11+ environment for TOML parsing
import tomllib
from dataclasses import dataclass
from pathlib import Path
from typing import Any, Optional, cast
# Require typeguard; no try-import fallbacks
from typeguard import typechecked
# Use the canonical BuildFlags/Compiler tooling
from ci.compiler.clang_compiler import commands_json
PROJECT_ROOT: Path = Path(__file__).resolve().parent.parent
# Default to bespoke config for compile-commands generation
DEFAULT_CONFIG_PATH: Path = PROJECT_ROOT / "ci" / "build_commands.toml"
@typechecked
@dataclass
class CliConfig:
build_flags_toml: Path
exclude_dirs: list[str]
@typechecked
@dataclass
class Args:
output: Path
clean: bool
verbose: bool
config: Path
exclude: list[str]
def parse_args(args: Optional[list[str]] = None) -> Args:
parser = argparse.ArgumentParser(
description=(
"Generate compile_commands.json for clangd/clang-tidy based on explicit build configuration."
)
)
parser.add_argument(
"--output",
dest="output",
metavar="PATH",
help="Output path for compile_commands.json (default: project root)",
)
parser.add_argument(
"--config",
dest="config",
metavar="PATH",
help=(
"Path to BuildFlags-style TOML (default: ci/build_example.toml). "
"Must be parseable by BuildFlags.parse()."
),
)
parser.add_argument(
"--clean",
dest="clean",
action="store_true",
help="Rebuild command set from scratch (ignored currently; reserved)",
)
parser.add_argument(
"--verbose",
dest="verbose",
action="store_true",
help="Print collected flags/files",
)
parser.add_argument(
"--exclude-dir",
dest="exclude",
action="append",
default=[],
help="Relative directory under project root to exclude (can be used multiple times)",
)
parsed = parser.parse_args(args)
output_path_str: Optional[str] = None
if hasattr(parsed, "output"):
output_path_str = parsed.output
if output_path_str is None or len(output_path_str) == 0:
# Default to project root compile_commands.json
output_path = PROJECT_ROOT / "compile_commands.json"
else:
output_path = Path(output_path_str).resolve()
clean_value: bool = False
if hasattr(parsed, "clean"):
clean_value = bool(parsed.clean)
verbose_value: bool = False
if hasattr(parsed, "verbose"):
verbose_value = bool(parsed.verbose)
# Config path
config_path: Path
if getattr(parsed, "config", None):
config_path = Path(parsed.config).resolve()
else:
config_path = DEFAULT_CONFIG_PATH
# Excluded directories
exclude_list: list[str] = list(getattr(parsed, "exclude", []) or [])
return Args(
output=output_path,
clean=clean_value,
verbose=verbose_value,
config=config_path,
exclude=exclude_list,
)
def _assert_config_exists(config_path: Path) -> None:
if not config_path.exists():
raise RuntimeError(
f"CRITICAL: Required BuildFlags TOML not found at {config_path}. "
f"Please provide a valid configuration parseable by BuildFlags.parse()."
)
def _create_cli_config(build_flags_toml: Path, exclude: list[str]) -> CliConfig:
return CliConfig(build_flags_toml=build_flags_toml, exclude_dirs=exclude)
def _collect_source_files(root: Path, exclude: list[str], verbose: bool) -> list[Path]:
source_files: list[Path] = []
# Compute excluded directories as absolute paths for quick prefix checks
excluded_dirs_abs: list[Path] = []
for ex in exclude:
p = (PROJECT_ROOT / ex).resolve()
excluded_dirs_abs.append(p)
src_root: Path = PROJECT_ROOT / "src"
if not src_root.exists():
raise RuntimeError(
f"CRITICAL: Expected source root at {src_root} not found. "
f"This path is required to enumerate translation units."
)
for dirpath, dirnames, filenames in os.walk(src_root):
current_dir = Path(dirpath).resolve()
# Apply directory exclusions
skip_dir: bool = False
for ex in excluded_dirs_abs:
try:
current_dir.relative_to(ex)
skip_dir = True
break
except ValueError:
# Not under this excluded path
pass
if skip_dir:
# Prune search by clearing dirnames to avoid descending further
del dirnames[:]
continue
for fname in filenames:
if fname.endswith(".cpp") or fname.endswith(".c"):
source_files.append(current_dir / fname)
if verbose:
print(f"Discovered {len(source_files)} source files under {src_root}")
if len(source_files) == 0:
raise RuntimeError(
"CRITICAL: No source files discovered under 'src/'. "
"Please verify the repository layout and filters.exclude_dirs configuration."
)
return source_files
# No per-file argument composition here; delegated to commands_json()
# This function intentionally removed to rely on canonical toolchain
# Emission handled by commands_json()
def main(entry_args: Optional[list[str]] = None) -> int:
args = parse_args(entry_args)
# Log startup context
print("[compile-commands] Starting generation")
print(f"[compile-commands] Project root: {PROJECT_ROOT}")
print(f"[compile-commands] Config (BuildFlags TOML): {args.config}")
print(f"[compile-commands] Output path: {args.output}")
if args.exclude:
print(f"[compile-commands] Excluding directories: {args.exclude}")
else:
print("[compile-commands] No excluded directories configured")
# Determine BuildFlags TOML path
_assert_config_exists(args.config)
# Collect source files
source_root = PROJECT_ROOT / "src"
print(f"[compile-commands] Source root: {source_root}")
sources = _collect_source_files(
source_root,
exclude=args.exclude,
verbose=args.verbose,
)
print(f"[compile-commands] Discovered {len(sources)} source files")
# Delegate to canonical commands_json() using BuildFlags toolchain
print("[compile-commands] Generating compile_commands.json entries...")
commands_json(
config_path=args.config,
include_root=source_root,
sources=sources,
output_json=args.output,
quick_build=True,
strict_mode=False,
)
# Post-generation verification
if not args.output.exists():
print(
f"CRITICAL: Generation completed but output file was not found at {args.output}"
)
return 2
try:
text = args.output.read_text(encoding="utf-8")
# Best-effort count of entries to report status
import json as _json
from typing import cast
entries = cast(list[dict[str, Any]], _json.loads(text))
num_entries = len(entries) if isinstance(entries, list) else 0
print(
f"[compile-commands] Wrote {num_entries} entries to {args.output} (size: {args.output.stat().st_size} bytes)"
)
except Exception as e: # noqa: BLE001
print(
f"WARNING: Output written to {args.output} but could not parse JSON for summary: {e}"
)
if args.verbose:
print("[compile-commands] Done.")
return 0
if __name__ == "__main__":
try:
sys.exit(main())
except KeyboardInterrupt:
print("Operation interrupted by user")
_thread.interrupt_main()
raise
except FileNotFoundError as e:
print(f"CRITICAL: Required file not found: {e}")
sys.exit(1)
except ValueError as e:
print(f"CRITICAL: Invalid configuration value: {e}")
sys.exit(1)
except Exception as e: # noqa: BLE001
print(f"CRITICAL: Unexpected error: {e}")
raise
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiled_size.py
+115
View File
@@ -0,0 +1,115 @@
import argparse
import json
import re
import subprocess
from pathlib import Path
from typing import Any, Dict
def _create_board_info(path: Path) -> Dict[str, Any]:
build_info = json.loads(path.read_text())
assert build_info.keys(), f"No boards found in {build_info}"
assert len(build_info.keys()) == 1, (
f"Multiple boards found in {build_info}, so correct board should be specified"
)
return build_info[next(iter(build_info))]
def _find_build_info(board: str) -> Path:
# Prefer PIO layout
candidate = Path(".build") / "pio" / board / "build_info.json"
if candidate.exists():
return candidate
# Fallback legacy layout
candidate = Path(".build") / board / "build_info.json"
if candidate.exists():
return candidate
raise FileNotFoundError(
f"build_info.json not found for board '{board}' in .build/pio/{board} or .build/{board}"
)
def _run_pio_size(build_dir: Path) -> int | None:
try:
# Try to compute size without building first
result = subprocess.run(
["pio", "run", "-d", str(build_dir), "-t", "size"],
capture_output=True,
text=True,
check=False,
)
output = (result.stdout or "") + "\n" + (result.stderr or "")
m = re.search(r"Program:\s*(\d+)\s*bytes", output)
if m:
return int(m.group(1))
# If size target did not yield, try a full build then retry size
subprocess.run(
["pio", "run", "-d", str(build_dir)], capture_output=False, check=False
)
result = subprocess.run(
["pio", "run", "-d", str(build_dir), "-t", "size"],
capture_output=True,
text=True,
check=False,
)
output = (result.stdout or "") + "\n" + (result.stderr or "")
m = re.search(r"Program:\s*(\d+)\s*bytes", output)
if m:
return int(m.group(1))
except Exception:
pass
return None
def check_firmware_size(board: str) -> int:
build_info_json = _find_build_info(board)
board_info = _create_board_info(build_info_json)
assert board_info, f"Board {board} not found in {build_info_json}"
prog_path = Path(board_info["prog_path"]) # absolute or relative path
base_path = prog_path.parent
suffixes = [".bin", ".hex", ".uf2"]
firmware: Path | None = None
for suffix in suffixes:
candidate = base_path / f"firmware{suffix}"
if candidate.exists():
firmware = candidate
break
if firmware is None:
# As a fallback, if firmware.* doesn't exist, use ELF size (filesize) if present
if prog_path.exists():
return prog_path.stat().st_size
# As a last resort, run PlatformIO size target in the build directory
build_dir = build_info_json.parent
size = _run_pio_size(build_dir)
if size is not None:
return size
msg = (
", ".join([f"firmware{suffix}" for suffix in suffixes])
+ f" not found in {base_path}"
)
raise FileNotFoundError(msg)
return firmware.stat().st_size
def main(board: str):
try:
size = check_firmware_size(board)
print(f"Firmware size for {board}: {size} bytes")
except FileNotFoundError as e:
print(f"Error: {e}")
except json.JSONDecodeError:
print(f"Error: Unable to parse build_info.json for {board}")
except Exception as e:
print(f"An unexpected error occurred: {e}")
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Check FastLED firmware size for the specified board."
)
parser.add_argument(
"--board", type=str, required=True, help="Board to check firmware size for"
)
args = parser.parse_args()
main(args.board)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiled_size_history.py
+237
View File
@@ -0,0 +1,237 @@
#!/usr/bin/env python3
import argparse
import csv
import json
import os
import shutil
import subprocess
from pathlib import Path
from typing import List
import dateutil.parser # type: ignore
HERE = Path(__file__).resolve().parent
def run_command(command: str):
process = subprocess.Popen(
command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True
)
output, error = process.communicate()
return output.decode("utf-8"), error.decode("utf-8")
def step_back_commits(steps: int) -> bool:
step_back_command = f"git reset --hard HEAD~{steps}"
output, error = run_command(step_back_command)
if error:
print(f"Error stepping back {steps} commit(s): {error}")
return False
return True
def check_firmware_size(board: str) -> int:
root_build_dir = Path(".build") / board
build_info_json = root_build_dir / "build_info.json"
build_info = json.loads(build_info_json.read_text())
board_info = build_info.get(board)
assert board_info, f"Board {board} not found in {build_info_json}"
prog_path = Path(board_info["prog_path"])
base_path = prog_path.parent
suffixes = [".bin", ".hex", ".uf2"]
firmware: Path
for suffix in suffixes:
firmware = base_path / f"firmware{suffix}"
if firmware.exists():
break
else:
msg = (
", ".join([f"firmware{suffix}" for suffix in suffixes])
+ f" not found in {base_path}"
)
raise FileNotFoundError(msg)
size_command = f"du -b {firmware}"
output, error = run_command(size_command)
if error:
print(f"Error checking firmware size: {error}")
return -1
size_in_bytes = output.strip().split()[0]
return int(size_in_bytes)
def get_commit_hash() -> str | None:
hash_command = "git rev-parse HEAD"
output, error = run_command(hash_command)
if error:
print(f"Error getting commit hash: {error}")
return None
return output.strip()
def get_commit_date(commit_hash: str) -> str | None:
date_command = f"git show -s --format=%ci {commit_hash}"
output, error = run_command(date_command)
if error:
print(f"Error getting commit date: {error}")
return None
return dateutil.parser.parse(output.strip()).isoformat()
def main(
board: str,
num_commits: int,
skip_step: int,
start_commit: str | None = None,
end_commit: str | None = None,
):
# change to the script dir
os.chdir(str(HERE))
# Create tmp directory if it doesn't exist
if os.path.exists("tmp"):
shutil.rmtree("tmp")
os.makedirs("tmp", exist_ok=True)
# Change to the tmp directory
os.chdir("tmp")
# 1. Git clone FastLED repository
print("Cloning FastLED repository...")
clone_command = "git clone https://github.com/FastLED/FastLED.git"
output, error = run_command(clone_command)
# if error:
# print(f"Error cloning repository: {error}")
# os.chdir("..")
# return
# Change to the FastLED directory
os.chdir("FastLED")
# Checkout the latest commit
run_command("git checkout master")
# If end_commit is specified, checkout that commit
# if end_commit:
# print(f"Checking out end commit: {end_commit}")
# checkout_command = f"git checkout {end_commit}"
# output, error = run_command(checkout_command)
# #if error:
# # print(f"Error checking out end commit: {error}")
# # return
# Prepare CSV file
csv_filename = "../../firmware_sizes.csv"
with open(csv_filename, "w", newline="") as csvfile:
csvwriter = csv.writer(csvfile)
csvwriter.writerow(["datetime", "commit_hash", "binary_size"])
commits_checked = 0
first_iteration = True
while True:
current_commit = get_commit_hash()
if first_iteration and start_commit:
first_iteration = False
while True:
if current_commit == start_commit:
break
if not step_back_commits(1):
break
current_commit = get_commit_hash()
if num_commits and commits_checked >= num_commits:
print(f"Checked {num_commits} commits")
break
if end_commit and current_commit == end_commit:
print(f"Checked until end commit: {end_commit}")
break
# 2. Run ci-compile.py for current commit
print(f"\nChecking commit {commits_checked + 1}")
# remove .build/esp32dev/pio/build/esp32dev/ directory
board_files = Path(".build") / board / ".pio" / "build" / board
if board_files.exists():
shutil.rmtree(str(board_files), ignore_errors=True)
compile_command = f"uv run -m ci.ci-compile {board} --examples Blink"
output, error = run_command(compile_command)
if error:
print(f"Error running ci-compile.py: {error}")
if not step_back_commits(skip_step):
break
continue
# 3. Check firmware size and get commit hash
print("Checking firmware size...")
try:
size = check_firmware_size(board)
except FileNotFoundError as e:
print(f"Error checking firmware size: {e}")
if not step_back_commits(skip_step):
break
continue
except AssertionError as e:
print(f"Error: {e}")
if not step_back_commits(skip_step):
break
continue
commit_hash = get_commit_hash()
if size and commit_hash:
commit_date = get_commit_date(commit_hash)
print(f"Firmware size: {size} bytes")
# Write to CSV incrementally
with open(csv_filename, "a", newline="") as csvfile:
csvwriter = csv.writer(csvfile)
csvwriter.writerow([commit_date, commit_hash, size])
print(f"Result appended to {csv_filename}")
commits_checked += 1
# 4. Step back one commit
print("Stepping back 1 commit...")
if not step_back_commits(1):
break
# Don't remove the tmp directory
print("\nTemporary directory 'tmp' has been left intact for inspection.")
if __name__ == "__main__":
parser = argparse.ArgumentParser(
description="Check FastLED firmware size for multiple commits."
)
parser.add_argument(
"--num-commits", type=int, default=1, help="Number of commits to check"
)
parser.add_argument(
"--skip-step",
type=int,
default=1,
help="Number of commits to skip between checks",
)
parser.add_argument("--start-commit", type=str, help="Starting commit hash")
parser.add_argument("--end-commit", type=str, help="Ending commit hash")
parser.add_argument(
"--board", type=str, required=True, help="Board to check firmware size for"
)
args = parser.parse_args()
if args.start_commit or args.end_commit:
if not (args.start_commit and not args.end_commit):
print("Both start commit and end commit must be specified.")
exit(1)
# if start_commit is specified, end_commit must be specified
num_commits = args.num_commits
if args.start_commit and args.end_commit:
if args.start_commit == args.end_commit:
print("Start commit and end commit are the same.")
exit(1)
num_commits = 999999
main(args.board, num_commits, args.skip_step, args.start_commit, args.end_commit)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/__init__.py
+5
View File
@@ -0,0 +1,5 @@
"""
FastLED Compiler Module
High-performance compilation system built on proven Python compiler API.
"""
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/build_dynamic_lib.py
+196
View File
@@ -0,0 +1,196 @@
#!/usr/bin/env python3
"""
FastLED Dynamic Library Builder
Builds FastLED as a shared/dynamic library for unit testing
"""
import os
import subprocess
import sys
from pathlib import Path
from typing import List
from ci.compiler.clang_compiler import (
BuildFlags,
Compiler,
CompilerOptions,
Result,
)
from ci.util.paths import PROJECT_ROOT
def build_fastled_dynamic_library(build_dir: Path) -> Path:
"""Build FastLED as a dynamic library"""
print("Building FastLED dynamic library...")
# Define library path with appropriate extension
if sys.platform == "win32":
fastled_lib_path = build_dir / "fastled.dll"
else:
fastled_lib_path = build_dir / "libfastled.so"
# Configure compiler with shared library flags
settings = CompilerOptions(
include_path=str(Path(PROJECT_ROOT) / "src"),
defines=[
"STUB_PLATFORM",
"ARDUINO=10808",
"FASTLED_USE_STUB_ARDUINO",
"SKETCH_HAS_LOTS_OF_MEMORY=1",
"FASTLED_STUB_IMPL",
"FASTLED_FORCE_NAMESPACE=1",
"FASTLED_TESTING",
"FASTLED_NO_AUTO_NAMESPACE",
"FASTLED_NO_PINMAP",
"HAS_HARDWARE_PIN_SUPPORT",
# "FASTLED_DEBUG_LEVEL=1",
# "BUILDING_FASTLED_DLL", # New define for DLL exports
# "FASTLED_EXPORT=__declspec(dllexport)" if sys.platform == "win32" else "FASTLED_EXPORT=", # DLL exports for Windows
],
std_version="c++17",
compiler="clang++",
compiler_args=[
# NOTE: All compiler flags should come from build_unit.toml
# Keep only platform-specific include paths and shared library flags
"-I" + str(Path(PROJECT_ROOT) / "src/platforms/stub"),
"-I" + str(Path(PROJECT_ROOT) / "tests"),
# Add shared library flags - platform specific
"-shared" if sys.platform != "win32" else "/DLL",
"-fPIC" if sys.platform != "win32" else "",
],
use_pch=True,
parallel=True,
)
# Load build flags from TOML
build_flags_path = Path(PROJECT_ROOT) / "ci" / "build_unit.toml"
build_flags = BuildFlags.parse(
build_flags_path, quick_build=False, strict_mode=False
)
compiler = Compiler(settings, build_flags)
# Compile all FastLED source files
fastled_src_dir = Path(PROJECT_ROOT) / "src"
all_cpp_files = list(fastled_src_dir.rglob("*.cpp"))
print(f"Found {len(all_cpp_files)} FastLED source files")
# Compile to object files
object_files: List[Path] = []
for src_file in all_cpp_files:
relative_path = src_file.relative_to(fastled_src_dir)
safe_name = (
str(relative_path.with_suffix("")).replace("/", "_").replace("\\", "_")
)
obj_path = build_dir / f"{safe_name}_fastled.o"
future = compiler.compile_cpp_file(
src_file,
output_path=obj_path,
additional_flags=[
"-c",
"-DFASTLED_STUB_IMPL",
"-DFASTLED_FORCE_NAMESPACE=1",
"-DFASTLED_NO_AUTO_NAMESPACE",
"-DFASTLED_NO_PINMAP",
"-DPROGMEM=",
"-DHAS_HARDWARE_PIN_SUPPORT",
"-DFASTLED_ENABLE_JSON=1",
"-fno-exceptions",
"-fno-rtti",
"-fPIC", # Position Independent Code for shared library
],
)
result: Result = future.result()
if not result.ok:
print(f"ERROR: Failed to compile {src_file}: {result.stderr}")
continue
object_files.append(obj_path)
print(f"Linking {len(object_files)} object files into dynamic library...")
# Link as shared library
if sys.platform == "win32":
# Use lld-link for Windows DLL creation
link_cmd = [
"lld-link",
"/DLL",
"/NOLOGO",
"/OUT:" + str(fastled_lib_path),
"/LIBPATH:C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/um/x64",
"/LIBPATH:C:/Program Files (x86)/Windows Kits/10/Lib/10.0.19041.0/ucrt/x64",
"/LIBPATH:C:/Program Files/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.37.32822/lib/x64",
*[str(obj) for obj in object_files],
"msvcrt.lib",
"legacy_stdio_definitions.lib",
"kernel32.lib",
"user32.lib",
]
else:
link_cmd = [
"clang++",
"-shared",
"-o",
str(fastled_lib_path),
*[str(obj) for obj in object_files],
]
try:
# Use streaming to prevent buffer overflow
process = subprocess.Popen(
link_cmd,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
bufsize=1,
encoding="utf-8",
errors="replace",
)
stdout_lines: list[str] = []
stderr_lines: list[str] = []
while True:
stdout_line = process.stdout.readline() if process.stdout else ""
stderr_line = process.stderr.readline() if process.stderr else ""
if stdout_line:
stdout_lines.append(stdout_line.rstrip())
if stderr_line:
stderr_lines.append(stderr_line.rstrip())
if process.poll() is not None:
remaining_stdout = process.stdout.read() if process.stdout else ""
remaining_stderr = process.stderr.read() if process.stderr else ""
if remaining_stdout:
for line in remaining_stdout.splitlines():
stdout_lines.append(line.rstrip())
if remaining_stderr:
for line in remaining_stderr.splitlines():
stderr_lines.append(line.rstrip())
break
stderr_result = "\n".join(stderr_lines)
if process.returncode != 0:
raise Exception(f"Failed to create dynamic library: {stderr_result}")
print(f"Successfully created dynamic library: {fastled_lib_path}")
return fastled_lib_path
except Exception as e:
print(f"ERROR: Exception during library creation: {e}")
raise
if __name__ == "__main__":
# Create build directory in temp
import tempfile
build_dir = Path(tempfile.gettempdir()) / "fastled_test_build"
build_dir.mkdir(parents=True, exist_ok=True)
# Build the dynamic library
build_fastled_dynamic_library(build_dir)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/cache_enhanced_compilation.py
+273
View File
@@ -0,0 +1,273 @@
#!/usr/bin/env python3
"""
Clean Cache-Enhanced Compilation
Simple wrapper that adds fingerprint cache to FastLED compilation.
Dramatically speeds up incremental builds by skipping unchanged files.
"""
import time
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional, Set
from ci.ci.fingerprint_cache import FingerprintCache
from ci.compiler.clang_compiler import Compiler
from ci.compiler.test_example_compilation import CompilationResult
class CacheAwareCompiler:
"""Simple wrapper that adds cache checking to compilation."""
def __init__(self, compiler: Compiler, cache_file: Path, verbose: bool = False):
self.compiler = compiler
self.cache = FingerprintCache(cache_file)
self.verbose = verbose
self.stats = {
"files_checked": 0,
"files_skipped": 0,
"files_compiled": 0,
"cache_hits": 0,
"cache_misses": 0,
}
def should_compile(self, file_path: Path, baseline_time: float) -> bool:
"""Check if file needs compilation using cache."""
self.stats["files_checked"] += 1
try:
needs_compile = self.cache.has_changed(file_path, baseline_time)
if needs_compile:
self.stats["cache_misses"] += 1
self.stats["files_compiled"] += 1
return True
else:
self.stats["cache_hits"] += 1
self.stats["files_skipped"] += 1
if self.verbose:
print(f"[CACHE] Skipping unchanged: {file_path.name}")
return False
except KeyboardInterrupt:
import _thread
_thread.interrupt_main()
raise
except Exception as e:
if self.verbose:
print(f"[CACHE] Error checking {file_path.name}: {e}")
self.stats["cache_misses"] += 1
self.stats["files_compiled"] += 1
return True
def _headers_changed(self, baseline_time: float) -> bool:
"""Detect if any relevant header dependency has changed since last run.
Uses the compiler's PCH dependency discovery to collect a conservative
set of headers that impact example compilation, then queries the
fingerprint cache to see if any actually changed content.
"""
try:
# Reuse the compiler's dependency discovery (covers src/** and platforms/**)
dependencies: List[Path] = []
if hasattr(self.compiler, "_get_pch_dependencies"):
dependencies = self.compiler._get_pch_dependencies() # type: ignore[attr-defined]
else:
# Fallback: hash all headers under the compiler's include path
include_root = Path(self.compiler.settings.include_path)
for pattern in ("**/*.h", "**/*.hpp"):
dependencies.extend(include_root.glob(pattern))
changed_any = False
for dep in dependencies:
try:
if self.cache.has_changed(dep, baseline_time):
if self.verbose:
print(f"[CACHE] Header changed: {dep}")
changed_any = True
break
except FileNotFoundError:
# Missing dependency means we must rebuild conservatively
if self.verbose:
print(f"[CACHE] Header missing (forces rebuild): {dep}")
changed_any = True
break
return changed_any
except KeyboardInterrupt:
import _thread
_thread.interrupt_main()
raise
except Exception as e:
if self.verbose:
print(f"[CACHE] Header scan failed, forcing rebuild: {e}")
return True
def compile_with_cache(
self,
ino_files: List[Path],
pch_compatible_files: Set[Path],
log_fn: Callable[[str], None],
full_compilation: bool,
verbose: bool = False,
baseline_time: Optional[float] = None,
) -> Dict[str, Any]:
"""Compile only files that have changed."""
if baseline_time is None:
baseline_time = time.time() - 3600 # Default: 1 hour ago
start_time = time.time()
# If any header dependency changed, conservatively rebuild all example files
force_recompile_due_to_headers = self._headers_changed(baseline_time)
if force_recompile_due_to_headers and self.verbose:
print(
"[CACHE] Header dependency changes detected - forcing recompilation of example sources"
)
# Check which files need compilation
files_to_compile: List[Path] = []
for ino_file in ino_files:
if force_recompile_due_to_headers or self.should_compile(
ino_file, baseline_time
):
files_to_compile.append(ino_file)
# Check .cpp files too
cpp_files_to_compile: List[Path] = []
for ino_file in ino_files:
cpp_files = self.compiler.find_cpp_files_for_example(ino_file)
for cpp_file in cpp_files:
if force_recompile_due_to_headers or self.should_compile(
cpp_file, baseline_time
):
cpp_files_to_compile.append(cpp_file)
# Log cache results
total_files = len(ino_files) + sum(
len(self.compiler.find_cpp_files_for_example(f)) for f in ino_files
)
log_fn(
f"[CACHE] {self.stats['files_skipped']}/{total_files} files unchanged, {self.stats['files_compiled']}/{total_files} need compilation"
)
if self.stats["files_skipped"] > 0:
time_saved = self.stats["files_skipped"] * 0.5 # Estimate
log_fn(f"[CACHE] Estimated time saved: {time_saved:.1f}s")
# Compile files that changed
if files_to_compile or cpp_files_to_compile:
result = self._run_actual_compilation(
files_to_compile,
cpp_files_to_compile,
pch_compatible_files,
log_fn,
full_compilation,
verbose,
)
else:
log_fn("[CACHE] All files unchanged - no compilation needed!")
result = self._create_success_result(len(ino_files), ino_files)
# Log final stats
if total_files > 0:
hit_rate = (
(
self.stats["cache_hits"]
/ (self.stats["cache_hits"] + self.stats["cache_misses"])
* 100
)
if (self.stats["cache_hits"] + self.stats["cache_misses"]) > 0
else 0
)
skip_rate = self.stats["files_skipped"] / total_files * 100
log_fn(
f"[CACHE] {hit_rate:.1f}% cache hit rate, {skip_rate:.1f}% files skipped"
)
return {
"compilation_result": result,
"cache_stats": self.stats,
"total_time": time.time() - start_time,
}
def _run_actual_compilation(
self,
ino_files: List[Path],
cpp_files_to_compile: List[Path],
pch_compatible_files: Set[Path],
log_fn: Callable[[str], None],
full_compilation: bool,
verbose: bool,
):
"""Run the actual compilation for changed files."""
from ci.compiler.test_example_compilation import compile_examples_simple
# Temporarily filter .cpp files to only compile changed ones
original_method = self.compiler.find_cpp_files_for_example
def filtered_cpp_files(ino_file: Path) -> List[Path]:
all_cpp = original_method(ino_file)
return [cpp for cpp in all_cpp if cpp in cpp_files_to_compile]
self.compiler.find_cpp_files_for_example = filtered_cpp_files
try:
return compile_examples_simple(
self.compiler,
ino_files,
pch_compatible_files,
log_fn,
full_compilation,
verbose,
)
finally:
self.compiler.find_cpp_files_for_example = original_method
def _create_success_result(
self, file_count: int, ino_files: Optional[List[Path]] = None
) -> CompilationResult:
"""Create a successful compilation result for cached files."""
# For cached files, we need to populate object_file_map with existing object files
# so that linking can proceed properly
object_file_map: Dict[Path, List[Path]] = {}
if ino_files:
for ino_file in ino_files:
# Find existing object files for this example
example_name = ino_file.parent.name
build_dir = Path(".build/examples") / example_name
obj_files: List[Path] = []
if build_dir.exists():
# Look for the main .ino object file
ino_obj = build_dir / f"{ino_file.stem}.o"
if ino_obj.exists():
obj_files.append(ino_obj)
# Look for additional .cpp files in the same directory as the .ino
cpp_files = self.compiler.find_cpp_files_for_example(ino_file)
for cpp_file in cpp_files:
cpp_obj = build_dir / f"{cpp_file.stem}.o"
if cpp_obj.exists():
obj_files.append(cpp_obj)
if obj_files:
object_file_map[ino_file] = obj_files
return CompilationResult(
successful_count=file_count,
failed_count=0,
compile_time=0.0,
failed_examples=[],
object_file_map=object_file_map,
)
def create_cache_compiler(
compiler: Compiler, cache_file: Optional[Path] = None, verbose: bool = False
) -> CacheAwareCompiler:
"""Create a cache-aware compiler. Simple factory function."""
if cache_file is None:
cache_file = Path(".build/fingerprint_cache.json")
return CacheAwareCompiler(compiler, cache_file, verbose)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/cache_setup.scons
+514
View File
@@ -0,0 +1,514 @@
#!/usr/bin/env python3
"""
Standalone cache setup script for PlatformIO builds with Python fake compilers.
This script is executed by PlatformIO as a post-build extra_script.
Configuration is passed through environment variables to avoid template string issues.
"""
# ruff: noqa: F405, F821 # Suppress SCons-specific import and undefined name warnings
import json
import os
import shutil
import sys
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
# Import env and try to import projenv (SCons-specific imports)
# These are dynamically available in PlatformIO/SCons environment
try:
Import("env") # type: ignore[name-defined] # SCons-specific import
env: Any # SCons environment object
except NameError:
env = None # For type checking when not in SCons context
_VERBOSE = os.environ.get("VERBOSE", "0") in ("1", "true", "True", "yes", "y")
# Try to import projenv if it exists
has_projenv: bool = False
projenv: Optional[Any] = None
try:
Import("projenv") # type: ignore[name-defined] # SCons-specific import
# projenv is now available in scope from Import
has_projenv = True
except (NameError, Exception):
has_projenv = False
projenv = None
# Import cached compiler functions with type stubs
create_cached_toolchain: Optional[Any] = None
get_platform_packages_paths: Optional[Any] = None
cached_compiler_available: bool = False
try:
from ci.util.cached_compiler import (
create_cached_toolchain,
get_platform_packages_paths,
)
cached_compiler_available = True
except ImportError as e:
print("WARNING: Could not import cached compiler module: " + str(e))
cached_compiler_available = False
# Debug: Dump the environment state to disk for inspection
env_dump: Dict[str, str] = {}
try:
if env is not None and hasattr(env, "Dictionary"): # type: ignore[has-type]
for key in env.Dictionary(): # type: ignore[union-attr]
try:
value = env[key] # type: ignore[index]
# Convert to string to avoid JSON serialization issues
env_dump[key] = str(value) # type: ignore[arg-type]
except Exception:
env_dump[key] = "<error getting value>"
else:
env_dump["error"] = "env not available or missing Dictionary method"
except Exception as e:
env_dump["error"] = f"Failed to access env: {e}"
# Write environment dump to disk
env_dump_path = "env_dump.json"
with open(env_dump_path, "w") as f:
json.dump(env_dump, f, indent=2)
print("Environment state dumped to: " + env_dump_path)
# Also dump projenv if available
if has_projenv and projenv is not None:
projenv_dump: Dict[str, str] = {}
for key in projenv.Dictionary(): # type: ignore
try:
value = projenv[key] # type: ignore
projenv_dump[key] = str(value)
except Exception:
projenv_dump[key] = "<error getting value>"
projenv_dump_path = "projenv_dump.json"
with open(projenv_dump_path, "w") as f:
json.dump(projenv_dump, f, indent=2)
print("Projenv state dumped to: " + projenv_dump_path)
# Read cache configuration from environment variables
cache_type = os.environ.get("FASTLED_CACHE_TYPE", "no_cache")
cache_executable = os.environ.get("FASTLED_CACHE_EXECUTABLE", "")
sccache_path = os.environ.get("FASTLED_SCCACHE_PATH", "")
sccache_dir = os.environ.get("FASTLED_SCCACHE_DIR", "")
sccache_cache_size = os.environ.get("FASTLED_SCCACHE_CACHE_SIZE", "2G")
xcache_path = os.environ.get("FASTLED_XCACHE_PATH", "")
debug_enabled = os.environ.get("FASTLED_CACHE_DEBUG", "0") == "1"
print("Cache configuration from environment:")
print(" Cache type: " + cache_type)
print(" Cache executable: " + cache_executable)
print(" SCCACHE path: " + sccache_path)
print(" SCCACHE dir: " + sccache_dir)
print(" Debug enabled: " + str(debug_enabled))
# Set up cache environment variables for subprocess execution
try:
if env is not None and hasattr(env, "Append"): # type: ignore[has-type]
if sccache_dir:
env.Append(ENV={"SCCACHE_DIR": sccache_dir}) # type: ignore[union-attr]
os.environ["SCCACHE_DIR"] = sccache_dir
if sccache_cache_size:
env.Append(ENV={"SCCACHE_CACHE_SIZE": sccache_cache_size}) # type: ignore[union-attr]
os.environ["SCCACHE_CACHE_SIZE"] = sccache_cache_size
# Ensure sccache binary directory is on PATH so xcache can find it even
# inside ESP-IDF's virtual environments where PATH is heavily modified.
if sccache_path:
sccache_dir_path = str(Path(sccache_path).parent)
env.PrependENVPath("PATH", sccache_dir_path) # type: ignore[union-attr]
if has_projenv and projenv is not None and hasattr(projenv, "PrependENVPath"):
projenv.PrependENVPath("PATH", sccache_dir_path) # type: ignore[union-attr]
os.environ["PATH"] = sccache_dir_path + os.pathsep + os.environ.get("PATH", "")
if debug_enabled:
env.Append(ENV={"XCACHE_DEBUG": "1", "SCCACHE_DEBUG": "1"}) # type: ignore[union-attr]
if has_projenv and projenv is not None and hasattr(projenv, "Append"):
projenv.Append(ENV={"XCACHE_DEBUG": "1", "SCCACHE_DEBUG": "1"}) # type: ignore[union-attr]
except Exception as e:
print(f"Warning: Failed to set up cache environment: {e}")
# Check if cache is available and cached compiler system can be used
USE_CACHE = False
if cached_compiler_available and cache_executable:
if cache_type == "xcache":
# For xcache, check if the Python script exists and sccache is available
USE_CACHE = (
cache_executable
and xcache_path
and Path(xcache_path).exists()
and sccache_path
and shutil.which(sccache_path)
)
if USE_CACHE:
print("xcache wrapper detected and configured for Python fake compilers")
print(" xcache path: " + str(xcache_path))
print(" cache executable: " + str(cache_executable))
else:
# For sccache/ccache, check if executable is in PATH
USE_CACHE = shutil.which(cache_executable) is not None
if USE_CACHE:
print(
str(cache_type) + " detected and configured for Python cached compilers"
)
print(" cache executable: " + str(cache_executable))
elif not cached_compiler_available:
print(
"WARNING: Python cached compiler system not available, cache will be disabled"
)
else:
print(
"Cache executable not found: "
+ str(cache_executable)
+ ", cache will be disabled"
)
if USE_CACHE and env is not None and hasattr(env, "get"): # type: ignore[has-type]
# Get current compilers from environment
original_cc = env.get("CC") # type: ignore[union-attr]
original_cxx = env.get("CXX") # type: ignore[union-attr]
print("DEBUG: Found compilers in env:")
# Use repr for safer type conversion
cc_str = repr(original_cc)
cxx_str = repr(original_cxx)
cc_type_str = type(original_cc).__name__
cxx_type_str = type(original_cxx).__name__
print(" CC: " + cc_str + " (type: " + cc_type_str + ")")
print(" CXX: " + cxx_str + " (type: " + cxx_type_str + ")")
# Extract compiler information for fake compiler generation
def extract_compiler_info(compiler_env_var: Any) -> Optional[str]:
"""Extract compiler name from environment variable value."""
if not compiler_env_var:
return None
if isinstance(compiler_env_var, list):
return str(compiler_env_var[0]) if compiler_env_var else None # type: ignore[arg-type]
else:
# Handle string values like "arm-none-eabi-gcc" or "gcc"
return str(compiler_env_var).split()[0] # type: ignore[arg-type]
cc_name = extract_compiler_info(original_cc) or "gcc"
cxx_name = extract_compiler_info(original_cxx) or "g++"
print("Extracted compiler names:")
print(" CC name: " + str(cc_name))
print(" CXX name: " + str(cxx_name))
# Create toolchain info for fake compiler generation
toolchain_info: Dict[str, str] = {
"CC": cc_name,
"CXX": cxx_name,
}
# Create cache config for fake compiler system
cache_config: Dict[str, str] = {
"CACHE_TYPE": cache_type,
"CACHE_EXECUTABLE": cache_executable,
"SCCACHE_PATH": sccache_path,
"SCCACHE_DIR": sccache_dir,
"XCACHE_PATH": xcache_path,
}
# Check if toolset exists and is valid
# Create a cache key based on the configuration to invalidate cache when config changes
cache_key = f"{cc_name}_{cxx_name}_{cache_type}_{cache_executable}"
# Cache in the local build directory (e.g., .build/pio/uno/)
cache_file = Path(current_dir) / "compiler_cache.json"
cached_tools: Optional[Dict[str, str]] = None
# Try to load from persistent cache file
if cache_file.exists():
try:
import json
with open(cache_file, "r") as f:
cache_data = json.load(f)
cached_real_cc = cache_data.get("real_cc")
cached_real_cxx = cache_data.get("real_cxx")
cached_fake_cc = cache_data.get("fake_cc")
cached_fake_cxx = cache_data.get("fake_cxx")
if (
cached_real_cc
and cached_real_cxx
and cached_fake_cc
and cached_fake_cxx
):
print("Found local compiler cache:")
print(f" Cache file: {cache_file}")
print(f" Real CC: {cached_real_cc}")
print(f" Real CXX: {cached_real_cxx}")
# Check if fake compiler scripts still exist, recreate if needed
fake_cc_path = Path(cached_fake_cc.replace("python ", ""))
fake_cxx_path = Path(cached_fake_cxx.replace("python ", ""))
if fake_cc_path.exists() and fake_cxx_path.exists():
# Fast path: use existing cached compilers
# On Windows, ensure we have .cmd shims to avoid 'python script.py @file'
# causing Python to treat @file as the script name.
import sys as _sys
import os as _os
is_win = (
_os.name == "nt"
or _sys.platform.startswith("win")
or _sys.platform.startswith("cygwin")
or _sys.platform.startswith("msys")
)
def _ensure_cmd_wrapper(fake_str: str, real_path: str) -> str:
p = Path(fake_str.replace("python ", ""))
if is_win and p.suffix.lower() == ".py":
cmd = p.with_suffix(".cmd")
if not cmd.exists():
# Create missing .cmd wrapper now
from ci.util.cached_compiler import create_cached_compiler_script
out_dir = p.parent
created = create_cached_compiler_script(
compiler_name=p.stem.split("cached_")[-1],
cache_executable=cache_config.get("CACHE_EXECUTABLE", "sccache"),
real_compiler_path=real_path,
output_dir=out_dir,
debug=debug_enabled,
)
cmd = Path(str(created).replace("python ", ""))
if cmd.exists():
return str(cmd)
return fake_str
# Create/choose proper wrapper paths
new_cc = _ensure_cmd_wrapper(cached_fake_cc, cached_real_cc)
new_cxx = _ensure_cmd_wrapper(cached_fake_cxx, cached_real_cxx)
cached_tools = {"CC": new_cc, "CXX": new_cxx}
# Update cache file if we switched to .cmd
try:
if new_cc != cached_fake_cc or new_cxx != cached_fake_cxx:
cache_data["fake_cc"] = new_cc
cache_data["fake_cxx"] = new_cxx
with open(cache_file, "w") as f:
json.dump(cache_data, f, indent=2)
except KeyboardInterrupt as ke:
import _thread
_thread.interrupt_main()
raise ke
except Exception:
pass
print(
"SUCCESS: Using cached compilers (instant, no platform search needed):"
)
print(f" CC: {cached_tools['CC']}")
print(f" CXX: {cached_tools['CXX']}")
print(" Platform search skipped - using cached toolset")
else:
print(
"Cached compiler scripts missing, recreating with cached real paths..."
)
# Recreate cached compilers using cached real paths (fast)
cached_compilers_dir = Path(current_dir) / "cached_compilers"
cached_compilers_dir.mkdir(parents=True, exist_ok=True)
from ci.util.cached_compiler import create_cached_compiler_script
cache_executable = cache_config.get("CACHE_EXECUTABLE", "sccache")
# Create cached CC script using cached real path
cached_cc_script = create_cached_compiler_script(
compiler_name="CC",
cache_executable=cache_executable,
real_compiler_path=cached_real_cc,
output_dir=cached_compilers_dir,
debug=debug_enabled,
)
# Create cached CXX script using cached real path
cached_cxx_script = create_cached_compiler_script(
compiler_name="CXX",
cache_executable=cache_executable,
real_compiler_path=cached_real_cxx,
output_dir=cached_compilers_dir,
debug=debug_enabled,
)
cached_tools = {
"CC": str(cached_cc_script),
"CXX": str(cached_cxx_script),
}
# Update cache file with new script paths
cache_data["fake_cc"] = cached_tools["CC"]
cache_data["fake_cxx"] = cached_tools["CXX"]
with open(cache_file, "w") as f:
json.dump(cache_data, f, indent=2)
print("Recreated cached compilers using cached real paths:")
print(f" CC: {cached_tools['CC']}")
print(f" CXX: {cached_tools['CXX']}")
except Exception as e:
print(f"Warning: Failed to load cache file {cache_file}: {e}")
# Fall through to full recreation
# If no valid cache found, create the toolset from scratch
if cached_tools is None:
print("No valid cache found, creating compiler toolset from scratch...")
print(" This is the first compile or configuration changed")
# Get platform packages paths for toolchain resolution (expensive operation)
platform_packages: List[str] = []
if get_platform_packages_paths is not None:
print("Searching platform packages (this may take ~10 seconds)...")
platform_packages = get_platform_packages_paths()
print(f"Found {len(platform_packages)} platform package directories")
if create_cached_toolchain is not None:
try:
# Find the real compiler paths (expensive operation, done only once)
from ci.util.cached_compiler import find_toolchain_compiler
print("Resolving real compiler paths...")
real_cc_path = find_toolchain_compiler(cc_name, platform_packages)
real_cxx_path = find_toolchain_compiler(cxx_name, platform_packages)
if not real_cc_path or not real_cxx_path:
print(f"ERROR: Could not find real compilers:")
print(f" CC '{cc_name}': {real_cc_path}")
print(f" CXX '{cxx_name}': {real_cxx_path}")
cached_tools = None
else:
print(f"Found real compilers:")
print(f" Real CC: {real_cc_path}")
print(f" Real CXX: {real_cxx_path}")
# Create cached compiler scripts
cached_compilers_dir = Path(current_dir) / "cached_compilers"
cached_compilers_dir.mkdir(parents=True, exist_ok=True)
from ci.util.cached_compiler import create_cached_compiler_script
cache_executable = cache_config.get("CACHE_EXECUTABLE", "sccache")
# Create cached CC script
cached_cc_script = create_cached_compiler_script(
compiler_name="CC",
cache_executable=cache_executable,
real_compiler_path=real_cc_path,
output_dir=cached_compilers_dir,
debug=debug_enabled,
)
# Create cached CXX script
cached_cxx_script = create_cached_compiler_script(
compiler_name="CXX",
cache_executable=cache_executable,
real_compiler_path=real_cxx_path,
output_dir=cached_compilers_dir,
debug=debug_enabled,
)
cached_tools = {
"CC": str(cached_cc_script),
"CXX": str(cached_cxx_script),
}
print("Created new compiler toolset:")
print(f" CC: {cached_tools['CC']}")
print(f" CXX: {cached_tools['CXX']}")
# Save to local build directory cache file
cache_data = {
"cache_key": cache_key,
"real_cc": real_cc_path,
"real_cxx": real_cxx_path,
"fake_cc": cached_tools["CC"],
"fake_cxx": cached_tools["CXX"],
"build_dir": str(current_dir),
"platform_packages_count": len(platform_packages),
}
with open(cache_file, "w") as f:
json.dump(cache_data, f, indent=2)
print(f"Saved compiler toolset to local cache: {cache_file}")
print(" This cache will persist across builds for this platform")
except Exception as e:
print("ERROR: Toolset creation failed with exception: " + str(e))
import traceback
traceback.print_exc()
cached_tools = None
else:
print("ERROR: create_cached_toolchain function is None")
if cached_tools:
# Use Python cached compilers instead of batch scripts
new_cc = cached_tools.get("CC")
new_cxx = cached_tools.get("CXX")
if new_cc and new_cxx:
print("Created Python cached compilers:")
print(" CC: " + str(new_cc))
print(" CXX: " + str(new_cxx))
# Apply to both environments
env.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
if has_projenv and projenv is not None:
projenv.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
print("Applied Python fake compilers to both env and projenv")
else:
print("Applied Python fake compilers to env (projenv not available)")
# Apply to library builders (critical for framework caching)
try:
for lib_builder in env.GetLibBuilders(): # type: ignore
lib_builder.env.Replace(CC=new_cc, CXX=new_cxx) # type: ignore
if _VERBOSE:
print(
"Applied Python fake compilers to library builder: "
+ str(getattr(lib_builder, "name", "unnamed"))
)
except KeyboardInterrupt:
import _thread
_thread.interrupt_main()
raise
except Exception as e:
print("WARNING: Could not apply to library builders: " + str(e))
print("Python fake compiler cache enabled: " + str(cache_type))
print(" Original CC: " + str(original_cc))
print(" Original CXX: " + str(original_cxx))
print(" Fake CC: " + str(new_cc))
print(" Fake CXX: " + str(new_cxx))
else:
print(
"ERROR: Failed to create Python fake compilers, falling back to no cache"
)
USE_CACHE = False
else:
print("ERROR: Python fake compiler creation failed, falling back to no cache")
USE_CACHE = False
if not USE_CACHE:
if cache_executable:
print("Warning: " + str(cache_type) + " setup failed; using default compilers")
else:
print("No cache executable configured; using default compilers")
print("Python fake compiler cache environment configured successfully")
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/clang_compiler.py
+3162
View File
File diff suppressed because it is too large. Load diff
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/compile_for_board.py
+261
View File
@@ -0,0 +1,261 @@
# pyright: reportUnknownMemberType=false
"""
Compilation support for individual boards.
"""
import os
import shutil
import subprocess
import time
from pathlib import Path
from threading import Lock
from typing import List
from ci.boards import Board # type: ignore
from ci.util.locked_print import locked_print
ERROR_HAPPENED = False
IS_GITHUB = "GITHUB_ACTIONS" in os.environ
FIRST_BUILD_LOCK = Lock()
USE_FIRST_BUILD_LOCK = IS_GITHUB
def errors_happened() -> bool:
"""Return whether any errors happened during the build."""
return ERROR_HAPPENED
def _fastled_js_is_parent_directory(p: Path) -> bool:
"""Check if fastled_js is a parent directory of the given path."""
# Check if fastled_js is a parent directory of p
return "fastled_js" in str(p.absolute())
def compile_for_board_and_example(
board: Board,
example: Path,
build_dir: str | None,
verbose_on_failure: bool,
libs: list[str] | None,
) -> tuple[bool, str]:
"""Compile the given example for the given board."""
global ERROR_HAPPENED # pylint: disable=global-statement
if board.board_name == "web":
locked_print(f"Skipping web target for example {example}")
return True, ""
board_name = board.board_name
use_pio_run = board.use_pio_run
real_board_name = board.get_real_board_name()
libs = libs or []
builddir = (
Path(build_dir) / board_name if build_dir else Path(".build") / board_name
)
builddir.mkdir(parents=True, exist_ok=True)
srcdir = builddir / "src"
# Remove the previous *.ino file if it exists, everything else is recycled
# to speed up the next build.
if srcdir.exists():
shutil.rmtree(srcdir, ignore_errors=False)
locked_print(f"*** Building example {example} for board {board_name} ***")
cwd: str | None = None
shell: bool = False
# Copy all files from the example directory to the "src" directory
for src_file in example.rglob("*"):
if src_file.is_file():
if _fastled_js_is_parent_directory(src_file):
# Skip the fastled_js folder, it's not needed for the build.
continue
src_dir = src_file.parent
path = src_dir.relative_to(example)
dst_dir = srcdir / path
os.makedirs(dst_dir, exist_ok=True)
locked_print(f"Copying {src_file} to {dst_dir / src_file.name}")
os.makedirs(srcdir, exist_ok=True)
shutil.copy(src_file, dst_dir / src_file.name)
# libs = ["src", "ci"]
if use_pio_run:
# we have to copy a few folders of pio ci in order to get this to work.
for lib in libs:
project_libdir = Path(lib)
assert project_libdir.exists()
build_lib = builddir / "lib" / lib
shutil.rmtree(build_lib, ignore_errors=True)
shutil.copytree(project_libdir, build_lib)
cwd = str(builddir)
cmd_list = [
"pio",
"run",
]
# in this case we need to manually copy the example to the src directory
# because platformio doesn't support building a single file.
# ino_file = example / f"{example.name}.ino"
else:
cmd_list = [
"pio",
"ci",
"--board",
real_board_name,
*[f"--lib={lib}" for lib in libs],
"--keep-build-dir",
f"--build-dir={builddir.as_posix()}",
]
cmd_list.append(f"{example.as_posix()}/*ino")
cmd_str = subprocess.list2cmdline(cmd_list)
msg_lsit = [
"\n\n******************************",
f"* Running command in cwd: {cwd if cwd else os.getcwd()}",
f"* {cmd_str}",
"******************************\n",
]
msg = "\n".join(msg_lsit)
locked_print(msg)
# Start timing for the process
start_time = time.time()
# Run the process with real-time output capture and timing
result = subprocess.Popen(
cmd_list,
cwd=cwd,
shell=shell,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)
# Capture output lines in real-time with timing
stdout_lines: List[str] = []
if result.stdout:
for line in iter(result.stdout.readline, ""):
if line:
elapsed = time.time() - start_time
# Format timing as seconds with 2 decimal places
timing_prefix = f"{elapsed:5.2f} "
timed_line = timing_prefix + line.rstrip()
stdout_lines.append(
line.rstrip()
) # Store original line for return value
locked_print(timed_line)
# Wait for process to complete
result.wait()
# Join all stdout lines for the return value
stdout = "\n".join(stdout_lines)
# replace all instances of "lib/src" => "src" so intellisense can find the files
# with one click.
stdout = stdout.replace("lib/src", "src").replace("lib\\src", "src")
if result.returncode != 0:
if not verbose_on_failure:
ERROR_HAPPENED = True
return False, stdout
if ERROR_HAPPENED:
return False, ""
ERROR_HAPPENED = True
locked_print(
f"*** Error compiling example {example} for board {board_name} ***"
)
# re-running command with verbose output to see what the defines are.
cmd_list.append("-v")
cmd_str = subprocess.list2cmdline(cmd_list)
msg_lsit = [
"\n\n******************************",
"* Re-running failed command but with verbose output:",
f"* {cmd_str}",
"******************************\n",
]
msg = "\n".join(msg_lsit)
locked_print(msg)
# Start timing for the verbose re-run
start_time_verbose = time.time()
# Run the verbose process with real-time output capture and timing
result = subprocess.Popen(
cmd_list,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT,
text=True,
)
# Capture output lines in real-time with timing for verbose run
stdout_lines_verbose: List[str] = []
if result.stdout:
for line in iter(result.stdout.readline, ""):
if line:
elapsed = time.time() - start_time_verbose
# Format timing as seconds with 2 decimal places
timing_prefix = f"{elapsed:5.2f} "
timed_line = timing_prefix + line.rstrip()
stdout_lines_verbose.append(
line.rstrip()
) # Store original line for return value
locked_print(timed_line)
# Wait for verbose process to complete
result.wait()
# Join all stdout lines for the return value
stdout = "\n".join(stdout_lines_verbose)
stdout = (
stdout
+ "\n\nThis is a second attempt, but with verbose output, look above for compiler errors.\n"
)
return False, stdout
locked_print(f"*** Finished building example {example} for board {board_name} ***")
return True, stdout
# Function to process task queues for each board
def compile_examples(
board: Board,
examples: list[Path],
build_dir: str | None,
verbose_on_failure: bool,
libs: list[str] | None,
) -> tuple[bool, str]:
"""Process the task queue for the given board."""
global ERROR_HAPPENED # pylint: disable=global-statement
board_name = board.board_name
is_first = True
for example in examples:
example = example.relative_to(Path(".").resolve())
if ERROR_HAPPENED:
return True, ""
locked_print(f"\n*** Building {example} for board {board_name} ***")
if is_first:
locked_print(
f"*** Building for first example {example} board {board_name} ***"
)
if is_first and USE_FIRST_BUILD_LOCK:
with FIRST_BUILD_LOCK:
# Github runners are memory limited and the first job is the most
# memory intensive since all the artifacts are being generated in parallel.
success, message = compile_for_board_and_example(
board=board,
example=example,
build_dir=build_dir,
verbose_on_failure=verbose_on_failure,
libs=libs,
)
else:
success, message = compile_for_board_and_example(
board=board,
example=example,
build_dir=build_dir,
verbose_on_failure=verbose_on_failure,
libs=libs,
)
is_first = False
if not success:
ERROR_HAPPENED = True
return (
False,
f"Error building {example} for board {board_name}. stdout:\n{message}",
)
return True, ""
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/compiler.py
+126
View File
@@ -0,0 +1,126 @@
#!/usr/bin/env python3
"""
Abstract Base Class for FastLED Compilers
Defines the interface that all FastLED compiler implementations must follow.
"""
from abc import ABC, abstractmethod
from concurrent.futures import Future
from dataclasses import dataclass
from enum import Enum
from pathlib import Path
from typing import Any
class CacheType(Enum):
"""Compiler cache type options."""
NO_CACHE = "no_cache"
SCCACHE = "sccache"
@dataclass
class CompilerResult:
"""Base result class for compiler operations."""
success: bool
output: str
build_dir: Path
@dataclass
class InitResult(CompilerResult):
"""Result from compiler initialization."""
@property
def platformio_ini(self) -> Path:
return self.build_dir / "platformio.ini"
@dataclass
class SketchResult(CompilerResult):
"""Result from sketch compilation."""
example: str
class Compiler(ABC):
"""Abstract base class defining the interface for FastLED compilers."""
def __init__(self) -> None:
"""Initialize the compiler."""
pass
@abstractmethod
def build(self, examples: list[str]) -> list[Future[SketchResult]]:
"""Build a list of examples with proper resource management.
Args:
examples: List of example names or paths to compile
Returns:
List of Future objects containing SketchResult for each example
"""
pass
@abstractmethod
def clean(self) -> None:
"""Clean build artifacts for this platform."""
pass
@abstractmethod
def clean_all(self) -> None:
"""Clean all build artifacts (local and global) for this platform."""
pass
@abstractmethod
def deploy(
self, example: str, upload_port: str | None = None, monitor: bool = False
) -> SketchResult:
"""Deploy (upload) a specific example to the target device.
Args:
example: Name of the example to deploy
upload_port: Optional specific port for upload
monitor: If True, attach to device monitor after successful upload
Returns:
SketchResult indicating success/failure of deployment
"""
pass
@abstractmethod
def cancel_all(self) -> None:
"""Cancel all currently running builds."""
pass
@abstractmethod
def check_usb_permissions(self) -> tuple[bool, str]:
"""Check if USB device access is properly configured.
Returns:
Tuple of (has_access, status_message)
"""
pass
@abstractmethod
def install_usb_permissions(self) -> bool:
"""Install platform-specific USB permissions or equivalent.
Returns:
True if installation succeeded, False otherwise
"""
pass
@abstractmethod
def get_cache_stats(self) -> str:
"""Get compiler statistics as a formatted string.
This can include cache statistics, build metrics, performance data,
or any other relevant compiler statistics.
Returns:
Formatted string containing compiler statistics, or empty string if none available
"""
pass
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/cpp_test_compile.py
+993
View File
@@ -0,0 +1,993 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false, reportReturnType=false, reportMissingParameterType=false
import argparse
import hashlib
import json
import logging
import os
import shutil
import subprocess
import sys
import time
from concurrent.futures import Future
from pathlib import Path
from typing import Any, Dict, List, Optional, Set, Tuple, Union
from ci.util.paths import PROJECT_ROOT
from ci.util.running_process import RunningProcess
from ci.util.test_args import parse_args as parse_global_test_args
from .clang_compiler import (
BuildFlags,
Compiler,
CompilerOptions,
LinkOptions,
test_clang_accessibility,
)
from .test_example_compilation import create_fastled_compiler
# Configure logging
logger = logging.getLogger(__name__)
BUILD_DIR = PROJECT_ROOT / "tests" / ".build"
BUILD_DIR.mkdir(parents=True, exist_ok=True)
CACHE_DIR = PROJECT_ROOT / ".cache"
CACHE_DIR.mkdir(parents=True, exist_ok=True)
TEST_FILES_LIST = CACHE_DIR / "test_files_list.txt"
# ============================================================================
# HASH-BASED LINKING CACHE (same optimization as examples)
# ============================================================================
def calculate_file_hash(file_path: Path) -> str:
"""Calculate SHA256 hash of a file (same as FastLEDTestCompiler)"""
if not file_path.exists():
return "no_file"
hash_sha256 = hashlib.sha256()
with open(file_path, "rb") as f:
for chunk in iter(lambda: f.read(4096), b""):
hash_sha256.update(chunk)
return hash_sha256.hexdigest()
def calculate_linker_args_hash(linker_args: list[str]) -> str:
"""Calculate SHA256 hash of linker arguments (same as FastLEDTestCompiler)"""
args_str = "|".join(sorted(linker_args))
hash_sha256 = hashlib.sha256()
hash_sha256.update(args_str.encode("utf-8"))
return hash_sha256.hexdigest()
def calculate_link_cache_key(
object_files: list[str | Path], fastled_lib_path: Path, linker_args: list[str]
) -> str:
"""Calculate comprehensive cache key for linking (same as examples)"""
# Calculate hash for each object file
obj_hashes: list[str] = []
for obj_file in object_files:
obj_hashes.append(calculate_file_hash(Path(obj_file)))
# Combine object file hashes in a stable way (sorted by path for consistency)
sorted_obj_paths = sorted(str(obj) for obj in object_files)
sorted_obj_hashes: list[str] = []
for obj_path in sorted_obj_paths:
obj_file = Path(obj_path)
sorted_obj_hashes.append(calculate_file_hash(obj_file))
combined_obj_hash = hashlib.sha256(
"|".join(sorted_obj_hashes).encode("utf-8")
).hexdigest()
# Calculate other hashes
fastled_hash = calculate_file_hash(fastled_lib_path)
linker_hash = calculate_linker_args_hash(linker_args)
# Combine all components (same format as FastLEDTestCompiler)
combined = f"fastled:{fastled_hash}|objects:{combined_obj_hash}|flags:{linker_hash}"
final_hash = hashlib.sha256(combined.encode("utf-8")).hexdigest()
return final_hash[:16] # Use first 16 chars for readability
def get_link_cache_dir() -> Path:
"""Get the link cache directory (same as examples)"""
cache_dir = Path(".build/link_cache")
cache_dir.mkdir(parents=True, exist_ok=True)
return cache_dir
def get_cached_executable(test_name: str, cache_key: str) -> Optional[Path]:
"""Check if cached executable exists (same as examples)"""
cache_dir = get_link_cache_dir()
cached_exe = cache_dir / f"{test_name}_{cache_key}.exe"
return cached_exe if cached_exe.exists() else None
def cache_executable(test_name: str, cache_key: str, exe_path: Path) -> None:
"""Cache an executable for future use (same as examples)"""
cache_dir = get_link_cache_dir()
cached_exe = cache_dir / f"{test_name}_{cache_key}.exe"
try:
shutil.copy2(exe_path, cached_exe)
except Exception as e:
print(f"Warning: Failed to cache {test_name}: {e}")
# ============================================================================
# END HASH-BASED LINKING CACHE
# ============================================================================
def get_test_files() -> Set[str]:
"""Get a set of all test files"""
test_files: Set[str] = set()
tests_dir = PROJECT_ROOT / "tests"
if tests_dir.exists():
for file_path in tests_dir.rglob("test_*.cpp"):
# Store relative paths for consistency
test_files.add(str(file_path.relative_to(PROJECT_ROOT)))
return test_files
def check_test_files_changed() -> bool:
"""Check if test files have changed since last run"""
try:
current_files = get_test_files()
if TEST_FILES_LIST.exists():
# Read previous file list
with open(TEST_FILES_LIST, "r") as f:
previous_files = set(line.strip() for line in f if line.strip())
# Compare file sets
if current_files == previous_files:
return False # No changes
else:
print("Test files have changed, cleaning build directory...")
return True # Files changed
else:
# No previous file list, need to clean
return True
except Exception as e:
print(f"Warning: Error checking test file changes: {e}")
return True # Default to cleaning on error
def save_test_files_list() -> None:
"""Save current test file list"""
try:
current_files = get_test_files()
with open(TEST_FILES_LIST, "w") as f:
for file_path in sorted(current_files):
f.write(f"{file_path}\n")
except KeyboardInterrupt:
raise
except Exception as e:
print(f"Warning: Failed to save test file list: {e}")
def clean_build_directory():
print("Cleaning build directory...")
shutil.rmtree(BUILD_DIR, ignore_errors=True)
BUILD_DIR.mkdir(parents=True, exist_ok=True)
print("Build directory cleaned.")
HERE = Path(__file__).resolve().parent
WASM_BUILD = False
USE_ZIG = False
USE_CLANG = False
# Legacy CMake helper functions removed - using optimized Python API
def get_unit_test_fastled_sources() -> list[Path]:
"""Get essential FastLED .cpp files for unit test library creation (optimized paradigm)."""
# Always work from project root
project_root = Path(
__file__
).parent.parent.parent # Go up from ci/compiler/ to project root
src_dir = project_root / "src"
# Core FastLED files that must be included for unit tests
core_files: list[Path] = [
src_dir / "FastLED.cpp",
src_dir / "colorutils.cpp",
src_dir / "hsv2rgb.cpp",
]
# Find all .cpp files in key directories
additional_sources: list[Path] = []
for pattern in ["*.cpp", "lib8tion/*.cpp", "platforms/stub/*.cpp"]:
additional_sources.extend(list(src_dir.glob(pattern)))
# Include essential .cpp files from nested directories
additional_sources.extend(list(src_dir.rglob("*.cpp")))
# Filter out duplicates and ensure files exist
all_sources: list[Path] = []
seen_files: set[Path] = set()
for cpp_file in core_files + additional_sources:
# Skip stub_main.cpp since unit tests have their own main
if cpp_file.name == "stub_main.cpp":
continue
# Skip platform-specific files that aren't needed for unit tests
rel_path_str = str(cpp_file)
if any(
skip in rel_path_str for skip in ["wasm", "esp", "avr", "arm", "teensy"]
):
continue
if cpp_file.exists() and cpp_file not in seen_files:
all_sources.append(cpp_file)
seen_files.add(cpp_file)
return all_sources
def create_unit_test_fastled_library(
clean: bool = False, use_pch: bool = True
) -> Path | None:
"""Create libfastled.a static library specifically for unit tests with FASTLED_FORCE_NAMESPACE=1.
CRITICAL: Unit tests need their own separate library from examples because they use
different compilation flags. Unit tests require FASTLED_FORCE_NAMESPACE=1 to put
all symbols in the fl:: namespace that the tests expect.
"""
# Unit tests get their own separate library directory under .build/fastled/unit/
fastled_build_dir = BUILD_DIR.parent / ".build" / "fastled" / "unit"
fastled_build_dir.mkdir(parents=True, exist_ok=True)
lib_file = fastled_build_dir / "libfastled.a"
if lib_file.exists() and not clean:
print(f"[LIBRARY] Using existing FastLED library: {lib_file}")
return lib_file
print("[LIBRARY] Creating FastLED static library with proper compiler flags...")
# Create a proper FastLED compiler (NOT unit test compiler) for the library
# This ensures the library is compiled without FASTLED_FORCE_NAMESPACE=1
print("[LIBRARY] Creating FastLED library compiler (without unit test flags)...")
# Save current directory and change to project root for create_fastled_compiler
import os
project_root = Path(__file__).parent.parent.parent
original_cwd = os.getcwd()
os.chdir(str(project_root))
try:
library_compiler = create_fastled_compiler(
use_pch=use_pch,
parallel=True,
)
# CRITICAL: Add required defines for unit test library compilation
if library_compiler.settings.defines is None:
library_compiler.settings.defines = []
# Add FASTLED_FORCE_NAMESPACE=1 to export symbols in fl:: namespace
library_compiler.settings.defines.append("FASTLED_FORCE_NAMESPACE=1")
print("[LIBRARY] Added FASTLED_FORCE_NAMESPACE=1 to library compiler")
# Add FASTLED_TESTING=1 to include MockTimeProvider and test utility functions
library_compiler.settings.defines.append("FASTLED_TESTING=1")
print("[LIBRARY] Added FASTLED_TESTING=1 to library compiler")
finally:
# Restore original working directory
os.chdir(original_cwd)
# Get FastLED sources using optimized selection
fastled_sources = get_unit_test_fastled_sources()
fastled_objects: list[Path] = []
obj_dir = fastled_build_dir / "obj"
obj_dir.mkdir(exist_ok=True)
print(f"[LIBRARY] Compiling {len(fastled_sources)} FastLED source files...")
# Compile each source file with optimized naming (same as examples)
futures: List[Tuple[Any, ...]] = []
project_root = Path(__file__).parent.parent.parent
src_dir = project_root / "src"
for cpp_file in fastled_sources:
# Create unique object file name by including relative path to prevent collisions
# Convert path separators to underscores to create valid filename
if cpp_file.is_relative_to(src_dir):
rel_path = cpp_file.relative_to(src_dir)
else:
rel_path = cpp_file
# Replace path separators with underscores for unique object file names
obj_name = str(rel_path.with_suffix(".o")).replace("/", "_").replace("\\", "_")
obj_file = obj_dir / obj_name
future = library_compiler.compile_cpp_file(cpp_file, obj_file)
futures.append((future, obj_file, cpp_file))
# Wait for compilation to complete
compiled_count = 0
for future, obj_file, cpp_file in futures:
try:
result = future.result()
if result.ok:
fastled_objects.append(obj_file)
compiled_count += 1
else:
print(
f"[LIBRARY] WARNING: Failed to compile {cpp_file.relative_to(src_dir)}: {result.stderr[:100]}..."
)
except Exception as e:
print(
f"[LIBRARY] WARNING: Exception compiling {cpp_file.relative_to(src_dir)}: {e}"
)
print(
f"[LIBRARY] Successfully compiled {compiled_count}/{len(fastled_sources)} FastLED sources"
)
if not fastled_objects:
print("[LIBRARY] ERROR: No FastLED source files compiled successfully")
return None
# Create static library using the same approach as examples
print(f"[LIBRARY] Creating static library: {lib_file}")
archive_future = library_compiler.create_archive(fastled_objects, lib_file)
archive_result = archive_future.result()
if not archive_result.ok:
print(f"[LIBRARY] ERROR: Library creation failed: {archive_result.stderr}")
return None
print(f"[LIBRARY] SUCCESS: FastLED library created: {lib_file}")
return lib_file
def create_unit_test_compiler(
use_pch: bool = True, enable_static_analysis: bool = False, debug: bool = False
) -> Compiler:
"""Create compiler optimized for unit test compilation with PCH support."""
# Always work from the project root, not from ci/compiler
project_root = Path(
__file__
).parent.parent.parent # Go up from ci/compiler/ to project root
current_dir = project_root
src_path = current_dir / "src"
# Load build flags configuration
build_flags_path = current_dir / "ci" / "build_unit.toml"
build_flags = BuildFlags.parse(
build_flags_path, quick_build=True, strict_mode=False
)
# Unit test specific defines
unit_test_defines = [
"FASTLED_UNIT_TEST=1",
"FASTLED_FORCE_NAMESPACE=1",
"FASTLED_USE_PROGMEM=0",
"STUB_PLATFORM",
"ARDUINO=10808",
"FASTLED_USE_STUB_ARDUINO",
"SKETCH_HAS_LOTS_OF_MEMORY=1",
"FASTLED_STUB_IMPL",
"FASTLED_USE_JSON_UI=1",
"FASTLED_TESTING",
"FASTLED_NO_AUTO_NAMESPACE",
"FASTLED_NO_PINMAP",
"HAS_HARDWARE_PIN_SUPPORT",
"FASTLED_DEBUG_LEVEL=1",
"FASTLED_NO_ATEXIT=1",
"DOCTEST_CONFIG_NO_EXCEPTIONS_BUT_WITH_ALL_ASSERTS",
"ENABLE_CRASH_HANDLER",
"RELEASE=1", # Disable FASTLED_FORCE_DBG to avoid fl::println dependency
]
# Unit test specific compiler args
unit_test_args = [
"-std=gnu++17",
"-fpermissive",
"-Wall",
"-Wextra",
"-Wno-deprecated-register",
"-Wno-backslash-newline-escape",
"-fno-exceptions",
"-fno-rtti",
# Optimization/debug controls set below based on debug flag
"-fno-omit-frame-pointer",
"-fno-inline-functions",
"-fno-vectorize",
"-fno-unroll-loops",
"-fno-strict-aliasing",
f"-I{current_dir}",
f"-I{src_path}",
f"-I{current_dir / 'tests'}",
f"-I{src_path / 'platforms' / 'stub'}",
]
# Apply quick vs debug modes
if debug:
unit_test_args.extend(
[
"-O0",
"-g3",
"-fstandalone-debug",
]
)
if os.name == "nt":
unit_test_args.extend(
["-gdwarf-4"]
) # GNU debug info for Windows GNU toolchain
else:
unit_test_args.extend(
[
"-O0",
"-g0",
]
)
# Note: DWARF flags are added above conditionally when debug=True
# PCH configuration with unit test specific headers
pch_output_path = None
pch_header_content = None
if use_pch:
cache_dir = current_dir / ".build" / "cache"
cache_dir.mkdir(parents=True, exist_ok=True)
pch_output_path = str(cache_dir / "fastled_unit_test_pch.hpp.pch")
# Unit test specific PCH header content
pch_header_content = """// FastLED Unit Test PCH - Common headers for faster test compilation
#pragma once
// Core test framework
#include "test.h"
// Core FastLED headers that are used in nearly all unit tests
#include "FastLED.h"
// Common C++ standard library headers used in tests
#include <string>
#include <vector>
#include <stdio.h>
#include <cstdint>
#include <cmath>
#include <cassert>
#include <iostream>
#include <memory>
// Platform headers for stub environment
#include "platforms/stub/fastled_stub.h"
// Commonly tested FastLED components
#include "lib8tion.h"
#include "colorutils.h"
#include "hsv2rgb.h"
#include "fl/math.h"
#include "fl/vector.h"
// Using namespace to match test files
using namespace fl;
"""
print(f"[PCH] Unit tests will use precompiled headers: {pch_output_path}")
print(
f"[PCH] PCH includes: test.h, FastLED.h, lib8tion.h, colorutils.h, and more"
)
else:
print("[PCH] Precompiled headers disabled for unit tests")
# Determine compiler
compiler_cmd = "python -m ziglang c++"
if USE_CLANG:
compiler_cmd = "clang++"
print("USING CLANG COMPILER FOR UNIT TESTS")
elif USE_ZIG:
print("USING ZIG COMPILER FOR UNIT TESTS")
else:
print("USING DEFAULT COMPILER FOR UNIT TESTS")
settings = CompilerOptions(
include_path=str(src_path),
defines=unit_test_defines,
std_version="c++17",
compiler=compiler_cmd,
compiler_args=unit_test_args,
use_pch=use_pch,
pch_output_path=pch_output_path,
pch_header_content=pch_header_content,
parallel=True,
)
return Compiler(settings, build_flags)
def compile_unit_tests_python_api(
specific_test: str | None = None,
enable_static_analysis: bool = False,
use_pch: bool = True,
clean: bool = False,
debug: bool = False,
) -> bool:
"""Compile unit tests using the fast Python API instead of CMake.
Returns:
bool: True if all tests compiled and linked successfully, False otherwise
"""
from .clang_compiler import Compiler, LinkOptions
print("=" * 60)
print("COMPILING UNIT TESTS WITH PYTHON API")
print("=" * 60)
if clean:
print("Cleaning build directory...")
shutil.rmtree(BUILD_DIR, ignore_errors=True)
BUILD_DIR.mkdir(parents=True, exist_ok=True)
# Create optimized compiler for unit tests
compiler = create_unit_test_compiler(
use_pch=use_pch, enable_static_analysis=enable_static_analysis, debug=debug
)
# Find all test files - work from project root
project_root = Path(__file__).parent.parent.parent
tests_dir = project_root / "tests"
test_files = []
if specific_test:
# Handle specific test with case-insensitive matching
test_name = (
specific_test
if specific_test.startswith("test_")
else f"test_{specific_test}"
)
test_file = tests_dir / f"{test_name}.cpp"
# First try exact case match
if test_file.exists():
test_files = [test_file]
print(f"Compiling specific test: {test_file.name}")
else:
# Try case-insensitive matching for all test files
found_match = False
for existing_file in tests_dir.glob("test_*.cpp"):
# Check if the file matches case-insensitively
existing_stem = existing_file.stem
existing_name = existing_stem.replace("test_", "")
if (
existing_stem.lower() == test_name.lower()
or existing_name.lower() == specific_test.lower()
):
test_files = [existing_file]
print(
f"Compiling specific test (case-insensitive match): {existing_file.name}"
)
found_match = True
break
if not found_match:
raise RuntimeError(f"Test file not found: {test_file}")
else:
# Find all test files
test_files = list(tests_dir.glob("test_*.cpp"))
print(f"Found {len(test_files)} unit test files")
if not test_files:
print("No test files found")
return
# Ensure output directory exists
bin_dir = BUILD_DIR / "bin"
bin_dir.mkdir(parents=True, exist_ok=True)
# Create clean execution directory as specified in requirements
clean_bin_dir = PROJECT_ROOT / "tests" / "bin"
clean_bin_dir.mkdir(parents=True, exist_ok=True)
# Step 1: Compile doctest main once
print("Compiling doctest main...")
doctest_main_path = tests_dir / "doctest_main.cpp"
doctest_main_obj = bin_dir / "doctest_main.o"
if not doctest_main_obj.exists() or clean:
doctest_compile_future = compiler.compile_cpp_file(
cpp_path=str(doctest_main_path), output_path=str(doctest_main_obj)
)
doctest_result = doctest_compile_future.result()
if doctest_result.return_code != 0:
raise RuntimeError(
f"Failed to compile doctest main: {doctest_result.stderr}"
)
# Step 2: Build FastLED library using optimized examples paradigm
print("Building FastLED library...")
fastled_lib_path = create_unit_test_fastled_library(clean, use_pch=use_pch)
# Step 3: Compile and link each test (PARALLEL OPTIMIZATION)
print(f"Compiling {len(test_files)} tests...")
start_time = time.time()
# Phase 1: Start all compilations in parallel (NON-BLOCKING)
print("🚀 Starting parallel compilation...")
compile_start = time.time()
compile_futures: Dict[str, Future[Any]] = {}
test_info: Dict[str, Any] = {}
for test_file in test_files:
test_name = test_file.stem
executable_path = bin_dir / test_name
object_path = bin_dir / f"{test_name}.o"
test_info[test_name] = {
"test_file": test_file,
"executable_path": executable_path,
"object_path": object_path,
}
print(f" Compiling {test_name}...")
# Start compilation (NON-BLOCKING)
compile_future = compiler.compile_cpp_file(
cpp_path=str(test_file), output_path=str(object_path)
)
compile_futures[test_name] = compile_future
# Phase 2: Wait for all compilations to complete and check cache before linking
compile_dispatch_time = time.time() - compile_start
print(
f"⏳ Waiting for compilations to complete... (dispatch took {compile_dispatch_time:.2f}s)"
)
compile_wait_start = time.time()
link_futures: Dict[str, Dict[str, Any]] = {}
cache_hits = 0
success_count = 0
for test_name, compile_future in compile_futures.items():
try:
compile_result = (
compile_future.result()
) # BLOCKING WAIT for this specific test
if compile_result.return_code != 0:
print(f"❌ Compilation failed for {test_name}: {compile_result.stderr}")
continue
# Prepare linking info (same logic as before)
info = test_info[test_name]
executable_path = info["executable_path"]
object_path = info["object_path"]
tests_with_own_main = ["test_example_compilation"]
if test_name in tests_with_own_main:
object_files: list[str | Path] = [object_path]
else:
object_files: list[str | Path] = [object_path, doctest_main_obj]
# Platform-specific linker arguments for crash handler support
if os.name == "nt": # Windows
linker_args = ["-ldbghelp", "-lpsapi"]
else: # Linux/macOS
linker_args = ["-pthread"]
# HASH-BASED CACHE CHECK (same as examples)
if not fastled_lib_path:
print(f"⚠️ No FastLED library found, skipping cache for {test_name}")
cache_key = "no_fastled_lib"
cached_exe = None
else:
cache_key = calculate_link_cache_key(
object_files, fastled_lib_path, linker_args
)
cached_exe = get_cached_executable(test_name, cache_key)
if cached_exe:
# Cache hit! Copy cached executable to target location
try:
shutil.copy2(cached_exe, executable_path)
# Also copy to clean execution directory
clean_executable_path = clean_bin_dir / f"{test_name}.exe"
shutil.copy2(cached_exe, clean_executable_path)
cache_hits += 1
success_count += 1
print(f" ⚡ {test_name}: Using cached executable (cache hit)")
continue # Skip linking entirely
except Exception as e:
print(f" ⚠️ Failed to copy cached {test_name}, will relink: {e}")
# Fall through to actual linking
# Cache miss - proceed with actual linking
static_libraries: List[Union[str, Path]] = []
if fastled_lib_path and fastled_lib_path.exists():
static_libraries.append(fastled_lib_path)
link_options = LinkOptions(
output_executable=str(executable_path),
object_files=object_files,
static_libraries=static_libraries,
linker_args=linker_args,
)
# Start linking (NON-BLOCKING) and store cache info for later
link_future = compiler.link_program(link_options)
link_futures[test_name] = {
"future": link_future,
"cache_key": cache_key,
"executable_path": executable_path,
}
except Exception as e:
print(f"❌ ERROR compiling {test_name}: {e}")
continue
# Phase 3: Wait for all linking to complete and cache successful results
compile_wait_time = time.time() - compile_wait_start
print(
f"🔗 Waiting for linking to complete... (compilation took {compile_wait_time:.2f}s)"
)
link_start = time.time()
cache_misses = 0
for test_name, link_info in link_futures.items():
try:
link_future = link_info["future"]
cache_key = link_info["cache_key"]
executable_path = link_info["executable_path"]
link_result = link_future.result() # BLOCKING WAIT for this specific test
if link_result.return_code != 0:
print(f"⚠️ Linking failed for {test_name}: {link_result.stderr}")
cache_misses += 1
continue
else:
success_count += 1
cache_misses += 1 # This was a fresh link, not from cache
# Cache the successful executable for future use (same as examples)
cache_executable(test_name, cache_key, executable_path)
# Also copy to clean execution directory
clean_executable_path = clean_bin_dir / f"{test_name}.exe"
shutil.copy2(executable_path, clean_executable_path)
except Exception as e:
print(f"❌ ERROR linking {test_name}: {e}")
cache_misses += 1
continue
link_time = time.time() - link_start
compilation_time = time.time() - start_time
print(f"✅ Unit test compilation completed in {compilation_time:.2f}s")
print(f" 📊 Time breakdown:")
print(f" • Dispatch: {compile_dispatch_time:.2f}s")
print(f" • Compilation: {compile_wait_time:.2f}s")
print(f" • Linking: {link_time:.2f}s")
print(f" 🎯 Cache statistics (archive + hash linking optimization):")
print(f" • Cache hits: {cache_hits} (skipped linking)")
print(f" • Cache misses: {cache_misses} (fresh linking)")
print(
f" • Cache hit ratio: {cache_hits / max(1, cache_hits + cache_misses) * 100:.1f}%"
)
print(f" Successfully compiled: {success_count}/{len(test_files)} tests")
print(f" Average: {compilation_time / len(test_files):.2f}s per test")
print(f" Output directory: {bin_dir}")
print(f" Cache directory: {get_link_cache_dir()}")
# Return success only if ALL tests compiled and linked successfully
all_tests_successful = success_count == len(test_files)
if not all_tests_successful:
failed_count = len(test_files) - success_count
print(f"❌ {failed_count} test(s) failed to compile or link")
return all_tests_successful
def parse_arguments():
parser = argparse.ArgumentParser(
description="Compile FastLED library with different compiler options."
)
parser.add_argument("--use-zig", action="store_true", help="Use Zig compiler")
parser.add_argument("--use-clang", action="store_true", help="Use Clang compiler")
parser.add_argument("--wasm", action="store_true", help="Build for WebAssembly")
parser.add_argument(
"--clean",
action="store_true",
help="Clean the build directory before compiling",
)
parser.add_argument(
"--test",
help="Specific test to compile (without test_ prefix)",
)
parser.add_argument(
"--check",
action="store_true",
help="Enable static analysis (IWYU, clang-tidy)",
)
parser.add_argument("--no-unity", action="store_true", help="Disable unity build")
parser.add_argument(
"--no-pch", action="store_true", help="Disable precompiled headers (PCH)"
)
parser.add_argument("--debug", action="store_true", help="Enable debug symbols")
parser.add_argument("--verbose", action="store_true", help="Enable verbose output")
return parser.parse_args()
def get_build_info(args: argparse.Namespace) -> dict[str, str | dict[str, str]]:
return {
"USE_ZIG": str(USE_ZIG),
"USE_CLANG": str(USE_CLANG),
"WASM_BUILD": str(WASM_BUILD),
"CC": os.environ.get("CC", ""),
"CXX": os.environ.get("CXX", ""),
"AR": os.environ.get("AR", ""),
"CFLAGS": os.environ.get("CFLAGS", ""),
"CXXFLAGS": os.environ.get("CXXFLAGS", ""),
"ARGS": {
"use_zig": str(args.use_zig),
"use_clang": str(args.use_clang),
"wasm": str(args.wasm),
"specific_test": str(args.test) if args.test else "all",
},
}
def should_clean_build(build_info: dict[str, str | dict[str, str]]) -> bool:
build_info_file = BUILD_DIR / "build_info.json"
if not build_info_file.exists():
return True
try:
with open(build_info_file, "r") as f:
old_build_info = json.load(f)
except (json.JSONDecodeError, ValueError) as e:
# If JSON is corrupted or empty, remove it and force clean build
logger.warning(
f"Corrupted build_info.json detected ({e}), removing and forcing clean build"
)
try:
build_info_file.unlink()
except OSError:
pass # Ignore if we can't remove it
return True
# If build parameters have changed, we need to rebuild
if old_build_info != build_info:
# Check if this is just a change in specific test target
old_args_raw = old_build_info.get("ARGS", {})
new_args_raw = build_info.get("ARGS", {})
# Ensure ARGS is a dictionary
if not isinstance(old_args_raw, dict) or not isinstance(new_args_raw, dict):
return True
old_args: Dict[str, Any] = old_args_raw # type: ignore
new_args: Dict[str, Any] = new_args_raw # type: ignore
# If only the specific test changed and everything else is the same,
# we don't need to clean the build directory
old_test = old_args.get("specific_test", "all")
new_test = new_args.get("specific_test", "all")
# Create copies without the specific_test field for comparison
old_args_no_test: Dict[str, Any] = {
k: v for k, v in old_args.items() if k != "specific_test"
}
new_args_no_test: Dict[str, Any] = {
k: v for k, v in new_args.items() if k != "specific_test"
}
# If only the test target changed, don't clean
if (
old_args_no_test == new_args_no_test
and old_build_info.get("USE_ZIG") == build_info.get("USE_ZIG")
and old_build_info.get("USE_CLANG") == build_info.get("USE_CLANG")
and old_build_info.get("WASM_BUILD") == build_info.get("WASM_BUILD")
and old_build_info.get("CC") == build_info.get("CC")
and old_build_info.get("CXX") == build_info.get("CXX")
and old_build_info.get("AR") == build_info.get("AR")
and old_build_info.get("CFLAGS") == build_info.get("CFLAGS")
and old_build_info.get("CXXFLAGS") == build_info.get("CXXFLAGS")
):
print(
f"Build parameters unchanged, only test target changed: {old_test} -> {new_test}"
)
return False
return True
return False
def update_build_info(build_info: dict[str, str | dict[str, str]]):
build_info_file = BUILD_DIR / "build_info.json"
with open(build_info_file, "w") as f:
json.dump(build_info, f, indent=2)
def main() -> None:
global USE_ZIG, USE_CLANG, WASM_BUILD
args = parse_arguments()
USE_ZIG = args.use_zig # use Zig's clang compiler
USE_CLANG = args.use_clang # Use pure Clang for WASM builds
WASM_BUILD = args.wasm
using_gcc = not USE_ZIG and not USE_CLANG and not WASM_BUILD
if using_gcc:
if not shutil.which("g++"):
print(
"gcc compiler not found in PATH, falling back zig's built in clang compiler"
)
USE_ZIG = True
USE_CLANG = False
if USE_CLANG:
if not test_clang_accessibility():
print(
"Clang compiler not found in PATH, falling back to Zig-clang compiler"
)
USE_ZIG = True
USE_CLANG = False
os.chdir(str(HERE))
print(f"Current directory: {Path('.').absolute()}")
# Auto-detection for --clean based on test file changes
need_clean = args.clean
if not need_clean:
# Only check for changes if --clean wasn't explicitly specified
need_clean = check_test_files_changed()
build_info = get_build_info(args)
if need_clean or should_clean_build(build_info):
clean_build_directory()
# Save the file list after cleaning
save_test_files_list()
elif args.clean:
# If --clean was explicitly specified but not needed according to build info,
# still clean and save file list
clean_build_directory()
save_test_files_list()
# Unit tests now use optimized Python API by default (same as examples)
use_pch = not getattr(
args, "no_pch", False
) # Default to PCH enabled unless --no-pch specified
# Use the optimized Python API with PCH optimization (now default for unit tests)
compilation_successful = compile_unit_tests_python_api(
specific_test=args.test,
enable_static_analysis=args.check,
use_pch=use_pch,
clean=need_clean,
debug=args.debug,
)
if compilation_successful:
update_build_info(build_info)
print("FastLED library compiled successfully.")
else:
print("❌ FastLED unit test compilation failed!")
import sys
sys.exit(1)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/cpp_test_run.py
+906
View File
@@ -0,0 +1,906 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false, reportMissingParameterType=false
import argparse
import multiprocessing
import os
import re
import subprocess
import sys
import tempfile
import time # Added for timing test execution
from dataclasses import dataclass
from pathlib import Path
from queue import Empty, PriorityQueue
from threading import Event, Lock, Thread
from typing import List
import psutil
from ci.util.paths import PROJECT_ROOT
def optimize_python_command(cmd: list[str]) -> list[str]:
"""
Optimize command list for subprocess execution in uv environment.
For python commands, we need to use 'uv run python' to ensure access to
installed packages like ziglang. Direct sys.executable bypasses uv environment.
Args:
cmd: Command list that may contain 'python' as first element
Returns:
list[str]: Optimized command with 'python' prefixed by 'uv run'
"""
if cmd and (cmd[0] == "python" or cmd[0] == "python3"):
# Use uv run python to ensure access to uv-managed packages
optimized_cmd = ["uv", "run", "python"] + cmd[1:]
return optimized_cmd
return cmd
from ci.util.test_exceptions import (
CompilationFailedException,
TestExecutionFailedException,
TestFailureInfo,
TestTimeoutException,
)
class OutputBuffer:
"""Thread-safe output buffer with ordered output display"""
def __init__(self) -> None:
self.output_queue: PriorityQueue[tuple[int, int, str]] = PriorityQueue()
self.next_sequence: int = 0
self.sequence_lock: Lock = Lock()
self.stop_event: Event = Event()
self.output_thread: Thread = Thread(target=self._output_worker, daemon=True)
self.output_thread.start()
def write(self, test_index: int, message: str) -> None:
"""Write a message to the buffer with test index for ordering"""
with self.sequence_lock:
sequence = self.next_sequence
self.next_sequence += 1
self.output_queue.put((test_index, sequence, message))
def _output_worker(self) -> None:
"""Worker thread that processes output in order"""
while not self.stop_event.is_set() or not self.output_queue.empty():
try:
item: tuple[int, int, str] = self.output_queue.get(timeout=0.1)
_, _, message = item
print(message, flush=True)
self.output_queue.task_done()
except Empty:
continue
except Exception as e:
print(f"Error in output worker: {e}")
continue
def stop(self) -> None:
"""Stop the output worker thread"""
self.stop_event.set()
if self.output_thread.is_alive():
self.output_thread.join()
# Configure console for UTF-8 output on Windows
if os.name == "nt": # Windows
# Try to set console to UTF-8 mode
try:
# Set stdout and stderr to UTF-8 encoding
# Note: reconfigure() was added in Python 3.7
if hasattr(sys.stdout, "reconfigure") and callable(
getattr(sys.stdout, "reconfigure", None)
):
sys.stdout.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
if hasattr(sys.stderr, "reconfigure") and callable(
getattr(sys.stderr, "reconfigure", None)
):
sys.stderr.reconfigure(encoding="utf-8", errors="replace") # type: ignore[attr-defined]
except (AttributeError, OSError):
# Fallback for older Python versions or if reconfigure fails
pass
# Environment flags for backward compatibility
_SHOW_COMPILE = os.environ.get("FASTLED_TEST_SHOW_COMPILE", "").lower() in (
"1",
"true",
"yes",
)
_SHOW_LINK = os.environ.get("FASTLED_TEST_SHOW_LINK", "").lower() in (
"1",
"true",
"yes",
)
@dataclass
class FailedTest:
name: str
return_code: int
stdout: str
def check_iwyu_available() -> bool:
"""Check if include-what-you-use is available in the system"""
try:
result = subprocess.run(
["include-what-you-use", "--version"],
capture_output=True,
text=True,
timeout=10,
)
return result.returncode == 0
except (
subprocess.CalledProcessError,
FileNotFoundError,
subprocess.TimeoutExpired,
):
return False
def run_command(
command: str | list[str],
use_gdb: bool = False,
*,
verbose: bool = False,
show_compile: bool = False,
show_link: bool = False,
) -> tuple[int, str]:
captured_lines: list[str] = []
# Determine command type
is_test_execution = False
is_compile = False
is_link = False
if isinstance(command, str):
cmd_lower = command.replace("\\", "/").lower()
# Check if running test executable
is_test_execution = (
"/test_" in cmd_lower
or ".build/bin/test_" in cmd_lower
or cmd_lower.endswith(".exe")
)
# Check if compiling
is_compile = "-c" in cmd_lower and (".cpp" in cmd_lower or ".c" in cmd_lower)
# Check if linking
is_link = (
not is_compile
and not is_test_execution
and ("-o" in cmd_lower or "lib" in cmd_lower)
)
if use_gdb:
with tempfile.NamedTemporaryFile(mode="w+", delete=False) as gdb_script:
gdb_script.write("set pagination off\n")
gdb_script.write("run\n")
gdb_script.write("bt full\n")
gdb_script.write("info registers\n")
gdb_script.write("x/16i $pc\n")
gdb_script.write("thread apply all bt full\n")
gdb_script.write("quit\n")
gdb_command = (
f"gdb -return-child-result -batch -x {gdb_script.name} --args {command}"
)
process = subprocess.Popen(
gdb_command,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, # Merge stderr into stdout
shell=True,
text=False,
)
assert process.stdout is not None
# Stream and capture output
while True:
line_bytes = process.stdout.readline()
line = line_bytes.decode("utf-8", errors="ignore")
if not line and process.poll() is not None:
break
if line:
captured_lines.append(line.rstrip())
# Always print GDB output (it's only used for crashes anyway)
try:
print(line, end="", flush=True)
except UnicodeEncodeError:
# Fallback: replace problematic characters
print(
line.encode("utf-8", errors="replace").decode(
"utf-8", errors="replace"
),
end="",
flush=True,
)
os.unlink(gdb_script.name)
output = "\n".join(captured_lines)
return process.returncode, output
else:
# Optimize list commands to avoid shell overhead
if isinstance(command, list):
# Optimize python commands and use shell=False for better performance
python_exe = optimize_python_command(command)
process = subprocess.Popen(
python_exe,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, # Merge stderr into stdout
shell=False, # Use shell=False for better performance with list commands
text=False,
)
else:
# String commands still need shell=True
process = subprocess.Popen(
command,
stdout=subprocess.PIPE,
stderr=subprocess.STDOUT, # Merge stderr into stdout
shell=True,
text=False,
)
assert process.stdout is not None
# Stream and capture output
while True:
line_bytes = process.stdout.readline()
line = line_bytes.decode("utf-8", errors="ignore")
if not line and process.poll() is not None:
break
if line:
captured_lines.append(line.rstrip())
# Determine if we should print this line
should_print = (
verbose # Always print in verbose mode
or (is_compile and show_compile) # Print compilation if enabled
or (is_link and show_link) # Print linking if enabled
or (not is_test_execution) # Print non-test output
or (
is_test_execution and process.returncode != 0
) # Print failed test output
or (
is_test_execution
and any(
marker in line
for marker in [
"Running test:",
"Test passed",
"Test FAILED",
"passed with return code",
"Test output:",
]
)
) # Print test status
)
if should_print:
try:
# Add prefix for compile/link commands
if is_compile and show_compile:
print("[COMPILE] ", end="", flush=True)
elif is_link and show_link:
print("[LINK] ", end="", flush=True)
print(line, end="", flush=True)
except UnicodeEncodeError:
# Fallback: replace problematic characters
print(
line.encode("utf-8", errors="replace").decode(
"utf-8", errors="replace"
),
end="",
flush=True,
)
output = "\n".join(captured_lines)
return process.returncode, output
def compile_tests(
clean: bool = False,
unknown_args: list[str] = [],
specific_test: str | None = None,
quick_build: bool = True,
*,
verbose: bool = False,
show_compile: bool = False,
show_link: bool = False,
) -> None:
"""
Compile C++ tests using the Python build system.
"""
os.chdir(str(PROJECT_ROOT))
print("🔧 Compiling tests using Python build system")
_compile_tests_python(
clean,
unknown_args,
specific_test,
quick_build=quick_build,
verbose=verbose,
show_compile=show_compile,
show_link=show_link,
)
def _compile_tests_python(
clean: bool = False,
unknown_args: list[str] = [],
specific_test: str | None = None,
quick_build: bool = True,
*,
verbose: bool = False,
show_compile: bool = False,
show_link: bool = False,
) -> None:
"""Python build system with PCH optimization"""
# Use the optimized cpp_test_compile system directly
import subprocess
cmd = ["uv", "run", "python", "-m", "ci.compiler.cpp_test_compile"]
if specific_test:
cmd.extend(["--test", specific_test])
if clean:
cmd.append("--clean")
if verbose:
cmd.append("--verbose")
if "--check" in unknown_args:
cmd.append("--check")
if "--no-pch" in unknown_args:
cmd.append("--no-pch")
# Forward debug mode to the compiler when quick_build is disabled
if not quick_build:
cmd.append("--debug")
print("🚀 Using Python build system with PCH optimization")
result = subprocess.run(cmd)
if result.returncode != 0:
raise RuntimeError(
f"Unit test compilation failed with return code {result.returncode}"
)
def run_tests(
specific_test: str | None = None,
*,
verbose: bool = False,
show_compile: bool = False,
show_link: bool = False,
) -> None:
"""
Run compiled tests with GDB crash analysis support.
"""
_run_tests_python(specific_test)
def _run_tests_python(
specific_test: str | None = None,
*,
verbose: bool = False,
show_compile: bool = False,
show_link: bool = False,
) -> None:
"""Run tests from Python build system"""
# Import the test compiler system
from ci.compiler.test_compiler import FastLEDTestCompiler
# Get test executables from Python build system
test_compiler = FastLEDTestCompiler.get_existing_instance()
if not test_compiler:
print("No compiled tests found. Run compilation first.")
sys.exit(1)
test_executables = test_compiler.get_test_executables(specific_test)
if not test_executables:
test_name = specific_test or "any tests"
print(f"No test executables found for: {test_name}")
sys.exit(1)
print(f"Running {len(test_executables)} tests from Python build...")
# Print list of tests that will be executed
print("Tests to execute:")
for i, test_exec in enumerate(test_executables, 1):
# Convert absolute path to relative for display
rel_path = os.path.relpath(test_exec.executable_path)
print(f" {i}. {test_exec.name} ({rel_path})")
print("")
failed_tests: list[FailedTest] = []
# Convert to file list format for compatibility with existing logic
files: list[str] = []
test_paths: dict[str, str] = {}
for test_exec in test_executables:
file_name = test_exec.name
if os.name == "nt" and not file_name.endswith(".exe"):
file_name += ".exe"
files.append(file_name)
test_paths[file_name] = str(test_exec.executable_path)
print(f"Starting test execution for {len(files)} test files...")
_execute_test_files(files, "", failed_tests, specific_test, test_paths)
_handle_test_results(failed_tests)
def _execute_test_files(
files: list[str],
test_dir: str,
failed_tests: list[FailedTest],
specific_test: str | None,
test_paths: dict[str, str] | None = None,
*,
verbose: bool = False,
show_compile: bool = False,
show_link: bool = False,
) -> None:
"""
Execute test files in parallel with full GDB crash analysis.
Args:
files: List of test file names
test_dir: Directory containing tests (for CMake) or empty string (for Python API)
failed_tests: List to collect failed tests
specific_test: Specific test name if filtering
test_paths: Dict mapping file names to full paths (for Python API)
"""
total_tests = len(files)
successful_tests = 0
completed_tests = 0
# Initialize output buffer for ordered output
output_buffer = OutputBuffer()
output_buffer.write(0, f"Executing {total_tests} test files in parallel...")
# Determine number of workers based on configuration
# Get configuration from args
args = parse_args()
# Force sequential execution if NO_PARALLEL is set
if os.environ.get("NO_PARALLEL"):
max_workers = 1
output_buffer.write(
0, "NO_PARALLEL environment variable set - forcing sequential execution"
)
elif args.sequential:
max_workers = 1
elif args.parallel:
max_workers = args.parallel
else:
max_workers = max(1, multiprocessing.cpu_count() - 1) # Leave one core free
# Check memory limit
if args.max_memory:
memory_limit = args.max_memory * 1024 * 1024 # Convert MB to bytes
available_memory = psutil.virtual_memory().available
if memory_limit > available_memory:
output_buffer.write(
0,
f"Warning: Requested memory limit {args.max_memory}MB exceeds available memory {available_memory / (1024 * 1024):.0f}MB",
)
output_buffer.write(
0, "Reducing number of parallel workers to stay within memory limits"
)
# Estimate memory per test based on previous runs or default to 100MB
memory_per_test = 100 * 1024 * 1024 # 100MB per test
max_parallel_by_memory = max(1, memory_limit // memory_per_test)
max_workers = min(max_workers, max_parallel_by_memory)
output_buffer.write(0, f"Using {max_workers} parallel workers")
# Thread-safe counter
counter_lock = Lock()
def run_single_test(
test_file: str, test_index: int
) -> tuple[bool, float, str, int]:
"""Run a single test and return its results"""
nonlocal completed_tests
if test_paths:
test_path = test_paths[test_file]
else:
test_path = os.path.join(test_dir, test_file)
# For .cpp files, compile them first
if test_path.endswith(".cpp"):
# Create a temporary directory for compilation
with tempfile.TemporaryDirectory() as temp_dir:
# Compile the test file
output_buffer.write(
test_index,
f"[{test_index}/{total_tests}] Compiling test: {test_file}",
)
compile_cmd = [
"python",
"-m",
"ziglang",
"c++",
"-o",
os.path.join(temp_dir, "test.exe"),
test_path,
"-I",
os.path.join(PROJECT_ROOT, "src"),
"-I",
os.path.join(PROJECT_ROOT, "tests"),
# NOTE: Compiler flags now come from build configuration TOML
]
return_code, stdout = run_command(compile_cmd)
if return_code != 0:
output_buffer.write(
test_index,
f"[{test_index}/{total_tests}] ERROR: Failed to compile test: {test_file}",
)
return False, 0.0, f"Failed to compile test: {stdout}", return_code
# Update test_path to point to the compiled executable
test_path = os.path.join(temp_dir, "test.exe")
if not (os.path.isfile(test_path) and os.access(test_path, os.X_OK)):
output_buffer.write(
test_index,
f"[{test_index}/{total_tests}] ERROR: Test file not found or not executable: {test_path}",
)
return False, 0.0, f"Test file not found or not executable: {test_path}", 1
output_buffer.write(
test_index, f"[{test_index}/{total_tests}] Running test: {test_file}"
)
if verbose:
output_buffer.write(test_index, f" Command: {test_path}")
start_time = time.time()
# Pass --minimal flag to doctest when not in verbose mode to suppress output unless tests fail
cmd = [test_path]
if not verbose:
cmd.append("--minimal")
return_code, stdout = run_command(cmd)
elapsed_time = time.time() - start_time
output = stdout
failure_pattern = re.compile(r"Test .+ failed with return code (\d+)")
failure_match = failure_pattern.search(output)
is_crash = failure_match is not None
# Handle crashes with GDB (must be done synchronously)
if is_crash:
output_buffer.write(
test_index, f"Test crashed. Re-running with GDB to get stack trace..."
)
_, gdb_stdout = run_command(test_path, use_gdb=True)
stdout += "\n--- GDB Output ---\n" + gdb_stdout
# Extract crash information
crash_info = extract_crash_info(gdb_stdout)
output_buffer.write(
test_index, f"Crash occurred at: {crash_info.file}:{crash_info.line}"
)
output_buffer.write(test_index, f"Cause: {crash_info.cause}")
output_buffer.write(test_index, f"Stack: {crash_info.stack}")
# Print output based on verbosity and status
if verbose or return_code != 0:
output_buffer.write(test_index, "Test output:")
output_buffer.write(test_index, stdout)
if return_code == 0:
output_buffer.write(
test_index, f" Test {test_file} passed in {elapsed_time:.2f}s"
)
else:
output_buffer.write(
test_index,
f" Test {test_file} FAILED with return code {return_code} in {elapsed_time:.2f}s",
)
with counter_lock:
completed_tests += 1
return return_code == 0, elapsed_time, stdout, return_code
try:
# Run tests in parallel using ThreadPoolExecutor
from concurrent.futures import ThreadPoolExecutor, as_completed
with ThreadPoolExecutor(max_workers=max_workers) as executor:
# Submit all tests
future_to_test = {
executor.submit(run_single_test, test_file, i + 1): test_file
for i, test_file in enumerate(files)
}
# Process results as they complete
for future in as_completed(future_to_test):
test_file = future_to_test[future]
try:
success, _, stdout, return_code = future.result()
if success:
with counter_lock:
successful_tests += 1
else:
failed_tests.append(
FailedTest(
name=test_file, return_code=return_code, stdout=stdout
)
)
except Exception as e:
output_buffer.write(
0, f"ERROR: Test {test_file} failed with exception: {e}"
)
failed_tests.append(
FailedTest(name=test_file, return_code=1, stdout=str(e))
)
# Print final summary
output_buffer.write(
0,
f"Test execution complete: {successful_tests} passed, {len(failed_tests)} failed",
)
if successful_tests == total_tests:
output_buffer.write(0, "All tests passed successfully!")
else:
output_buffer.write(
0, f"Some tests failed ({len(failed_tests)} of {total_tests})"
)
finally:
# Ensure output buffer is stopped
output_buffer.stop()
def _handle_test_results(
failed_tests: list[FailedTest], *, verbose: bool = False
) -> None:
"""Handle test results and exit appropriately (preserving existing logic)"""
if failed_tests:
print("Failed tests summary:")
failures: List[TestFailureInfo] = []
for failed_test in failed_tests:
print(
f"Test {failed_test.name} failed with return code {failed_test.return_code}"
)
# Always show output on failure
print("Output:")
# Show indented output for better readability
for line in failed_test.stdout.splitlines():
print(f" {line}")
print() # Add spacing between failed tests
failures.append(
TestFailureInfo(
test_name=failed_test.name,
command=f"test_{failed_test.name}",
return_code=failed_test.return_code,
output=failed_test.stdout,
error_type="test_execution_failure",
)
)
tests_failed = len(failed_tests)
failed_test_names = [test.name for test in failed_tests]
print(
f"{tests_failed} test{'s' if tests_failed != 1 else ''} failed: {', '.join(failed_test_names)}"
)
raise TestExecutionFailedException(f"{tests_failed} test(s) failed", failures)
if verbose:
print("All tests passed.")
@dataclass
class CrashInfo:
cause: str = "Unknown"
stack: str = "Unknown"
file: str = "Unknown"
line: str = "Unknown"
def extract_crash_info(gdb_output: str) -> CrashInfo:
lines = gdb_output.split("\n")
crash_info = CrashInfo()
try:
for i, line in enumerate(lines):
if line.startswith("Program received signal"):
try:
crash_info.cause = line.split(":", 1)[1].strip()
except IndexError:
crash_info.cause = line.strip()
elif line.startswith("#0"):
crash_info.stack = line
for j in range(i, len(lines)):
if "at" in lines[j]:
try:
_, location = lines[j].split("at", 1)
location = location.strip()
if ":" in location:
crash_info.file, crash_info.line = location.rsplit(
":", 1
)
else:
crash_info.file = location
except ValueError:
pass # If split fails, we keep the default values
break
break
except Exception as e:
print(f"Error parsing GDB output: {e}")
return crash_info
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Compile and run C++ tests")
parser.add_argument(
"--compile-only",
action="store_true",
help="Only compile the tests without running them",
)
parser.add_argument(
"--run-only",
action="store_true",
help="Only run the tests without compiling them",
)
parser.add_argument(
"--only-run-failed-test",
action="store_true",
help="Only run the tests that failed in the previous run",
)
parser.add_argument(
"--clean", action="store_true", help="Clean build before compiling"
)
parser.add_argument(
"--test",
help="Specific test to run (without extension)",
)
parser.add_argument(
"--verbose",
action="store_true",
help="Enable verbose output",
)
parser.add_argument(
"--show-compile",
action="store_true",
help="Show compilation commands and output",
)
parser.add_argument(
"--show-link",
action="store_true",
help="Show linking commands and output",
)
parser.add_argument(
"--parallel",
type=int,
help="Number of parallel test processes to run (default: CPU count - 1)",
)
parser.add_argument(
"--sequential",
action="store_true",
help="Run tests sequentially (disables parallel execution)",
)
parser.add_argument(
"--max-memory",
type=int,
help="Maximum memory usage in MB for parallel test execution",
)
# Create mutually exclusive group for compiler selection
compiler_group = parser.add_mutually_exclusive_group()
compiler_group.add_argument(
"--clang",
help="Use Clang compiler",
action="store_true",
)
compiler_group.add_argument(
"--gcc",
help="Use GCC compiler (default on non-Windows)",
action="store_true",
)
parser.add_argument(
"--check",
action="store_true",
help="Enable static analysis (IWYU, clang-tidy)",
)
parser.add_argument(
"--no-unity",
action="store_true",
help="Disable unity builds for cpp tests",
)
parser.add_argument(
"--no-pch",
action="store_true",
help="Disable precompiled headers (PCH) for unit tests",
)
parser.add_argument(
"--debug",
action="store_true",
help="Use debug build mode with full debug symbols (default is quick mode with -g0)",
)
args, unknown = parser.parse_known_args()
args.unknown = unknown
return args
def main() -> None:
try:
args = parse_args()
# Get verbosity flags from args
run_only = args.run_only
compile_only = args.compile_only
specific_test = args.test
# only_run_failed_test feature to be implemented in future
_ = args.only_run_failed_test
use_clang = args.clang
no_unity = args.no_unity
quick_build = (
not args.debug
) # Default to quick mode unless --debug is specified
# use_gcc = args.gcc
if not run_only:
passthrough_args = args.unknown
if use_clang:
passthrough_args.append("--use-clang")
if args.check:
passthrough_args.append("--check")
if no_unity:
passthrough_args.append("--no-unity")
if args.no_pch:
passthrough_args.append("--no-pch")
# Note: --gcc is handled by not passing --use-clang (GCC is the default in compiler/cpp_test_compile.py)
compile_tests(
clean=args.clean,
unknown_args=passthrough_args,
specific_test=specific_test,
quick_build=quick_build,
verbose=args.verbose,
show_compile=args.show_compile,
show_link=args.show_link,
)
if not compile_only:
if specific_test:
run_tests(
specific_test,
verbose=args.verbose,
show_compile=args.show_compile,
show_link=args.show_link,
)
else:
# Use our own test runner instead of CTest since CTest integration is broken
run_tests(
None,
verbose=args.verbose,
show_compile=args.show_compile,
show_link=args.show_link,
)
except (
CompilationFailedException,
TestExecutionFailedException,
TestTimeoutException,
) as e:
# Print detailed failure information
print("\n" + "=" * 60)
print("FASTLED TEST FAILURE DETAILS")
print("=" * 60)
print(e.get_detailed_failure_info())
print("=" * 60)
# Exit with appropriate code
if e.failures:
# Use the return code from the first failure, or 1 if none available
exit_code = (
e.failures[0].return_code if e.failures[0].return_code != 0 else 1
)
else:
exit_code = 1
sys.exit(exit_code)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/packages.py
+784
View File
@@ -0,0 +1,784 @@
"""
Enhanced Arduino Package Index Implementation with Pydantic
This module provides a robust, production-ready Arduino package management system
that fully complies with the Arduino CLI Package Index JSON Specification.
Key Features:
- Pydantic models with comprehensive validation
- Multi-source package index support
- Caching and persistence
- Package installation and dependency resolution
- Search and filtering capabilities
- Checksum validation and error handling
"""
import asyncio
import hashlib
import json
import shutil
import sys
import tarfile
import zipfile
from datetime import datetime
from pathlib import Path
from typing import Any, Callable, Dict, List, Optional, Set, Union
from urllib.parse import urlparse
from urllib.request import urlopen
import httpx
from pydantic import (
BaseModel,
EmailStr,
Field,
HttpUrl,
ValidationError,
field_validator,
model_validator,
)
# Custom Exceptions
class PackageParsingError(Exception):
"""Error parsing package index data"""
pass
class PackageInstallationError(Exception):
"""Error during package installation"""
pass
class PackageValidationError(Exception):
"""Error validating package data"""
pass
# Core Pydantic Models with Enhanced Validation
class Help(BaseModel):
"""Help information with online resources"""
online: HttpUrl = Field(description="URL to online help resources")
class Config:
schema_extra = {
"example": {"online": "https://github.com/espressif/arduino-esp32"}
}
class Board(BaseModel):
"""Board information with complete properties"""
name: str = Field(min_length=1, max_length=200, description="Board display name")
properties: Dict[str, Any] = Field(
default_factory=dict, description="Board configuration properties"
)
@field_validator("name")
@classmethod
def validate_board_name(cls, v: str) -> str:
"""Validate board name format"""
if not v.strip():
raise ValueError("Board name cannot be empty or whitespace")
return v.strip()
class Config:
schema_extra = {
"example": {
"name": "ESP32 Dev Module",
"properties": {
"upload.tool": "esptool_py",
"upload.maximum_size": "1310720",
"build.mcu": "esp32",
},
}
}
class ToolDependency(BaseModel):
"""Tool dependency specification with validation"""
packager: str = Field(
min_length=1, max_length=100, description="Tool packager name"
)
name: str = Field(pattern=r"^[a-zA-Z0-9_.-]+$", description="Tool name")
version: str = Field(min_length=1, description="Required tool version")
@field_validator("version")
@classmethod
def validate_version_format(cls, v: str) -> str:
"""Validate version format - supports semantic versioning and Arduino versioning"""
if not v.strip():
raise ValueError("Version cannot be empty")
# Allow flexible versioning (semantic, date-based, etc.)
return v.strip()
class Config:
schema_extra = {
"example": {
"packager": "esp32",
"name": "xtensa-esp32-elf-gcc",
"version": "esp-2021r2-patch5-8.4.0",
}
}
class Platform(BaseModel):
"""Platform specification with comprehensive validation"""
name: str = Field(min_length=1, max_length=200, description="Platform display name")
architecture: str = Field(
pattern=r"^[a-zA-Z0-9_-]+$", description="Target architecture"
)
version: str = Field(description="Platform version")
category: str = Field(min_length=1, description="Platform category")
url: HttpUrl = Field(description="Download URL for platform archive")
archive_filename: str = Field(
alias="archiveFileName", description="Archive file name"
)
checksum: str = Field(
pattern=r"^SHA-256:[a-fA-F0-9]{64}$", description="SHA-256 checksum"
)
size_mb: float = Field(gt=0, alias="size", description="Archive size in megabytes")
boards: List[Board] = Field(
default_factory=lambda: [], description="Supported boards"
)
tool_dependencies: List[ToolDependency] = Field(
default_factory=lambda: [],
alias="toolsDependencies",
description="Required tool dependencies",
)
help: Help = Field(description="Help and documentation links")
@field_validator("version")
@classmethod
def validate_version_format(cls, v: str) -> str:
"""Validate version format"""
if not v.strip():
raise ValueError("Version cannot be empty")
return v.strip()
@field_validator("size_mb", mode="before")
@classmethod
def convert_size_from_bytes(cls, v: Union[str, int, float]) -> float:
"""Convert size from bytes to megabytes if needed"""
if isinstance(v, str):
try:
size_bytes = int(v)
return size_bytes / (1024 * 1024)
except ValueError:
raise ValueError(f"Invalid size format: {v}")
elif isinstance(v, (int, float)):
if v > 1024 * 1024: # Assume bytes if > 1MB
return v / (1024 * 1024)
return v # Already in MB
return v
@field_validator("archive_filename")
@classmethod
def validate_archive_filename(cls, v: str) -> str:
"""Validate archive filename format"""
valid_extensions = [".zip", ".tar.gz", ".tar.bz2", ".tar.xz", ".tar.zst"]
if not any(v.lower().endswith(ext) for ext in valid_extensions):
raise ValueError(
f"Archive must have one of these extensions: {valid_extensions}"
)
return v
class Config:
allow_population_by_field_name = True
schema_extra: Dict[str, Any] = {
"example": {
"name": "ESP32 Arduino",
"architecture": "esp32",
"version": "2.0.5",
"category": "ESP32",
"url": "https://github.com/espressif/arduino-esp32/releases/download/2.0.5/esp32-2.0.5.zip",
"archiveFileName": "esp32-2.0.5.zip",
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
"size": "50000000",
"boards": [],
"toolsDependencies": [],
}
}
class SystemDownload(BaseModel):
"""System-specific download information for tools"""
host: str = Field(min_length=1, description="Target host system identifier")
url: HttpUrl = Field(description="Download URL for this system")
archive_filename: str = Field(
alias="archiveFileName", description="Archive file name"
)
checksum: str = Field(
pattern=r"^SHA-256:[a-fA-F0-9]{64}$", description="SHA-256 checksum"
)
size_mb: float = Field(gt=0, alias="size", description="Archive size in megabytes")
@field_validator("size_mb", mode="before")
@classmethod
def convert_size_from_bytes(cls, v: Union[str, int, float]) -> float:
"""Convert size from bytes to megabytes if needed"""
if isinstance(v, str):
try:
size_bytes = int(v)
return size_bytes / (1024 * 1024)
except ValueError:
raise ValueError(f"Invalid size format: {v}")
elif isinstance(v, (int, float)):
if v > 1024 * 1024: # Assume bytes if > 1MB
return v / (1024 * 1024)
return v # Already in MB
return v
@field_validator("host")
@classmethod
def validate_host_format(cls, v: str) -> str:
"""Validate host system identifier"""
# Common host patterns: i686-pc-linux-gnu, x86_64-apple-darwin, etc.
if not v.strip():
raise ValueError("Host identifier cannot be empty")
return v.strip()
class Config:
allow_population_by_field_name = True
schema_extra = {
"example": {
"host": "x86_64-pc-linux-gnu",
"url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
"archiveFileName": "xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
"checksum": "SHA-256:abcdef1234567890abcdef1234567890abcdef1234567890abcdef1234567890",
"size": "150000000",
}
}
class Tool(BaseModel):
"""Tool with system-specific downloads and enhanced functionality"""
name: str = Field(pattern=r"^[a-zA-Z0-9_.-]+$", description="Tool name")
version: str = Field(min_length=1, description="Tool version")
systems: List[SystemDownload] = Field(
min_length=1, description="System-specific downloads"
)
@field_validator("systems")
@classmethod
def validate_unique_systems(cls, v: List[SystemDownload]) -> List[SystemDownload]:
"""Ensure no duplicate host systems"""
hosts = [system.host for system in v]
if len(hosts) != len(set(hosts)):
raise ValueError("Duplicate host systems found in tool downloads")
return v
def get_system_download(self, host_pattern: str) -> Optional[SystemDownload]:
"""Get system download matching host pattern"""
for system in self.systems:
if host_pattern in system.host:
return system
return None
def get_compatible_systems(self) -> List[str]:
"""Get list of compatible host systems"""
return [system.host for system in self.systems]
class Config:
schema_extra = {
"example": {
"name": "xtensa-esp32-elf-gcc",
"version": "esp-2021r2-patch5-8.4.0",
"systems": [
{
"host": "x86_64-pc-linux-gnu",
"url": "https://github.com/espressif/crosstool-NG/releases/download/esp-2021r2-patch5/xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
"archiveFileName": "xtensa-esp32-elf-gcc8_4_0-esp-2021r2-patch5-linux-amd64.tar.gz",
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
"size": "150000000",
}
],
}
}
class Package(BaseModel):
"""Package containing platforms and tools with comprehensive validation"""
name: str = Field(pattern=r"^[A-Za-z0-9_.-]+$", description="Package identifier")
maintainer: str = Field(
min_length=1, max_length=200, description="Package maintainer"
)
website_url: HttpUrl = Field(alias="websiteURL", description="Package website URL")
email: EmailStr = Field(description="Maintainer contact email")
help: Help = Field(description="Help and documentation links")
platforms: List[Platform] = Field(
default_factory=lambda: [], description="Available platforms"
)
tools: List[Tool] = Field(default_factory=lambda: [], description="Available tools")
@field_validator("platforms")
@classmethod
def validate_unique_platforms(cls, v: List[Platform]) -> List[Platform]:
"""Ensure no duplicate platform architecture/version combinations"""
seen: Set[tuple[str, str]] = set()
for platform in v:
key = (platform.architecture, platform.version)
if key in seen:
raise ValueError(
f"Duplicate platform found: {platform.architecture} v{platform.version}"
)
seen.add(key)
return v
@field_validator("tools")
@classmethod
def validate_unique_tools(cls, v: List[Tool]) -> List[Tool]:
"""Ensure no duplicate tool name/version combinations"""
seen: Set[tuple[str, str]] = set()
for tool in v:
key = (tool.name, tool.version)
if key in seen:
raise ValueError(f"Duplicate tool found: {tool.name} v{tool.version}")
seen.add(key)
return v
def find_platform(
self, architecture: str, version: Optional[str] = None
) -> Optional[Platform]:
"""Find platform by architecture and optionally version"""
for platform in self.platforms:
if platform.architecture == architecture:
if version is None or platform.version == version:
return platform
return None
def find_tool(self, name: str, version: Optional[str] = None) -> Optional[Tool]:
"""Find tool by name and optionally version"""
for tool in self.tools:
if tool.name == name:
if version is None or tool.version == version:
return tool
return None
def get_latest_platform_version(self, architecture: str) -> Optional[str]:
"""Get the latest version for a given architecture"""
versions = [p.version for p in self.platforms if p.architecture == architecture]
if not versions:
return None
# Simple version sorting - can be enhanced with proper semver parsing
return sorted(versions)[-1]
class Config:
allow_population_by_field_name = True
schema_extra: Dict[str, Any] = {
"example": {
"name": "esp32",
"maintainer": "Espressif Systems",
"websiteURL": "https://github.com/espressif/arduino-esp32",
"email": "hr@espressif.com",
"help": {"online": "https://github.com/espressif/arduino-esp32"},
"platforms": [],
"tools": [],
}
}
class PackageIndex(BaseModel):
"""Root package index containing multiple packages"""
packages: List[Package] = Field(min_length=1, description="Available packages")
@field_validator("packages")
@classmethod
def validate_unique_packages(cls, v: List[Package]) -> List[Package]:
"""Ensure no duplicate package names"""
names = [pkg.name for pkg in v]
if len(names) != len(set(names)):
raise ValueError("Duplicate package names found in index")
return v
def find_package(self, name: str) -> Optional[Package]:
"""Find package by name"""
for package in self.packages:
if package.name == name:
return package
return None
def get_all_platforms(self) -> List[Platform]:
"""Get all platforms from all packages"""
platforms: List[Platform] = []
for package in self.packages:
platforms.extend(package.platforms)
return platforms
def get_all_tools(self) -> List[Tool]:
"""Get all tools from all packages"""
tools: List[Tool] = []
for package in self.packages:
tools.extend(package.tools)
return tools
# Enhanced Parser with Validation and Error Handling
class PackageIndexParser:
"""Enhanced parser with comprehensive validation and error handling"""
def __init__(self, timeout: int = 30):
"""Initialize parser with timeout configuration"""
self.timeout = timeout
def parse_package_index(self, json_str: str) -> PackageIndex:
"""Parse and validate package index JSON"""
try:
raw_data = json.loads(json_str)
return PackageIndex(**raw_data)
except ValidationError as e:
raise PackageParsingError(f"Invalid package index format: {e}")
except json.JSONDecodeError as e:
raise PackageParsingError(f"Invalid JSON format: {e}")
def parse_from_url(self, url: str) -> PackageIndex:
"""Fetch and parse package index from URL with validation"""
try:
print(f"Fetching package index from: {url}")
with urlopen(url, timeout=self.timeout) as response:
content = response.read()
json_str = content.decode("utf-8")
return self.parse_package_index(json_str)
except Exception as e:
raise PackageParsingError(f"Error fetching package index from {url}: {e}")
async def parse_from_url_async(self, url: str) -> PackageIndex:
"""Async version of URL parsing"""
try:
print(f"Fetching package index from: {url}")
async with httpx.AsyncClient(timeout=self.timeout) as client:
response = await client.get(url)
response.raise_for_status()
json_str = response.text
return self.parse_package_index(json_str)
except Exception as e:
raise PackageParsingError(f"Error fetching package index from {url}: {e}")
# Package Manager Configuration
class PackageManagerConfig(BaseModel):
"""Configuration for package manager with validation"""
cache_dir: Path = Field(default_factory=lambda: Path.home() / ".arduino_packages")
sources: List[HttpUrl] = Field(
default_factory=lambda: [], description="Package index URLs"
)
timeout: int = Field(
default=30, gt=0, le=300, description="Request timeout in seconds"
)
max_retries: int = Field(
default=3, ge=0, le=10, description="Maximum retry attempts"
)
verify_checksums: bool = Field(
default=True, description="Verify download checksums"
)
allow_insecure: bool = Field(default=False, description="Allow insecure downloads")
@field_validator("cache_dir")
@classmethod
def validate_cache_dir(cls, v: Path) -> Path:
"""Ensure cache directory is valid"""
if v.exists() and not v.is_dir():
raise ValueError(f"Cache path exists but is not a directory: {v}")
return v
class Config:
validate_assignment = True
schema_extra = {
"example": {
"cache_dir": "~/.arduino_packages",
"sources": [
"https://espressif.github.io/arduino-esp32/package_esp32_index.json"
],
"timeout": 30,
"max_retries": 3,
"verify_checksums": True,
"allow_insecure": False,
}
}
# Utility Functions for Enhanced Functionality
def verify_checksum(file_path: Path, expected_checksum: str) -> bool:
"""Verify file checksum against expected SHA-256 value"""
if not expected_checksum.startswith("SHA-256:"):
raise ValueError(f"Invalid checksum format: {expected_checksum}")
expected_hash = expected_checksum[8:] # Remove 'SHA-256:' prefix
sha256_hash = hashlib.sha256()
with open(file_path, "rb") as f:
for chunk in iter(lambda: f.read(4096), b""):
sha256_hash.update(chunk)
actual_hash = sha256_hash.hexdigest()
return actual_hash.lower() == expected_hash.lower()
def extract_archive(archive_path: Path, extract_to: Path) -> bool:
"""Extract archive to specified directory with format detection"""
try:
archive_str = str(archive_path)
if archive_path.suffix == ".zip":
with zipfile.ZipFile(archive_path, "r") as zip_ref:
zip_ref.extractall(extract_to)
elif archive_str.endswith((".tar.gz", ".tgz")):
with tarfile.open(archive_path, "r:gz") as tar_ref:
tar_ref.extractall(extract_to)
elif archive_str.endswith(".tar.bz2"):
with tarfile.open(archive_path, "r:bz2") as tar_ref:
tar_ref.extractall(extract_to)
elif archive_str.endswith(".tar.xz"):
with tarfile.open(archive_path, "r:xz") as tar_ref:
tar_ref.extractall(extract_to)
else:
raise ValueError(f"Unsupported archive format: {archive_path}")
return True
except Exception as e:
print(f"Error extracting archive {archive_path}: {e}")
return False
def format_size(size_mb: float) -> str:
"""Format size in a human-readable way"""
if size_mb < 1:
return f"{size_mb * 1024:.1f} KB"
elif size_mb < 1024:
return f"{size_mb:.1f} MB"
else:
return f"{size_mb / 1024:.1f} GB"
# Display Functions with Enhanced Formatting
def display_package_info(package: Package) -> None:
"""Display package information with enhanced formatting"""
print(f"\n📦 Package: {package.name}")
print(f"👤 Maintainer: {package.maintainer}")
print(f"🌐 Website: {package.website_url}")
print(f"📧 Email: {package.email}")
print(f"📚 Help: {package.help.online}")
print(f"🛠️ Platforms: {len(package.platforms)}")
print(f"🔧 Tools: {len(package.tools)}")
# Show platform information
for i, platform in enumerate(package.platforms[:3]): # Show first 3 platforms
print(f"\n 📋 Platform {i + 1}: {platform.name} v{platform.version}")
print(f" Architecture: {platform.architecture}")
print(f" Category: {platform.category}")
print(f" Size: {format_size(platform.size_mb)}")
print(f" Archive: {platform.archive_filename}")
print(f" Checksum: {platform.checksum[:24]}...")
print(f" Help: {platform.help.online}")
print(f" Boards: {len(platform.boards)}")
# Show first 5 boards
for board in platform.boards[:5]:
print(f" • {board.name}")
if len(platform.boards) > 5:
print(f" ... and {len(platform.boards) - 5} more")
print(f" Tool Dependencies: {len(platform.tool_dependencies)}")
for dep in platform.tool_dependencies[:3]: # Show first 3 dependencies
print(f" • {dep.name} v{dep.version} ({dep.packager})")
if len(platform.tool_dependencies) > 3:
print(f" ... and {len(platform.tool_dependencies) - 3} more")
if len(package.platforms) > 3:
print(f"\n ... and {len(package.platforms) - 3} more platforms")
def display_validation_summary(package_index: PackageIndex) -> None:
"""Display validation summary for the package index"""
print(f"\n✅ VALIDATION SUMMARY")
print(f" 📦 Total packages: {len(package_index.packages)}")
total_platforms = sum(len(pkg.platforms) for pkg in package_index.packages)
total_tools = sum(len(pkg.tools) for pkg in package_index.packages)
total_boards = sum(
len(platform.boards)
for pkg in package_index.packages
for platform in pkg.platforms
)
print(f" 🛠️ Total platforms: {total_platforms}")
print(f" 🔧 Total tools: {total_tools}")
print(f" 💾 Total boards: {total_boards}")
# Show architectures
architectures: Set[str] = set()
for pkg in package_index.packages:
for platform in pkg.platforms:
architectures.add(platform.architecture)
print(f" 🏗️ Architectures: {', '.join(sorted(architectures))}")
# Demonstration Functions
def demo_esp32_parsing() -> Optional[PackageIndex]:
"""Demonstrate parsing ESP32 package index with enhanced validation"""
ESP32_URL = "https://espressif.github.io/arduino-esp32/package_esp32_index.json"
try:
parser = PackageIndexParser(timeout=30)
package_index = parser.parse_from_url(ESP32_URL)
print("🎉 Successfully parsed ESP32 package index with Pydantic validation!")
display_validation_summary(package_index)
# Display first package
if package_index.packages:
display_package_info(package_index.packages[0])
return package_index
except PackageParsingError as e:
print(f"❌ Package parsing error: {e}")
sys.exit(1)
except Exception as e:
print(f"❌ Unexpected error: {e}")
sys.exit(1)
def demo_model_validation() -> None:
"""Demonstrate Pydantic model validation capabilities"""
print("\n🧪 TESTING PYDANTIC MODEL VALIDATION")
# Define valid platform data
valid_platform_data: Dict[str, Any] = {
"name": "ESP32 Arduino",
"architecture": "esp32",
"version": "2.0.5",
"category": "ESP32",
"url": "https://github.com/espressif/arduino-esp32/releases/download/2.0.5/esp32-2.0.5.zip",
"archiveFileName": "esp32-2.0.5.zip",
"checksum": "SHA-256:1234567890abcdef1234567890abcdef1234567890abcdef1234567890abcdef",
"size": "50000000", # Should convert to MB
"boards": [],
"toolsDependencies": [],
"help": {"online": "https://github.com/espressif/arduino-esp32"},
}
# Test valid platform
try:
platform = Platform(**valid_platform_data)
print(f"✅ Valid platform created: {platform.name} v{platform.version}")
print(f" Size converted: {platform.size_mb:.1f} MB")
except ValidationError as e:
print(f"❌ Unexpected validation error: {e}")
# Test invalid platform
try:
invalid_platform_data = valid_platform_data.copy()
invalid_platform_data["checksum"] = "invalid-checksum-format"
platform = Platform(**invalid_platform_data)
print("❌ Should have failed validation!")
except ValidationError as e:
print(f"✅ Correctly caught invalid checksum: {e.errors()[0]['msg']}")
# Test invalid URL
try:
invalid_url_data = valid_platform_data.copy()
invalid_url_data["url"] = "not-a-valid-url"
platform = Platform(**invalid_url_data)
print("❌ Should have failed URL validation!")
except ValidationError as e:
print(f"✅ Correctly caught invalid URL: {e.errors()[0]['msg']}")
def main() -> None:
"""Main function demonstrating enhanced package index functionality"""
print("🚀 ENHANCED ARDUINO PACKAGE INDEX WITH PYDANTIC")
print("=" * 60)
try:
# Demo model validation
demo_model_validation()
# Demo ESP32 parsing
package_index = demo_esp32_parsing()
# Interactive options
try:
print(f"\n📋 AVAILABLE OPTIONS:")
print("1. Show detailed tools information")
print("2. Search for specific architecture")
print("3. Exit")
choice = input("\nEnter your choice (1-3): ").strip()
if choice == "1" and package_index and package_index.packages:
# Simple tools info display (without importing original)
pkg = package_index.packages[0]
print(f"\n🔧 TOOLS INFORMATION for {pkg.name}")
print(f"Total tools: {len(pkg.tools)}")
for tool in pkg.tools[:3]: # Show first 3 tools
print(
f" • {tool.name} v{tool.version} ({len(tool.systems)} systems)"
)
if len(pkg.tools) > 3:
print(f" ... and {len(pkg.tools) - 3} more tools")
elif choice == "2" and package_index:
arch = input("Enter architecture to search for: ").strip()
platforms = [
p
for pkg in package_index.packages
for p in pkg.platforms
if p.architecture == arch
]
if platforms:
print(
f"\n🔍 Found {len(platforms)} platforms for architecture '{arch}':"
)
for platform in platforms:
print(
f" • {platform.name} v{platform.version} ({format_size(platform.size_mb)})"
)
else:
print(f"❌ No platforms found for architecture '{arch}'")
else:
print("👋 Goodbye!")
except KeyboardInterrupt:
print("\n👋 Interrupted by user")
except KeyboardInterrupt:
print("\n👋 Interrupted by user")
sys.exit(1)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/pio.py
+1950
View File
File diff suppressed because it is too large. Load diff
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/platformio_cache.py
+937
View File
@@ -0,0 +1,937 @@
#!/usr/bin/env python3
"""
PlatformIO artifact cache implementation for speeding up CI builds.
This module implements the cache mechanism described in FEATURE_PIO_SPEEDUP.md,
providing functionality to:
1. Parse platformio.ini files with zip URLs
2. Download and cache platform/framework artifacts
3. Install artifacts via PlatformIO CLI
4. Modify platformio.ini in-place with resolved local paths
"""
import _thread
import configparser
import json
import logging
import os
import shutil
import subprocess
import tempfile
import threading
import time
import urllib.parse
import zipfile
from concurrent.futures import Future, ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
from types import TracebackType
from typing import Any, Dict, List, Optional, Tuple, Union, cast
import fasteners
import httpx
from ci.compiler.platformio_ini import PlatformIOIni
from ci.util.running_process import RunningProcess
from ci.util.url_utils import sanitize_url_for_path
@dataclass
class DownloadResult:
"""Result of a download operation."""
url: str
temp_path: Path
exception: Optional[BaseException] = None
@property
def success(self) -> bool:
"""True if download succeeded."""
return self.exception is None
@dataclass
class ArtifactProcessingResult:
"""Result of processing an artifact."""
url: str
is_framework: bool
env_section: str
resolved_path: Optional[str] = None
exception: Optional[BaseException] = None
@property
def success(self) -> bool:
"""True if processing succeeded."""
return self.exception is None and self.resolved_path is not None
@dataclass
class ZipUrlInfo:
"""Information about a zip URL found in platformio.ini."""
url: str
section_name: str
option_name: str
@dataclass
class ManifestResult:
"""Result of manifest validation and type detection."""
is_valid: bool
manifest_path: Optional[Path]
is_framework: bool
def _get_remote_file_size(url: str) -> Optional[int]:
"""Get file size from URL using HEAD request."""
try:
parsed_url = urllib.parse.urlparse(url)
if parsed_url.scheme in ("http", "https"):
with httpx.Client(follow_redirects=True) as client:
response = client.head(
url, headers={"User-Agent": "PlatformIO-Cache/1.0"}
)
response.raise_for_status()
content_length = response.headers.get("Content-Length")
if content_length:
return int(content_length)
except Exception as e:
logger.debug(f"Failed to get file size for {url}: {e}")
return None
def _format_file_size(size_bytes: Optional[int]) -> str:
"""Format file size in human-readable format."""
if size_bytes is None:
return "unknown size"
if size_bytes < 1024:
return f"{size_bytes} B"
elif size_bytes < 1024 * 1024:
return f"{size_bytes / 1024:.1f} KB"
elif size_bytes < 1024 * 1024 * 1024:
return f"{size_bytes / (1024 * 1024):.1f} MB"
else:
return f"{size_bytes / (1024 * 1024 * 1024):.1f} GB"
# Global cache to track PlatformIO installations in this session
_session_installation_cache: set[str] = set()
# Global cancellation event for handling keyboard interrupts
_global_cancel_event = threading.Event()
# Lock for thread-safe operations
_download_lock = threading.Lock()
def _get_status_file(artifact_dir: Path, cache_key: str) -> Path:
"""Get the JSON status file path for an artifact."""
# Use simple descriptive filename since artifacts are already in unique directories
return artifact_dir / "info.json"
def _read_status(status_file: Path) -> Optional[Dict[str, Any]]:
"""Read status from JSON file."""
if not status_file.exists():
return None
try:
with open(status_file, "r") as f:
return json.load(f)
except (json.JSONDecodeError, OSError):
return None
def _write_status(status_file: Path, status: Dict[str, Any]) -> None:
"""Write status to JSON file."""
try:
with open(status_file, "w") as f:
json.dump(status, f, indent=2)
except OSError as e:
logger.warning(f"Failed to write status file {status_file}: {e}")
def _is_processing_complete(status_file: Path) -> bool:
"""Check if processing is complete based on status file."""
status = _read_status(status_file)
return status is not None and status.get("status") == "complete"
def _download_with_progress(
url: str, temp_path: Path, cancel_event: threading.Event
) -> DownloadResult:
"""Download HTTP/HTTPS file with cancellation support."""
try:
# HTTP/HTTPS download using httpx with streaming
with httpx.Client(follow_redirects=True, timeout=30.0) as client:
with client.stream(
"GET", url, headers={"User-Agent": "PlatformIO-Cache/1.0"}
) as response:
response.raise_for_status()
with open(temp_path, "wb") as f:
for chunk in response.iter_bytes(chunk_size=8192):
# Check for cancellation periodically
if cancel_event.is_set():
cancelled_error = RuntimeError(
f"Download cancelled for {url}"
)
logger.warning(
f"Download cancelled for {url}",
exc_info=cancelled_error,
)
return DownloadResult(url, temp_path, cancelled_error)
f.write(chunk)
return DownloadResult(url, temp_path) # Success
except KeyboardInterrupt as e:
# Set cancel event and interrupt main thread
cancel_event.set()
_thread.interrupt_main()
logger.warning(
f"Download interrupted by KeyboardInterrupt for {url}", exc_info=e
)
return DownloadResult(url, temp_path, e)
except Exception as e:
if not cancel_event.is_set():
logger.error(f"Download failed for {url}: {e}", exc_info=e)
else:
logger.warning(f"Download failed for {url} (cancelled): {e}", exc_info=e)
return DownloadResult(url, temp_path, e)
def _copy_file_with_progress(
url: str, temp_path: Path, cancel_event: threading.Event
) -> DownloadResult:
"""Copy local file with cancellation support."""
try:
parsed_url = urllib.parse.urlparse(url)
# File URL - copy local file
logger.debug(f"Parsing file URL: {url}")
logger.debug(f"Parsed URL path: {parsed_url.path}")
# Handle both Unix and Windows file URLs
if os.name == "nt": # Windows
# On Windows, file:///C:/path becomes /C:/path, so remove leading slash
if (
parsed_url.path.startswith("/")
and len(parsed_url.path) > 3
and parsed_url.path[2] == ":"
):
source_path = Path(parsed_url.path[1:])
else:
source_path = Path(parsed_url.path)
else: # Unix-like
source_path = Path(parsed_url.path)
logger.debug(f"Resolved file path: {source_path}")
if not source_path.exists():
raise FileNotFoundError(f"Source file not found: {source_path}")
# Check for cancellation before copy
if cancel_event.is_set():
cancelled_error = RuntimeError(f"Copy cancelled for {url}")
logger.warning(f"Copy cancelled for {url}", exc_info=cancelled_error)
return DownloadResult(url, temp_path, cancelled_error)
shutil.copy2(source_path, temp_path)
return DownloadResult(url, temp_path) # Success
except KeyboardInterrupt as e:
# Set cancel event and interrupt main thread
cancel_event.set()
_thread.interrupt_main()
logger.warning(f"Copy interrupted by KeyboardInterrupt for {url}", exc_info=e)
return DownloadResult(url, temp_path, e)
except Exception as e:
if not cancel_event.is_set():
logger.error(f"Copy failed for {url}: {e}", exc_info=e)
else:
logger.warning(f"Copy failed for {url} (cancelled): {e}", exc_info=e)
return DownloadResult(url, temp_path, e)
def clear_session_cache() -> None:
"""Clear the session installation cache. Useful for testing."""
global _session_installation_cache
_session_installation_cache.clear()
logger.debug("Cleared session installation cache")
# Configure logging
logging.basicConfig(level=logging.INFO)
logger = logging.getLogger(__name__)
# Reduce httpx verbosity to avoid log spew from HTTP requests
logging.getLogger("httpx").setLevel(logging.WARNING)
class PlatformIOCache:
"""Enhanced cache manager for PlatformIO artifacts."""
def __init__(self, cache_dir: Path):
"""Initialize cache manager with directory structure."""
self.cache_dir = cache_dir
# Simplified structure: each artifact gets its own directory directly in cache root
# containing both the .zip and extracted/ folder
# Create directory structure
self.cache_dir.mkdir(parents=True, exist_ok=True)
def _get_cache_key(self, url: str) -> str:
"""Generate cache key from URL - sanitized for filesystem use."""
return str(sanitize_url_for_path(url))
def download_artifact(self, url: str) -> str:
"""
Download and cache an artifact from the given URL.
Returns the absolute path to the cached zip file.
"""
cache_key = self._get_cache_key(url)
# Each artifact gets its own directory
artifact_dir = self.cache_dir / cache_key
artifact_dir.mkdir(parents=True, exist_ok=True)
cached_path = artifact_dir / "artifact.zip"
# Use read-write locking for concurrent safety
# Start with write lock for downloading, can upgrade to read if cache hit
lock_path = str(artifact_dir / "artifact.lock")
rw_lock = fasteners.InterProcessReaderWriterLock(lock_path)
with rw_lock.write_lock():
if cached_path.exists():
# Check if processing was completed successfully
status_file = _get_status_file(artifact_dir, cache_key)
if not _is_processing_complete(status_file):
logger.warning(
f"Cache incomplete (no completion status), re-downloading: {url}"
)
# Remove incomplete zip file
cached_path.unlink()
# Also remove any partial extraction
extracted_dir = artifact_dir / "extracted"
if extracted_dir.exists():
shutil.rmtree(extracted_dir)
else:
print(f"Using cached artifact: {cached_path}")
# Cache hit - return the cached path (write lock will be released)
return str(cached_path)
# Check URL scheme to determine action
parsed_url = urllib.parse.urlparse(url)
if parsed_url.scheme in ("http", "https"):
# Download to temporary file first (atomic operation)
file_size = _get_remote_file_size(url)
size_str = _format_file_size(file_size)
print(f"Downloading: {url} ({size_str})")
temp_file_handle = tempfile.NamedTemporaryFile(
delete=False, suffix=".zip"
)
temp_path = Path(temp_file_handle.name)
temp_file_handle.close() # Close immediately to avoid Windows file locking issues
# Create a thread-local cancel event
thread_cancel_event = threading.Event()
# Download directly (we're already in a thread from the main pool)
download_result = _download_with_progress(
url, temp_path, thread_cancel_event
)
elif parsed_url.scheme == "file":
# File URL - copy from local file
print(f"Copying from local file: {url}")
temp_file_handle = tempfile.NamedTemporaryFile(
delete=False, suffix=".zip"
)
temp_path = Path(temp_file_handle.name)
temp_file_handle.close() # Close immediately to avoid Windows file locking issues
# Create a thread-local cancel event
thread_cancel_event = threading.Event()
# Copy directly (we're already in a thread from the main pool)
download_result = _copy_file_with_progress(
url, temp_path, thread_cancel_event
)
else:
raise ValueError(f"Unsupported URL scheme: {parsed_url.scheme}")
try:
# Check if global cancellation was requested
if _global_cancel_event.is_set():
raise KeyboardInterrupt(
"Download cancelled due to global interrupt"
)
# Process the download result
if not download_result.success:
if download_result.exception is not None:
raise download_result.exception
else:
raise RuntimeError(
f"Download failed for unknown reason: {download_result.url}"
)
else:
print(f"Download completed successfully: {download_result.url}")
# Atomic move to final location
shutil.move(str(temp_path), str(cached_path))
print(f"Successfully cached: {cached_path}")
return str(cached_path)
except Exception as e:
logger.error(f"Download failed: {e}")
if temp_path.exists():
temp_path.unlink()
raise
def _is_zip_web_url(value: str) -> bool:
"""Enhanced URL detection for zip artifacts."""
if not isinstance(value, str):
return False
parsed = urllib.parse.urlparse(value)
# Direct zip URLs
if value.endswith(".zip"):
return parsed.scheme in ("http", "https")
return False
def validate_and_detect_manifest(
content_path: Path,
) -> ManifestResult:
"""
Validate manifest files and auto-detect artifact type.
"""
# Try framework manifests first
framework_manifests = ["framework.json", "package.json"]
for manifest in framework_manifests:
manifest_path = content_path / manifest
if manifest_path.exists():
try:
with open(manifest_path, "r") as f:
json.load(f) # Validate JSON syntax
print(f"Found valid framework manifest: {manifest_path}")
return ManifestResult(
is_valid=True, manifest_path=manifest_path, is_framework=True
)
except json.JSONDecodeError as e:
logger.warning(f"Invalid JSON in {manifest_path}: {e}")
# Try platform manifest
platform_manifest = content_path / "platform.json"
if platform_manifest.exists():
try:
with open(platform_manifest, "r") as f:
json.load(f) # Validate JSON syntax
print(f"Found valid platform manifest: {platform_manifest}")
return ManifestResult(
is_valid=True, manifest_path=platform_manifest, is_framework=False
)
except json.JSONDecodeError as e:
logger.warning(f"Invalid JSON in {platform_manifest}: {e}")
return ManifestResult(is_valid=False, manifest_path=None, is_framework=False)
def get_platformio_command_path(path: Path) -> str:
"""Get path format for PlatformIO command line usage."""
import platform
resolved_path = path.resolve()
# On Windows, PlatformIO has issues with file:// URLs for local directories
# Use native Windows paths instead
if platform.system() == "Windows":
return str(resolved_path)
else:
# On Unix systems, use proper file:// URLs
posix_path = resolved_path.as_posix()
return f"file://{posix_path}"
def get_proper_file_url(path: Path) -> str:
"""Convert a path to a proper file:// URL for platformio.ini files."""
# For PlatformIO on Windows, use the Windows path directly (not file:// URL)
# This avoids issues with path parsing in PlatformIO
import platform
resolved_path = path.resolve()
if platform.system() == "Windows":
# On Windows, PlatformIO handles Windows paths directly better than file:// URLs
# Convert to Windows path format
return str(resolved_path)
else:
# On Unix systems, use proper file:// URL format
posix_path = resolved_path.as_posix()
return f"file://{posix_path}"
def unzip_and_install(
cached_zip_path: Path,
cache_manager: PlatformIOCache,
is_framework: bool,
env_section: str,
) -> bool:
"""
Enhanced unzip and install with manifest validation and cleanup.
"""
cached_zip_path_obj = cached_zip_path
# Extract to a directory alongside the zip file
artifact_dir = cached_zip_path_obj.parent
extracted_dir = artifact_dir / "extracted"
temp_unzip_dir = artifact_dir / "temp_extract"
try:
# Clean extraction directory
if temp_unzip_dir.exists():
shutil.rmtree(temp_unzip_dir)
temp_unzip_dir.mkdir(parents=True)
print(f"Extracting {cached_zip_path_obj} to {temp_unzip_dir}")
# Extract with better error handling
try:
with zipfile.ZipFile(cached_zip_path_obj, "r") as zip_ref:
# Check for zip bombs (basic protection)
total_size = sum(info.file_size for info in zip_ref.infolist())
if total_size > 500 * 1024 * 1024: # 500MB limit
raise ValueError("Archive too large, possible zip bomb")
zip_ref.extractall(temp_unzip_dir)
print("Extraction completed successfully")
except zipfile.BadZipFile:
logger.error(f"Invalid zip file: {cached_zip_path_obj}")
return False
except Exception as e:
logger.error(f"Extraction failed: {e}")
return False
# Find content directory (handle nested structures like GitHub archives)
content_items = list(temp_unzip_dir.iterdir())
if len(content_items) == 1 and content_items[0].is_dir():
# Single root directory (common with GitHub/GitLab archives) - use it
unzipped_content_path = content_items[0]
print(f"Found nested directory structure: {content_items[0].name}")
else:
# Multiple items at root - use temp dir
unzipped_content_path = temp_unzip_dir
print("Using flat directory structure")
# Validate manifest files and auto-detect type
manifest_result = validate_and_detect_manifest(unzipped_content_path)
if not manifest_result.is_valid:
logger.error(f"No valid manifest found in {unzipped_content_path}")
return False
# Move to final location
if extracted_dir.exists():
shutil.rmtree(extracted_dir)
shutil.move(unzipped_content_path, extracted_dir)
# Install via PlatformIO
return install_with_platformio(
extracted_dir, manifest_result.is_framework, env_section
)
finally:
# Cleanup temporary extraction
if temp_unzip_dir.exists():
print(f"Cleaning up extraction directory: {temp_unzip_dir}")
shutil.rmtree(temp_unzip_dir)
def install_with_platformio(
content_path: Path, is_framework: bool, env_section: str
) -> bool:
"""Install extracted content using appropriate PlatformIO command."""
content_path_obj = Path(content_path)
command_path = get_platformio_command_path(content_path_obj)
# Create a cache key based on the installation type and path
cache_key = f"{'framework' if is_framework else 'platform'}:{command_path}"
# Check if we've already installed this in this session
if cache_key in _session_installation_cache:
print(
f"Skipping installation for {env_section}: already installed in this session ({command_path})"
)
return True
if is_framework:
# Framework installation - use pkg install
command = ["pio", "pkg", "install", "--global", command_path]
else:
# Platform installation - also use pkg install (new recommended way)
command = ["pio", "pkg", "install", "--global", "--platform", command_path]
try:
print(f"Installing for {env_section}: {' '.join(command)}")
# Use RunningProcess for streaming output
process = RunningProcess(
command,
check=False, # We'll handle errors ourselves
timeout=300, # 5 minute timeout
)
# Stream output in real-time
for line in process.line_iter(timeout=300):
line_str = cast(str, line)
print(f"PIO: {line_str}")
# Wait for completion and check result
process.wait()
if process.returncode == 0:
print("PlatformIO installation successful")
# Add to session cache to avoid redundant installations
_session_installation_cache.add(cache_key)
return True
else:
logger.error(
f"PlatformIO installation failed with return code: {process.returncode}"
)
return False
except Exception as e:
if "No such file or directory" in str(e) or "not found" in str(e).lower():
logger.error("PlatformIO CLI not found. Is it installed and in PATH?")
else:
logger.error(f"PlatformIO installation failed: {e}")
logger.error(f"Command: {' '.join(command)}")
return False
def handle_zip_artifact(
zip_source: str,
cache_manager: PlatformIOCache,
env_section: str,
) -> str:
"""
Enhanced artifact handler with validation and error recovery.
Returns the resolved local path for the artifact.
"""
try:
# Download and cache the artifact
cached_zip_path = cache_manager.download_artifact(zip_source)
if not cached_zip_path or not Path(cached_zip_path).exists():
raise FileNotFoundError(f"Download failed for {zip_source}")
# Extract and get the content path
cache_key = cache_manager._get_cache_key(zip_source)
cached_zip_path_obj = Path(cached_zip_path)
# Extract to a directory alongside the zip file
artifact_dir = cached_zip_path_obj.parent
extracted_dir = artifact_dir / "extracted"
temp_unzip_dir = artifact_dir / "temp_extract"
# Check if already extracted
if extracted_dir.exists():
# Validate existing extraction and auto-detect type
manifest_result = validate_and_detect_manifest(extracted_dir)
if manifest_result.is_valid:
print(f"Using existing extraction: {extracted_dir}")
# Check for completion status using URL-based cache key (consistent with status file creation)
url_cache_key = cache_manager._get_cache_key(zip_source)
status_file = _get_status_file(artifact_dir, url_cache_key)
if _is_processing_complete(status_file):
print(
f"Skipping PlatformIO installation for {env_section}: found completion status"
)
return get_proper_file_url(extracted_dir)
# Create a session cache key for this artifact
session_cache_key = f"{'framework' if manifest_result.is_framework else 'platform'}:{get_platformio_command_path(extracted_dir)}"
# If we've already handled this exact artifact in this session, skip PlatformIO entirely
if session_cache_key in _session_installation_cache:
print(
f"Skipping PlatformIO installation for {env_section}: already processed in this session"
)
return get_proper_file_url(extracted_dir)
# Otherwise, install via PlatformIO
install_success = install_with_platformio(
extracted_dir, manifest_result.is_framework, env_section
)
# Status file will be created later using URL-based cache key
if install_success:
print(f"Successfully installed {zip_source} for {env_section}")
else:
logger.warning(
f"Installation completed with warnings for {zip_source}"
)
return get_proper_file_url(extracted_dir)
# Clean extraction directory for fresh extraction
if temp_unzip_dir.exists():
shutil.rmtree(temp_unzip_dir)
temp_unzip_dir.mkdir(parents=True)
# Extract the zip
with zipfile.ZipFile(cached_zip_path, "r") as zip_ref:
zip_ref.extractall(temp_unzip_dir)
# Find content directory (handle nested structures)
content_items = list(temp_unzip_dir.iterdir())
if len(content_items) == 1 and content_items[0].is_dir():
unzipped_content_path = content_items[0]
else:
unzipped_content_path = temp_unzip_dir
# Validate manifest files and auto-detect type
manifest_result = validate_and_detect_manifest(unzipped_content_path)
if not manifest_result.is_valid:
raise ValueError(f"No valid manifest found in {unzipped_content_path}")
# Move to final location
if extracted_dir.exists():
shutil.rmtree(extracted_dir)
shutil.move(unzipped_content_path, extracted_dir)
# Clean up temp directory
if temp_unzip_dir.exists():
shutil.rmtree(temp_unzip_dir)
# Install via PlatformIO
install_success = install_with_platformio(
extracted_dir, manifest_result.is_framework, env_section
)
# Create status file with processing results
cache_key = cache_manager._get_cache_key(zip_source)
status_file = _get_status_file(artifact_dir, cache_key)
# Get zip file size
zip_file = artifact_dir / "artifact.zip"
zip_size = zip_file.stat().st_size if zip_file.exists() else 0
status = {
"status": "complete" if install_success else "warning",
"timestamp": datetime.now().isoformat(),
"url": zip_source,
"env_section": env_section,
"extracted_dir": str(extracted_dir.relative_to(artifact_dir)),
"zip_size_bytes": zip_size,
}
_write_status(status_file, status)
if install_success:
print(f"Successfully installed {zip_source} for {env_section}")
else:
logger.warning(f"Installation completed with warnings for {zip_source}")
# Return the file URL for the extracted directory
return get_proper_file_url(extracted_dir)
except Exception as e:
import traceback
traceback.print_exc()
logger.error(f"Failed to handle artifact {zip_source}: {e}")
raise
def _process_artifact(
artifact_url: str,
env_section: str,
cache_manager: "PlatformIOCache",
) -> ArtifactProcessingResult:
"""Process a single artifact (download, extract, install)."""
try:
resolved_path = handle_zip_artifact(artifact_url, cache_manager, env_section)
return ArtifactProcessingResult(
url=artifact_url,
is_framework=False, # This will be determined during processing
env_section=env_section,
resolved_path=resolved_path,
)
except Exception as e:
logger.error(f"Failed to process {artifact_url}: {e}", exc_info=e)
return ArtifactProcessingResult(
url=artifact_url,
is_framework=False, # This will be determined during processing
env_section=env_section,
exception=e,
)
def _collect_all_zip_urls(pio_ini: PlatformIOIni) -> List[ZipUrlInfo]:
"""
Collect all zip URLs from platformio.ini in a single pass.
"""
all_urls: List[ZipUrlInfo] = []
# Collect platform URLs
for section_name, option_name, url in pio_ini.get_platform_urls():
if _is_zip_web_url(url):
print(f"Found platform zip: {url} in {section_name}")
all_urls.append(ZipUrlInfo(url, section_name, option_name))
# Collect framework URLs
for section_name, option_name, url in pio_ini.get_framework_urls():
if _is_zip_web_url(url):
print(f"Found framework zip: {url} in {section_name}")
all_urls.append(ZipUrlInfo(url, section_name, option_name))
return all_urls
def _dedupe_urls(
all_urls: List[ZipUrlInfo],
) -> Dict[str, str]:
"""
Deduplicate URLs, keeping the first occurrence of each unique URL.
Returns dict mapping url -> env_section.
"""
unique_urls: Dict[str, str] = {}
for url_info in all_urls:
if url_info.url not in unique_urls:
unique_urls[url_info.url] = url_info.section_name
return unique_urls
def _download_and_process_urls(
unique_urls: Dict[str, str], cache_manager: PlatformIOCache
) -> Dict[str, str]:
"""
Download and process all URLs concurrently.
Returns dict mapping original_url -> resolved_local_path.
Raises exception immediately if any download fails.
"""
if not unique_urls:
return {}
max_workers = min(4, len(unique_urls)) # Don't create more threads than needed
with ThreadPoolExecutor(
max_workers=max_workers, thread_name_prefix="download"
) as executor:
# Submit all download tasks
futures: List[Future[ArtifactProcessingResult]] = []
for url, env_section in unique_urls.items():
future = executor.submit(_process_artifact, url, env_section, cache_manager)
futures.append(future)
replacements: Dict[str, str] = {}
try:
for future in as_completed(futures):
try:
result = future.result()
if result.success and result.resolved_path is not None:
replacements[result.url] = result.resolved_path
print(f"✅ Resolved {result.url} -> {result.resolved_path}")
else:
logger.error(
f"❌ Failed to process {result.url}: {result.exception}"
)
# Re-raise all exceptions - no downloads should fail silently
if result.exception:
raise result.exception
except Exception as e:
logger.error(
f"Future failed with unexpected error: {e}", exc_info=e
)
raise # Re-raise unexpected exceptions
except KeyboardInterrupt:
logger.warning("Processing interrupted, cancelling remaining downloads...")
_global_cancel_event.set()
# The context manager will handle cleanup of the executor
raise
return replacements
def _replace_all_urls(
pio_ini: PlatformIOIni,
all_urls: List[ZipUrlInfo],
replacements: Dict[str, str],
) -> None:
"""
Replace all URLs in platformio.ini with their resolved local paths.
"""
for url_info in all_urls:
if url_info.url in replacements:
pio_ini.replace_url(
url_info.section_name,
url_info.option_name,
url_info.url,
replacements[url_info.url],
)
def _apply_board_specific_config(
platformio_ini_path: Path, custom_zip_cache_dir: Path
) -> None:
"""
Enhanced config parser with improved zip detection and error handling.
Modifies platformio.ini in-place with resolved local paths.
Uses concurrent downloads for better performance.
"""
cache_manager = PlatformIOCache(custom_zip_cache_dir)
try:
pio_ini = PlatformIOIni.parseFile(platformio_ini_path)
print(f"Parsed platformio.ini: {platformio_ini_path}")
except (configparser.Error, FileNotFoundError) as e:
logger.error(f"Error reading platformio.ini: {e}")
return
# Step 1: Collect all URLs from platformio.ini
all_urls = _collect_all_zip_urls(pio_ini)
if not all_urls:
print("No zip artifacts found to process")
return
# Step 2: Deduplicate URLs
unique_urls = _dedupe_urls(all_urls)
print(f"Found {len(all_urls)} total URLs, {len(unique_urls)} unique")
# Step 3: Download and process all unique URLs
replacements = _download_and_process_urls(unique_urls, cache_manager)
# Step 4: Replace all URLs in platformio.ini with resolved local paths
if replacements:
_replace_all_urls(pio_ini, all_urls, replacements)
# Write back atomically
try:
pio_ini.dump(platformio_ini_path)
print(f"Updated platformio.ini with {len(replacements)} resolved artifacts")
except Exception as e:
logger.error(f"Failed to update platformio.ini: {e}")
raise
# Public API function
def resolve_and_cache_platform_artifacts(
platformio_ini_path: Path, cache_dir: Path
) -> None:
"""
Main entry point for resolving and caching PlatformIO platform artifacts.
Args:
platformio_ini_path: Path to the platformio.ini file to process
cache_dir: Cache directory for storing artifacts
"""
print(f"Starting platform artifact resolution for {platformio_ini_path}")
print(f"Using cache directory: {cache_dir}")
_apply_board_specific_config(platformio_ini_path, cache_dir)
print("Platform artifact resolution completed")
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/platformio_ini.py
+2170
View File
File diff suppressed because it is too large. Load diff
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/test_compiler.py
+983
View File
@@ -0,0 +1,983 @@
#!/usr/bin/env python3
"""
FastLED Test Build System - Python Compiler API Integration
This module provides a high-performance test compilation system built on the proven
ci.util.ci.util.clang_compiler API that delivers 8x faster build times compared to CMake.
Key Features:
- Parallel test compilation using ThreadPoolExecutor
- 90%+ cache hit rates with proven compiler settings
- Integration with build_unit.toml configuration
- Support for STUB platform testing without hardware dependencies
- GDB-compatible debug symbol generation
- Consistent performance across all platforms
"""
import hashlib
import os
import shutil
import subprocess
import sys
import tempfile
import textwrap
import time
import tomllib
from concurrent.futures import Future, as_completed
from dataclasses import dataclass, field
from pathlib import Path
from types import TracebackType
from typing import Any, Callable, Dict, List, Optional
from ci.util.paths import PROJECT_ROOT
def optimize_python_command(cmd: list[str]) -> list[str]:
"""
Optimize command list for subprocess execution in uv environment.
For python commands, we need to use 'uv run python' to ensure access to
installed packages like ziglang. Direct sys.executable bypasses uv environment.
Args:
cmd: Command list that may contain 'python' as first element
Returns:
list[str]: Optimized command with 'python' prefixed by 'uv run'
"""
if cmd and (cmd[0] == "python" or cmd[0] == "python3"):
# Use uv run python to ensure access to uv-managed packages
optimized_cmd = ["uv", "run", "python"] + cmd[1:]
return optimized_cmd
return cmd
from ci.compiler.clang_compiler import (
BuildFlags,
BuildTools,
Compiler,
CompilerOptions,
LinkOptions,
Result,
create_compiler_options_from_toml,
link_program_sync,
load_build_flags_from_toml,
)
@dataclass
class TestExecutable:
"""Represents a compiled test executable"""
name: str
executable_path: Path
test_source_path: Path
@dataclass
class CompileResult:
"""Result of test compilation operation"""
success: bool
compiled_count: int
duration: float
errors: List["CompileError"] = field(default_factory=lambda: list())
@dataclass
class CompileError:
"""Compilation error for a specific test"""
test_name: str
message: str
class FastLEDTestCompiler:
"""
Test compiler built on proven Compiler API for 8x faster builds.
This class leverages the proven ci.util.ci.util.clang_compiler.Compiler API that has already
demonstrated 4x+ performance improvements in example compilation. By using the same
proven patterns and configurations, we inherit the optimized build flags, caching,
and parallel compilation capabilities.
"""
# Class variable to store existing instance for get_existing_instance()
_existing_instance: Optional["FastLEDTestCompiler"] = None
def __init__(
self,
compiler: Compiler,
build_dir: Path,
project_root: Path,
quick_build: bool = True,
strict_mode: bool = False,
no_unity: bool = False,
):
self.compiler = compiler
self.build_dir = build_dir
self.project_root = project_root
self.quick_build = quick_build
self.strict_mode = strict_mode
self.no_unity = no_unity
self.compiled_tests: List[TestExecutable] = []
self.linking_failures: List[str] = []
self.build_flags = self._load_build_flags()
FastLEDTestCompiler._existing_instance = self
def _load_build_flags(self) -> BuildFlags:
"""Load build flags from TOML configuration"""
toml_path = self.project_root / "ci" / "build_unit.toml"
return BuildFlags.parse(toml_path, self.quick_build, self.strict_mode)
@classmethod
def create_for_unit_tests(
cls,
project_root: Path,
clean_build: bool = False,
enable_static_analysis: bool = False,
specific_test: str | None = None,
quick_build: bool = True,
strict_mode: bool = False,
no_unity: bool = False,
) -> "FastLEDTestCompiler":
"""
Create compiler configured for FastLED unit tests using TOML build flags.
This configuration uses the new TOML-based build flag system:
- ci/build_unit.toml for centralized flag configuration
- Support for build modes (quick/debug) and strict mode
- STUB platform for hardware-free testing
- Precompiled headers for faster compilation
- Parallel compilation with ThreadPoolExecutor
"""
# Set up build directory in .build/fled/unit for consistency
build_dir = project_root / ".build" / "fled" / "unit"
if clean_build and build_dir.exists():
print("###########################")
print("# CLEANING UNIT TEST BUILD DIR #")
print("###########################")
import errno
import os
import shutil
import stat
import time
def handle_remove_readonly(
func: Callable[[str], None],
path: str,
exc: tuple[type[BaseException], BaseException, TracebackType | None],
) -> None:
"""Error handler for Windows readonly files"""
if hasattr(exc[1], "errno") and exc[1].errno == errno.EACCES: # type: ignore
os.chmod(path, stat.S_IWRITE)
func(path)
else:
raise
# Windows-compatible directory removal with retry
max_retries = 3
for attempt in range(max_retries):
try:
# Use onerror parameter for compatibility across Python versions
shutil.rmtree(build_dir, onerror=handle_remove_readonly)
break
except (OSError, PermissionError) as e:
if attempt < max_retries - 1:
print(
f"Warning: Failed to remove build directory (attempt {attempt + 1}): {e}"
)
time.sleep(0.01) # Brief pause before retry
continue
else:
print(
f"Warning: Could not remove build directory after {max_retries} attempts: {e}"
)
print("Continuing with existing directory...")
break
build_dir.mkdir(parents=True, exist_ok=True)
# Additional compiler args beyond TOML
additional_compiler_args = [
f"-I{project_root}/src/platforms/stub", # STUB platform headers
]
# Load build flags using BuildFlags.parse() to get platform-specific configuration
toml_path = project_root / "ci" / "build_unit.toml"
build_flags = BuildFlags.parse(toml_path, quick_build, strict_mode)
# Load TOML config for test-specific flags that aren't in BuildFlags
with open(toml_path, "rb") as f:
config = tomllib.load(f)
# Get test-specific defines from TOML (not included in BuildFlags.parse)
test_defines = config.get("test", {}).get("defines", [])
# Create compiler args from BuildFlags (includes platform-specific flags)
compiler_args: List[str] = []
compiler_args.extend(
build_flags.compiler_flags
) # Platform-aware compiler flags
compiler_args.extend(build_flags.include_flags) # Platform-aware include flags
# Add strict mode flags if enabled (already handled by BuildFlags.parse)
# Add additional compiler args
if additional_compiler_args:
compiler_args.extend(additional_compiler_args)
# Extract defines without the "-D" prefix for CompilerOptions
defines: List[str] = []
for define in test_defines:
if define.startswith("-D"):
defines.append(define[2:]) # Remove "-D" prefix
else:
defines.append(define)
# Get tools configuration from BuildFlags (respects build_unit.toml)
# Use modern command-based approach
compiler_command = build_flags.tools.cpp_compiler
print(f"Using compiler from build_unit.toml: {compiler_command}")
# Use compiler command from TOML + flags as final compiler args
final_compiler_args = compiler_command + compiler_args
# Create compiler options with TOML-loaded flags and tools
settings = CompilerOptions(
include_path=str(project_root / "src"),
defines=defines,
compiler_args=final_compiler_args, # Use cache-aware compiler args
# Note: archiver and compiler fields are now handled via command-based approach
)
compiler = Compiler(settings, build_flags)
# Create final instance with proper compiler and flags
instance = cls(
compiler, build_dir, project_root, quick_build, strict_mode, no_unity
)
return instance
def discover_test_files(self, specific_test: str | None = None) -> List[Path]:
"""
Discover test_*.cpp files using proven patterns.
Uses the same file discovery patterns as the existing CMake system
to ensure 100% compatibility with current test discovery logic.
"""
tests_dir = Path(PROJECT_ROOT) / "tests"
test_files: List[Path] = []
# Find all test files in tests directory and subdirectories
for test_file in tests_dir.rglob("test_*.cpp"):
# Skip the doctest_main.cpp file (not a test file)
if test_file.name == "doctest_main.cpp":
continue
# Handle both "test_name" and "name" formats for compatibility
test_stem = test_file.stem
test_name = test_stem.replace("test_", "")
# Check if we should do fuzzy matching (if there's a * in the name)
if specific_test:
if "*" in specific_test:
# Convert glob pattern to regex pattern
import re
pattern = specific_test.replace("*", ".*")
if re.search(pattern, test_stem) or re.search(pattern, test_name):
test_files.append(test_file)
else:
# Exact matching - either the full name or with test_ prefix (case-insensitive)
if (
test_stem.lower() == specific_test.lower()
or test_stem.lower() == f"test_{specific_test}".lower()
or test_name.lower() == specific_test.lower()
):
test_files.append(test_file)
else:
test_files.append(test_file)
return test_files
def compile_all_tests(self, specific_test: str | None = None) -> CompileResult:
"""
Compile all tests in parallel using proven API patterns.
This method uses the same parallel compilation patterns proven in
example compilation that deliver 4x+ performance improvements.
"""
compile_start = time.time()
# If specific_test is not provided in the method call, use the one from discover_test_files
test_files = self.discover_test_files(specific_test)
if not test_files:
return CompileResult(success=True, compiled_count=0, duration=0.0)
print(f"Compiling {len(test_files)} test files using proven Python API...")
# Convert absolute path to relative for display
rel_build_dir = os.path.relpath(self.build_dir)
print(f"Build directory: {rel_build_dir}")
# Always show build configuration in unit test mode
print("\nFastLED Library Build Configuration:")
if self.no_unity:
print(" ❌ Unity build disabled - Compiling individual source files")
else:
print(" ✅ Unity build enabled - Using glob pattern:")
print(" src/**/*.cpp")
print("\nPrecompiled header status:")
if self.compiler.settings.use_pch:
print(" ✅ PCH enabled - Using precompiled headers for faster compilation")
print(" PCH content:")
print(textwrap.indent(self.compiler.generate_pch_header(), " "))
# Create PCH for faster compilation
print("\n🔧 Creating precompiled header...")
pch_success = self.compiler.create_pch_file()
if pch_success:
print(" ✅ PCH created successfully")
else:
print(" ❌ PCH creation failed - continuing with direct compilation")
else:
print(" ❌ PCH disabled - Not using precompiled headers")
print("\nCompiler flags:")
for flag in self.compiler.get_compiler_args():
print(f" {flag}")
print("\nLinker flags:")
for flag in self._get_platform_linker_args(self.build_dir / "libfastled.lib"):
print(f" {flag}")
print("")
# Print list of test files being compiled
print("Test files to compile:")
for i, test_file in enumerate(test_files, 1):
print(f" {i}. {test_file.name}")
print("")
# Submit parallel compilation jobs (proven pattern from example compilation)
future_to_test: Dict[Future[Result], Path] = {}
# Check if we're running in parallel or sequential mode
if os.environ.get("NO_PARALLEL"):
print("Starting sequential compilation of test files...")
else:
print("Starting parallel compilation of test files...")
for test_file in test_files:
# Compile to object file first (since compile_cpp_file uses -c flag)
obj_path = self.build_dir / f"{test_file.stem}.o"
# Convert absolute path to relative for display
rel_obj_path = os.path.relpath(obj_path)
print(f"Submitting compilation job for: {test_file.name} -> {rel_obj_path}")
# Show compilation command if enabled
if os.environ.get("FASTLED_TEST_SHOW_COMPILE", "").lower() in (
"1",
"true",
"yes",
):
# Convert absolute path to relative for display
rel_obj_path = os.path.relpath(obj_path)
print(f"[COMPILE] {test_file.name} -> {rel_obj_path}")
compile_future = self.compiler.compile_cpp_file(
test_file,
output_path=obj_path,
additional_flags=[
"-I",
str(self.project_root / "src"),
"-I",
str(self.project_root / "tests"),
# NOTE: Compiler flags now come from build_unit.toml
],
)
future_to_test[compile_future] = test_file
# Collect compilation results with progress reporting
compiled_objects: List[Path] = []
errors: List[CompileError] = []
completed = 0
print(f"Waiting for {len(future_to_test)} compilation jobs to complete...")
try:
for future in as_completed(future_to_test.keys()):
test_file = future_to_test[future]
result: Result = future.result()
completed += 1
if result.ok:
obj_path = self.build_dir / f"{test_file.stem}.o"
compiled_objects.append(obj_path)
print(f"[{completed}/{len(test_files)}] Compiled {test_file.name}")
else:
errors.append(
CompileError(
test_name=test_file.stem,
message=result.stderr
or result.stdout
or "Compilation failed",
)
)
print(
f"[{completed}/{len(test_files)}] FAILED {test_file.name}: {result.stderr}"
)
except KeyboardInterrupt:
print("\nKeyboard interrupt detected during compilation")
# Clean up any in-progress compilations
for future in future_to_test.keys():
future.cancel()
import _thread
_thread.interrupt_main()
raise
if errors:
duration = time.time() - compile_start
print(f"Compilation failed with {len(errors)} errors in {duration:.2f}s")
return CompileResult(
success=False, compiled_count=0, duration=duration, errors=errors
)
print(f"All {len(compiled_objects)} object files compiled successfully")
# Link each test to executable using proven linking API
self.compiled_tests = self._link_tests(compiled_objects)
# Check for linking failures and add them to errors
if hasattr(self, "linking_failures") and self.linking_failures:
for failure in self.linking_failures:
# Parse the failure string to extract test name and error
if ":" in failure:
test_name, error_msg = failure.split(":", 1)
errors.append(
CompileError(
test_name=test_name.strip(),
message=f"Linking failed: {error_msg.strip()}",
)
)
duration = time.time() - compile_start
# Success requires both compilation AND linking to succeed
success = len(errors) == 0 and len(self.compiled_tests) > 0
if success:
print(
f"SUCCESS: Compiled and linked {len(self.compiled_tests)} tests in {duration:.2f}s"
)
# List all the successful test executables
print("Test executables created:")
for i, test in enumerate(self.compiled_tests, 1):
# Convert absolute path to relative for display
rel_exe_path = os.path.relpath(test.executable_path)
print(f" {i}. {test.name} -> {rel_exe_path}")
else:
print(f"FAILED: {len(errors)} total failures (compilation + linking)")
return CompileResult(
success=success,
compiled_count=len(self.compiled_tests),
duration=duration,
errors=errors,
)
def _link_tests(self, compiled_objects: List[Path]) -> List[TestExecutable]:
"""Link each test to executable using proven linking API"""
print(f"Linking {len(compiled_objects)} test executables...")
# First, build a complete FastLED library similar to CMake approach
fastled_lib_path = self._build_fastled_library()
# Compile doctest_main.cpp once for all tests (provides main function and doctest implementation)
doctest_main_path = self.project_root / "tests" / "doctest_main.cpp"
doctest_obj_path = self.build_dir / "doctest_main.o"
if not doctest_obj_path.exists():
print("Compiling doctest_main.cpp...")
doctest_future = self.compiler.compile_cpp_file(
doctest_main_path,
output_path=doctest_obj_path,
additional_flags=["-c"], # Compile only
)
doctest_result: Result = doctest_future.result()
if not doctest_result.ok:
print(
f"ERROR: Failed to compile doctest_main.cpp: {doctest_result.stderr}"
)
return []
print(f"Compiled FastLED library and doctest_main")
compiled_tests: List[TestExecutable] = []
linking_failures: List[str] = []
cache_hits = 0
cache_misses = 0
# Link each test with the FastLED library
for obj_path in compiled_objects:
# Derive test name from object file
test_name = obj_path.stem.replace("test_", "")
exe_path = self.build_dir / f"test_{test_name}.exe"
# Define clean execution path in tests/bin/ (this is where tests will actually run from)
tests_bin_dir = self.project_root / "tests" / "bin"
tests_bin_dir.mkdir(parents=True, exist_ok=True)
clean_exe_path = tests_bin_dir / f"test_{test_name}.exe"
# Check if this test has its own main() function
test_source = self.project_root / "tests" / f"test_{test_name}.cpp"
has_own_main = self._test_has_own_main(test_source)
# Get linker args (needed for cache key calculation)
linker_args = self._get_platform_linker_args(fastled_lib_path)
# Calculate comprehensive cache key based on all linking inputs
if has_own_main:
# For standalone tests, only include test object file
cache_key = self._calculate_link_cache_key(
obj_path, fastled_lib_path, linker_args
)
else:
# For doctest tests, include both test object and doctest_main object
# Calculate hashes for both object files and combine them
test_obj_hash = self._calculate_file_hash(obj_path)
doctest_obj_hash = self._calculate_file_hash(doctest_obj_path)
combined_obj_hash = hashlib.sha256(
f"{test_obj_hash}+{doctest_obj_hash}".encode("utf-8")
).hexdigest()
# Create a temporary cache key with combined object hash
fastled_hash = self._calculate_file_hash(fastled_lib_path)
linker_hash = self._calculate_linker_args_hash(linker_args)
combined = f"fastled:{fastled_hash}|test:{combined_obj_hash}|flags:{linker_hash}"
cache_key = hashlib.sha256(combined.encode("utf-8")).hexdigest()[:16]
# Check for cached executable based on comprehensive cache key
cached_exe = self._get_cached_executable(test_name, cache_key)
if cached_exe:
print(f" {test_name}: Using cached executable (link cache hit)")
try:
shutil.copy2(cached_exe, exe_path)
# Also copy to clean execution location
shutil.copy2(cached_exe, clean_exe_path)
cache_hits += 1
test_exe = TestExecutable(
name=test_name,
executable_path=clean_exe_path, # Point to clean location for execution
test_source_path=test_source,
)
compiled_tests.append(test_exe)
continue # Skip linking, we have cached result
except Exception as e:
print(
f" {test_name}: Warning - failed to copy cached executable, will relink: {e}"
)
# Fall through to linking logic
if has_own_main:
# Link standalone test without doctest_main.o
link_options = LinkOptions(
object_files=[obj_path], # Only the test object file
output_executable=exe_path,
linker=f'"{sys.executable}" -m ziglang c++', # Use optimized Python executable for linking
linker_args=linker_args, # Use pre-calculated args for consistency
)
else:
# Link with the FastLED library instead of individual objects
link_options = LinkOptions(
object_files=[
obj_path,
doctest_obj_path,
], # Test object + doctest main
output_executable=exe_path,
linker=f'"{sys.executable}" -m ziglang c++', # Use optimized Python executable for linking
linker_args=linker_args, # Use pre-calculated args for consistency
)
# Show linking command if enabled
if os.environ.get("FASTLED_TEST_SHOW_LINK", "").lower() in (
"1",
"true",
"yes",
):
# Convert absolute path to relative for display
rel_exe_path = os.path.relpath(exe_path)
print(f"[LINK] {test_name} -> {rel_exe_path}")
print(f"Linking test: {test_name}")
# STRICT: Provide explicit BuildFlags - NO defaults allowed
link_result: Result = link_program_sync(
link_options, self.compiler.build_flags
)
if not link_result.ok:
# Create more detailed error message showing link command context
error_msg = f" {test_name}: Linking failed - Linker: {link_options.linker}, Output: {link_options.output_executable}, Objects: {len(link_options.object_files)} files"
print(error_msg)
print(f" Error details: {link_result.stderr}")
linking_failures.append(f"{test_name}: {link_result.stderr}")
continue
print(f" {test_name}: Linking successful")
cache_misses += 1
# Cache the newly linked executable for future use
self._cache_executable(test_name, cache_key, exe_path)
# Copy to clean execution location
shutil.copy2(exe_path, clean_exe_path)
test_exe = TestExecutable(
name=test_name,
executable_path=clean_exe_path,
test_source_path=test_source, # Point to clean location for execution
)
compiled_tests.append(test_exe)
if linking_failures:
print(f"ERROR: {len(linking_failures)} linking failures:")
for failure in linking_failures:
print(f" {failure}")
# Store linking failures for later reporting
self.linking_failures = linking_failures
else:
self.linking_failures = []
# Report link cache statistics
total_tests = cache_hits + cache_misses
if total_tests > 0:
hit_rate = (cache_hits / total_tests) * 100
print(
f"Link cache: {cache_hits} hits, {cache_misses} misses ({hit_rate:.1f}% hit rate) - caching by lib+test+flags"
)
print(f"Successfully linked {len(compiled_tests)} test executables")
return compiled_tests
def _test_has_own_main(self, test_source_path: Path) -> bool:
"""Check if a test file defines its own main() function"""
try:
if not test_source_path.exists():
return False
content = test_source_path.read_text(encoding="utf-8")
# Look for main function definition (simple pattern matching)
import re
# Match "int main(" allowing for whitespace and various formats
main_pattern = r"\bint\s+main\s*\("
return bool(re.search(main_pattern, content))
except Exception:
# If we can't read the file, assume it doesn't have main
return False
def _build_fastled_library(self) -> Path:
"""Build a complete FastLED static library like CMake does"""
print("Building FastLED static library...")
# Define library path
fastled_lib_path = self.build_dir / "libfastled.lib"
# If library already exists, return it (for faster rebuilds)
if fastled_lib_path.exists():
# Convert absolute path to relative for display
rel_lib_path = os.path.relpath(fastled_lib_path)
print(f"Using existing FastLED library: {rel_lib_path}")
return fastled_lib_path
# Compile essential FastLED source files for STUB platform
# User directive: Just glob ALL .cpp files in src/** - no exclusions needed
import glob
fastled_src_dir = self.project_root / "src"
# Include ALL .cpp files recursively - no platform exclusions
all_cpp_files: List[str] = glob.glob(
str(fastled_src_dir / "**" / "*.cpp"), recursive=True
)
fastled_sources: List[Path] = []
for cpp_file in all_cpp_files:
# Include ALL files - no filtering needed
fastled_sources.append(Path(cpp_file))
if self.no_unity:
print(
f"Individual compilation: Found {len(fastled_sources)} FastLED .cpp files for compilation"
)
else:
print(
f"Unity build mode: Found {len(fastled_sources)} FastLED .cpp files for compilation"
)
# Compile all FastLED source files to object files
fastled_objects: List[Path] = []
for src_file in fastled_sources:
src_path = self.project_root / src_file
# Create unique object file name including directory path to avoid collisions
# Convert path separators to underscores to create valid filename
relative_path = src_path.relative_to(self.project_root / "src")
safe_name = (
str(relative_path.with_suffix("")).replace("/", "_").replace("\\", "_")
)
obj_path = self.build_dir / f"{safe_name}_fastled.o"
if not obj_path.exists():
future = self.compiler.compile_cpp_file(
src_path,
output_path=obj_path,
additional_flags=[
"-c", # Compile only
"-DFASTLED_STUB_IMPL", # Essential for STUB platform
# NOTE: All defines and compiler flags now come from build_unit.toml
# Remove hardcoded defines - they should be in [test] section
],
)
result: Result = future.result()
if not result.ok:
print(f"ERROR: Failed to compile {src_file}: {result.stderr}")
# Continue with other files rather than failing completely
continue
fastled_objects.append(obj_path)
print(f"Creating static library from {len(fastled_objects)} object files...")
# Create static library using ar (archiver)
# First try llvm-ar since it's more likely to be available
ar_cmd: List[str] = [
"llvm-ar", # LLVM archiver tool (works with Clang)
"rcs", # Create archive with symbol table
str(fastled_lib_path),
] + [str(obj) for obj in fastled_objects]
# Create static library using ziglang ar (archiver)
print("Creating static library using ziglang ar...")
lib_cmd: List[str] = [
"python",
"-m",
"ziglang",
"ar",
"rcs", # Create archive with symbol table
str(fastled_lib_path),
] + [str(obj) for obj in fastled_objects]
try:
# Optimize command to use sys.executable instead of shell 'python' resolution
python_exe = optimize_python_command(lib_cmd)
# Use streaming to prevent buffer overflow
process = subprocess.Popen(
python_exe,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
bufsize=1,
encoding="utf-8",
errors="replace",
cwd=self.build_dir,
)
stdout_lines: list[str] = []
stderr_lines: list[str] = []
while True:
stdout_line = process.stdout.readline() if process.stdout else ""
stderr_line = process.stderr.readline() if process.stderr else ""
if stdout_line:
stdout_lines.append(stdout_line.rstrip())
if stderr_line:
stderr_lines.append(stderr_line.rstrip())
if process.poll() is not None:
remaining_stdout = process.stdout.read() if process.stdout else ""
remaining_stderr = process.stderr.read() if process.stderr else ""
if remaining_stdout:
for line in remaining_stdout.splitlines():
stdout_lines.append(line.rstrip())
if remaining_stderr:
for line in remaining_stderr.splitlines():
stderr_lines.append(line.rstrip())
break
if process.returncode != 0:
print(f"ERROR: Failed to create static library: {stderr_lines}")
return fastled_lib_path # Return path even if creation failed
# Convert absolute path to relative for display
rel_lib_path = os.path.relpath(fastled_lib_path)
print(f"Successfully created FastLED library: {rel_lib_path}")
return fastled_lib_path
except Exception as e:
print(f"ERROR: Exception during library creation: {e}")
raise
def get_test_executables(
self, specific_test: str | None = None
) -> List[TestExecutable]:
"""Get compiled test executables with name filtering"""
if specific_test:
# Check if we should do fuzzy matching (if there's a * in the name)
if "*" in specific_test:
# Convert glob pattern to regex pattern
import re
pattern = specific_test.replace("*", ".*")
return [
t
for t in self.compiled_tests
if re.search(pattern, t.name)
or re.search(pattern, f"test_{t.name}")
]
else:
# Exact matching - either the full name or with test_ prefix (case-insensitive)
return [
t
for t in self.compiled_tests
if (
t.name.lower() == specific_test.lower()
or t.name.lower() == f"test_{specific_test}".lower()
or t.name.lower() == specific_test.replace("test_", "").lower()
)
]
return self.compiled_tests
@property
def link_cache_dir(self) -> Path:
"""Get the link cache directory"""
cache_dir = self.build_dir / "link_cache"
cache_dir.mkdir(exist_ok=True)
return cache_dir
def _calculate_file_hash(self, file_path: Path) -> str:
"""Calculate SHA256 hash of a file"""
if not file_path.exists():
return "no_file"
hash_sha256 = hashlib.sha256()
with open(file_path, "rb") as f:
# Read in chunks to handle large files efficiently
for chunk in iter(lambda: f.read(4096), b""):
hash_sha256.update(chunk)
return hash_sha256.hexdigest()
def _calculate_linker_args_hash(self, linker_args: List[str]) -> str:
"""Calculate SHA256 hash of linker arguments"""
# Convert linker args to a stable string representation
args_str = "|".join(sorted(linker_args))
hash_sha256 = hashlib.sha256()
hash_sha256.update(args_str.encode("utf-8"))
return hash_sha256.hexdigest()
def _calculate_link_cache_key(
self, test_obj_path: Path, fastled_lib_path: Path, linker_args: List[str]
) -> str:
"""Calculate comprehensive cache key for linking"""
# Hash all the factors that affect linking output
fastled_hash = self._calculate_file_hash(fastled_lib_path)
test_obj_hash = self._calculate_file_hash(test_obj_path)
linker_hash = self._calculate_linker_args_hash(linker_args)
# Combine all hashes into a single cache key
combined = f"fastled:{fastled_hash}|test:{test_obj_hash}|flags:{linker_hash}"
final_hash = hashlib.sha256(combined.encode("utf-8")).hexdigest()
return final_hash[:16] # Use first 16 chars for readability
def _get_cached_executable(self, test_name: str, cache_key: str) -> Optional[Path]:
"""Check if a cached executable exists for this test and cache key"""
cached_exe = self.link_cache_dir / f"{test_name}_{cache_key}.exe"
return cached_exe if cached_exe.exists() else None
def _cache_executable(self, test_name: str, cache_key: str, exe_path: Path) -> None:
"""Cache an executable with the given cache key"""
if not exe_path.exists():
return
cached_exe = self.link_cache_dir / f"{test_name}_{cache_key}.exe"
try:
shutil.copy2(exe_path, cached_exe)
print(f" {test_name}: Cached executable for future use")
except Exception as e:
print(f" {test_name}: Warning - failed to cache executable: {e}")
def _get_platform_linker_args(self, fastled_lib_path: Path) -> List[str]:
"""Get platform-specific linker arguments using BuildFlags"""
# Use the build_flags that includes platform-specific configuration
args = self.build_flags.link_flags.copy()
# Load TOML config for test-specific and build mode flags
toml_path = self.project_root / "ci" / "build_unit.toml"
with open(toml_path, "rb") as f:
config = tomllib.load(f)
# Add test-specific linking flags if available
test_link_flags = config.get("linking", {}).get("test", {}).get("flags", [])
args.extend(test_link_flags)
# Add build mode specific flags (use test-specific debug mode)
mode = "quick" if self.quick_build else "test_debug"
mode_flags = config.get("build_modes", {}).get(mode, {}).get("link_flags", [])
args.extend(mode_flags)
# Add FastLED library
args.append(str(fastled_lib_path)) # Link against the FastLED static library
return args
@classmethod
def get_existing_instance(cls) -> Optional["FastLEDTestCompiler"]:
"""Get existing compiler instance for reuse"""
return cls._existing_instance
def check_iwyu_available() -> bool:
"""Check if include-what-you-use is available (preserving existing logic)"""
import shutil
return (
shutil.which("include-what-you-use") is not None
or shutil.which("iwyu") is not None
)
# For testing this module directly
if __name__ == "__main__":
print("Testing FastLEDTestCompiler...")
# Create test compiler
test_compiler = FastLEDTestCompiler.create_for_unit_tests(
project_root=Path(PROJECT_ROOT), clean_build=True
)
# Discover test files
test_files = test_compiler.discover_test_files()
print(f"Discovered {len(test_files)} test files")
# Test compilation of a few files
if test_files:
limited_files = test_files[:3] # Test first 3 files
print(f"Testing compilation of {len(limited_files)} test files...")
compile_result = test_compiler.compile_all_tests()
if compile_result.success:
print(
f"SUCCESS: Compiled {compile_result.compiled_count} tests in {compile_result.duration:.2f}s"
)
executables = test_compiler.get_test_executables()
for exe in executables:
print(f" Executable: {exe.name} -> {exe.executable_path}")
else:
print(f"FAILED: {len(compile_result.errors)} errors")
for error in compile_result.errors:
print(f" {error.test_name}: {error.message}")
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/test_example_compilation.py
+2595
View File
File diff suppressed because it is too large. Load diff
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/compiler/unity_archive.py
+171
View File
@@ -0,0 +1,171 @@
#!/usr/bin/env python3
import argparse
import os
import sys
from concurrent.futures import Future
from pathlib import Path
from typing import List, Sequence
from ci.compiler.clang_compiler import Compiler, CompilerOptions, Result
from ci.compiler.test_example_compilation import (
create_fastled_compiler,
get_fastled_core_sources,
)
def _sorted_src_files(files: Sequence[Path]) -> List[Path]:
project_root = Path.cwd()
def key_fn(p: Path) -> str:
try:
rel = p.relative_to(project_root)
except Exception:
rel = p
return rel.as_posix()
return sorted(files, key=key_fn)
def _partition(files: List[Path], chunks: int) -> List[List[Path]]:
total = len(files)
if chunks <= 0:
raise ValueError("chunks must be >= 1")
chunks = min(chunks, total) if total > 0 else 1
base = total // chunks
rem = total % chunks
partitions: List[List[Path]] = []
start = 0
for i in range(chunks):
size = base + (1 if i < rem else 0)
end = start + size
if size > 0:
partitions.append(files[start:end])
start = end
return partitions
def build_unity_chunks_and_archive(
compiler: Compiler,
chunks: int,
output_archive: Path,
unity_dir: Path,
no_parallel: bool,
) -> Path:
unity_dir.mkdir(parents=True, exist_ok=True)
# Gather and sort source files consistently with existing library logic
files = get_fastled_core_sources()
# Exclude any main-like TU
files = [p for p in files if p.name != "stub_main.cpp"]
files = _sorted_src_files(files)
if not files:
raise RuntimeError("No source files found under src/** for unity build")
partitions = _partition(files, chunks)
# Prepare per-chunk unity paths
unity_cpp_paths: List[Path] = [
unity_dir / f"unity{i + 1}.cpp" for i in range(len(partitions))
]
unity_obj_paths: List[Path] = [p.with_suffix(".o") for p in unity_cpp_paths]
# Compile chunks
futures: List[Future[Result]] = []
compile_opts = CompilerOptions(
include_path=compiler.settings.include_path,
compiler=compiler.settings.compiler,
defines=compiler.settings.defines,
std_version=compiler.settings.std_version,
compiler_args=compiler.settings.compiler_args,
use_pch=False,
additional_flags=["-c"],
parallel=not no_parallel,
)
def submit(idx: int) -> Future[Result]:
chunk_files = partitions[idx]
unity_cpp = unity_cpp_paths[idx]
return compiler.compile_unity(compile_opts, chunk_files, unity_cpp)
if no_parallel:
# Sequential execution
for i in range(len(partitions)):
result = submit(i).result()
if not result.ok:
raise RuntimeError(f"Unity chunk {i + 1} failed: {result.stderr}")
else:
futures = [submit(i) for i in range(len(partitions))]
for i, fut in enumerate(futures):
result = fut.result()
if not result.ok:
raise RuntimeError(f"Unity chunk {i + 1} failed: {result.stderr}")
# Create archive from chunk objects
from ci.compiler.clang_compiler import LibarchiveOptions
output_archive.parent.mkdir(parents=True, exist_ok=True)
archive_result = compiler.create_archive(
unity_obj_paths, output_archive, LibarchiveOptions()
).result()
if not archive_result.ok:
raise RuntimeError(f"Archive creation failed: {archive_result.stderr}")
return output_archive
def main() -> int:
parser = argparse.ArgumentParser(
description="Chunked UNITY build and archive for FastLED src/**"
)
parser.add_argument(
"--chunks", type=int, default=1, help="Number of unity chunks (1-4 typical)"
)
parser.add_argument(
"--no-parallel", action="store_true", help="Compile unity chunks sequentially"
)
parser.add_argument(
"--output",
type=str,
default=str(Path(".build/fastled/libfastled.a")),
help="Path to output archive libfastled.a",
)
parser.add_argument(
"--unity-dir",
type=str,
default=str(Path(".build/fastled/unity")),
help="Directory to place generated unity{N}.cpp/.o files",
)
parser.add_argument(
"--use-pch",
action="store_true",
help="Enable PCH (not recommended for unity builds)",
)
args = parser.parse_args()
# Anchor to project root (two levels up from this file)
here = Path(__file__).resolve()
project_root = here.parent.parent.parent
os.chdir(project_root)
try:
compiler = create_fastled_compiler(
use_pch=args.use_pch, parallel=not args.no_parallel
)
out = build_unity_chunks_and_archive(
compiler=compiler,
chunks=args.chunks,
output_archive=Path(args.output),
unity_dir=Path(args.unity_dir),
no_parallel=args.no_parallel,
)
print(f"[UNITY] Created archive: {out}")
return 0
except KeyboardInterrupt:
raise
except Exception as e:
print(f"\x1b[31m[UNITY] FAILED: {e}\x1b[0m")
return 1
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/demo_symbol_analysis.py
+119
View File
@@ -0,0 +1,119 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
"""
Demo script showing how to use the new generic symbol analysis functionality
"""
import json
import subprocess
import sys
from pathlib import Path
from typing import List
def run_symbol_analysis(board_name: str):
"""Run symbol analysis for a specific board"""
print(f"Running symbol analysis for {board_name}...")
try:
result = subprocess.run(
[sys.executable, "ci/util/symbol_analysis.py", "--board", board_name],
cwd=".",
capture_output=True,
text=True,
)
if result.returncode == 0:
print(f"✅ Symbol analysis completed successfully for {board_name}")
return True
else:
print(f"❌ Symbol analysis failed for {board_name}")
print(f"Error: {result.stderr}")
return False
except Exception as e:
print(f"❌ Error running symbol analysis for {board_name}: {e}")
return False
def load_and_summarize_results(board_name: str):
"""Load and display a summary of symbol analysis results"""
json_file = Path(f".build/{board_name}_symbol_analysis.json")
if not json_file.exists():
print(f"❌ Results file not found: {json_file}")
return
try:
with open(json_file) as f:
data = json.load(f)
summary = data["summary"]
print(f"\n📊 SUMMARY FOR {board_name.upper()}:")
print(f" Total symbols: {summary['total_symbols']}")
print(
f" Total size: {summary['total_size']} bytes ({summary['total_size'] / 1024:.1f} KB)"
)
print("\n🔝 TOP 5 LARGEST SYMBOLS:")
for i, symbol in enumerate(summary["largest_symbols"][:5], 1):
name = symbol.get("demangled_name", symbol["name"])
print(
f" {i}. {symbol['size']:6d} bytes - {name[:80]}{'...' if len(name) > 80 else ''}"
)
print("\n📁 SYMBOL TYPES:")
for sym_type, stats in list(summary["type_breakdown"].items())[:5]:
print(f" {sym_type}: {stats['count']} symbols, {stats['size']} bytes")
except Exception as e:
print(f"❌ Error loading results for {board_name}: {e}")
def main():
"""Demo the symbol analysis functionality"""
print("=" * 80)
print("FASTLED SYMBOL ANALYSIS DEMO")
print("=" * 80)
# List of boards to analyze (you can add more)
boards_to_analyze: List[str] = []
# Check which boards have build info available
build_dir = Path(".build")
if build_dir.exists():
for item in build_dir.iterdir():
if item.is_dir() and (item / "build_info.json").exists():
boards_to_analyze.append(item.name)
if not boards_to_analyze:
print("❌ No boards with build_info.json found in .build directory")
print(
" Please compile some examples first using: uv run -m ci.ci-compile <board> --examples Blink"
)
sys.exit(1)
print(
f"📋 Found {len(boards_to_analyze)} boards with build info: {', '.join(boards_to_analyze)}"
)
# Run symbol analysis for each board
successful_boards: List[str] = []
for board in boards_to_analyze:
if run_symbol_analysis(board):
successful_boards.append(board)
print(f"\n📈 RESULTS SUMMARY FOR {len(successful_boards)} BOARDS:")
print("=" * 80)
# Load and summarize results
for board in successful_boards:
load_and_summarize_results(board)
print()
print("🎉 Symbol analysis demo completed!")
print("📁 Detailed JSON files saved in .build/ directory")
print("💡 Use these files to identify optimization opportunities")
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/dockerfiles/DockerManager.py
+339
View File
@@ -0,0 +1,339 @@
import os
import subprocess
import sys
from typing import Any, Dict, List, Optional
class DockerManager:
def __init__(self):
# No direct client object, will use subprocess for docker CLI commands
pass
def _run_docker_command(
self,
command: List[str],
check: bool = True,
stream_output: bool = False,
timeout: Optional[int] = None,
interrupt_pattern: Optional[str] = None,
output_file: Optional[str] = None,
) -> subprocess.CompletedProcess[str]:
full_command = ["docker"] + command
print(f"Executing Docker command: {' '.join(full_command)}")
# Set environment to prevent MSYS2/Git Bash path conversion on Windows
env = os.environ.copy()
# Set UTF-8 encoding environment variables for Windows
env["PYTHONIOENCODING"] = "utf-8"
env["PYTHONUTF8"] = "1"
# Only set MSYS_NO_PATHCONV if we're in a Git Bash/MSYS2 environment
if (
"MSYSTEM" in os.environ
or os.environ.get("TERM") == "xterm"
or "bash.exe" in os.environ.get("SHELL", "")
):
env["MSYS_NO_PATHCONV"] = "1"
if stream_output:
# Use RunningProcess for streaming output
import re
import time
from ci.util.running_process import RunningProcess
proc = RunningProcess(full_command, env=env, auto_run=True)
pattern_found = False
timeout_occurred = False
start_time = time.time()
# Open output file if specified
output_handle = None
if output_file:
try:
output_handle = open(output_file, "w", encoding="utf-8")
except Exception as e:
print(f"Warning: Could not open output file {output_file}: {e}")
try:
with proc.line_iter(timeout=None) as it:
for line in it:
print(line)
# Write to output file if specified
if output_handle:
output_handle.write(line + "\n")
output_handle.flush() # Ensure immediate write
# Check for interrupt pattern (for informational purposes only)
if interrupt_pattern and re.search(interrupt_pattern, line):
print(f"Pattern detected: {line}")
pattern_found = True
# Check timeout
if timeout and (time.time() - start_time) > timeout:
print(
f"Timeout reached ({timeout}s), terminating container..."
)
timeout_occurred = True
break
# Wait for process to complete
returncode = proc.wait()
# Handle timeout case: return success (0) for timeouts as per requirement
if timeout_occurred:
print("Process terminated due to timeout - treating as success")
final_returncode = 0
else:
# Return the actual container exit code for all other cases
final_returncode = returncode
return subprocess.CompletedProcess(
args=full_command, returncode=final_returncode, stdout="", stderr=""
)
except Exception as e:
print(f"Error during streaming: {e}")
returncode = proc.wait() if hasattr(proc, "wait") else 1
return subprocess.CompletedProcess(
args=full_command, returncode=returncode, stdout="", stderr=str(e)
)
finally:
# Close output file if it was opened
if output_handle:
output_handle.close()
else:
# Use regular subprocess for non-streaming commands
result = subprocess.run(
full_command,
capture_output=True,
text=True,
check=check,
env=env,
encoding="utf-8",
errors="replace",
)
if check and result.returncode != 0:
print(
f"Docker command failed with exit code {result.returncode}",
file=sys.stderr,
)
print(f"Stdout: {result.stdout}", file=sys.stderr)
print(f"Stderr: {result.stderr}", file=sys.stderr)
return result
def pull_image(self, image_name: str, tag: str = "latest"):
"""Pulls the specified Docker image."""
print(f"Pulling image: {image_name}:{tag}")
self._run_docker_command(["pull", f"{image_name}:{tag}"])
print(f"Image {image_name}:{tag} pulled successfully.")
def run_container(
self,
image_name: str,
command: List[str],
volumes: Optional[Dict[str, Dict[str, str]]] = None,
environment: Optional[Dict[str, str]] = None,
detach: bool = False,
name: Optional[str] = None,
) -> str: # Return container ID as string
"""
Runs a Docker container with specified command, volume mounts, and environment variables.
Returns the container ID.
"""
print(
f"Running container from image: {image_name} with command: {' '.join(command)}"
)
docker_cmd = ["run", "--rm"] # --rm to automatically remove container on exit
if detach:
docker_cmd.append("-d")
if name:
docker_cmd.extend(["--name", name])
if volumes:
for host_path, container_path_info in volumes.items():
mode = container_path_info.get("mode", "rw")
docker_cmd.extend(
["-v", f"{host_path}:{container_path_info['bind']}:{mode}"]
)
if environment:
for key, value in environment.items():
docker_cmd.extend(["-e", f"{key}={value}"])
docker_cmd.append(image_name)
docker_cmd.extend(command)
result = self._run_docker_command(docker_cmd)
container_id = result.stdout.strip()
print(f"Container {container_id} started.")
return container_id
def run_container_streaming(
self,
image_name: str,
command: List[str],
volumes: Optional[Dict[str, Dict[str, str]]] = None,
environment: Optional[Dict[str, str]] = None,
name: Optional[str] = None,
timeout: Optional[int] = None,
interrupt_pattern: Optional[str] = None,
output_file: Optional[str] = None,
) -> int:
"""
Runs a Docker container with streaming output.
Returns actual container exit code (0 for success, timeout is treated as success).
Args:
image_name: Docker image to run
command: Command to execute in container
volumes: Volume mounts
environment: Environment variables
name: Container name
timeout: Timeout in seconds (if exceeded, returns 0)
interrupt_pattern: Pattern to detect in output (informational only)
output_file: Optional file path to write output to
"""
print(
f"Running container from image: {image_name} with command: {' '.join(command)}"
)
docker_cmd = ["run", "--rm"] # --rm to automatically remove container on exit
if name:
docker_cmd.extend(["--name", name])
if volumes:
for host_path, container_path_info in volumes.items():
mode = container_path_info.get("mode", "rw")
docker_cmd.extend(
["-v", f"{host_path}:{container_path_info['bind']}:{mode}"]
)
if environment:
for key, value in environment.items():
docker_cmd.extend(["-e", f"{key}={value}"])
docker_cmd.append(image_name)
docker_cmd.extend(command)
result = self._run_docker_command(
docker_cmd,
check=False,
stream_output=True,
timeout=timeout,
interrupt_pattern=interrupt_pattern,
output_file=output_file,
)
return result.returncode
def get_container_logs(self, container_id_or_name: str) -> str:
"""Retrieves logs from a container."""
print(f"Getting logs for container: {container_id_or_name}")
try:
result = self._run_docker_command(
["logs", container_id_or_name], check=False
)
if result.returncode == 0:
logs = result.stdout
print(f"Logs retrieved for {container_id_or_name}.")
return logs
else:
# Container may have exited, try to get logs anyway
print(f"Warning: docker logs returned exit code {result.returncode}")
logs = result.stdout if result.stdout else result.stderr
return logs
except Exception as e:
print(f"Warning: Failed to get container logs: {e}")
return ""
def stop_container(self, container_id_or_name: str):
"""Stops a running container."""
print(f"Stopping container: {container_id_or_name}")
self._run_docker_command(["stop", container_id_or_name])
print(f"Container {container_id_or_name} stopped.")
def remove_container(self, container_id_or_name: str):
"""Removes a container."""
print(f"Removing container: {container_id_or_name}")
self._run_docker_command(["rm", container_id_or_name])
print(f"Container {container_id_or_name} removed.")
def execute_command_in_container(
self, container_id_or_name: str, command: List[str]
) -> tuple[int, str, str]:
"""Executes a command inside a running container and returns exit code, stdout, and stderr."""
print(
f"Executing command '{' '.join(command)}' in container: {container_id_or_name}"
)
result = self._run_docker_command(
["exec", container_id_or_name] + command, check=False
)
stdout = result.stdout
stderr = result.stderr
print(
f"Command executed in {container_id_or_name}. Exit code: {result.returncode}"
)
return result.returncode, stdout, stderr
if __name__ == "__main__":
# Example Usage:
manager = DockerManager()
image_to_use = "mluis/qemu-esp32"
container_name = "test_esp32_qemu"
try:
print(f"--- Starting DockerManager test with image: {image_to_use} ---")
manager.pull_image(image_to_use)
# Example: Run a simple command in the container
print("\n--- Running a simple command ---")
container_id = manager.run_container(
image_name=image_to_use,
command=["echo", "Hello from Docker!"],
detach=True,
name=container_name,
)
# Wait a bit for the command to execute and logs to be generated
import time
time.sleep(2)
logs = manager.get_container_logs(container_id)
print("Container Logs:\n", logs)
# Check if the expected output is in the logs
if "Hello from Docker!" in logs:
print("SUCCESS: Expected output found in container logs.")
else:
print("FAILURE: Expected output NOT found in container logs.")
# Example: Execute a command inside a running container
print("\n--- Executing command inside container ---")
exit_code, stdout, stderr = manager.execute_command_in_container(
container_id, ["sh", "-c", "echo 'Executed inside!'; exit 42"]
)
print(f"Exec command stdout: {stdout}")
print(f"Exec command stderr: {stderr}")
print(f"Exec command exit code: {exit_code}")
if exit_code == 42 and "Executed inside!" in stdout:
print("SUCCESS: Command executed inside container as expected.")
else:
print("FAILURE: Command execution inside container failed.")
except subprocess.CalledProcessError as e:
print(f"Error during Docker operation: {e}", file=sys.stderr)
print(f"Stdout: {e.stdout}", file=sys.stderr)
print(f"Stderr: {e.stderr}", file=sys.stderr)
sys.exit(1)
except Exception as e:
print(f"An unexpected error occurred: {e}", file=sys.stderr)
sys.exit(1)
finally:
# Ensure cleanup
try:
manager.stop_container(container_name)
manager.remove_container(container_name)
print(f"Container {container_name} stopped and removed.")
except subprocess.CalledProcessError:
print(
f"Container {container_name} not found or already removed during cleanup."
)
except Exception as e:
print(f"Error during cleanup of {container_name}: {e}", file=sys.stderr)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/dockerfiles/Dockerfile.qemu-esp32
+106
View File
@@ -0,0 +1,106 @@
# Dockerfile for ESP32 QEMU emulation environment
# Based on ESP-IDF with QEMU support
FROM espressif/idf:v5.1.2
# Install additional dependencies for QEMU
RUN apt-get update && apt-get install -y \
qemu-system-misc \
qemu-utils \
python3-pip \
&& rm -rf /var/lib/apt/lists/*
# Install Python dependencies for ESP tools
RUN pip3 install --no-cache-dir \
esptool \
pyserial
# Create workspace directory
WORKDIR /workspace
# Set up environment for QEMU ESP32
ENV QEMU_ESP32_VERSION=esp-develop-8.2.0-20240122
# Download and install ESP32 QEMU if not using system QEMU
RUN mkdir -p /opt/qemu-esp32 && \
cd /opt/qemu-esp32 && \
wget -q https://github.com/espressif/qemu/releases/download/${QEMU_ESP32_VERSION}/qemu-xtensa-softmmu-esp_develop_8.2.0_20240122-x86_64-linux-gnu.tar.xz && \
tar -xf qemu-xtensa-softmmu-esp_develop_8.2.0_20240122-x86_64-linux-gnu.tar.xz && \
rm qemu-xtensa-softmmu-esp_develop_8.2.0_20240122-x86_64-linux-gnu.tar.xz && \
ln -sf /opt/qemu-esp32/qemu/bin/qemu-system-xtensa /usr/local/bin/qemu-system-xtensa
# Download ESP32 ROM files for QEMU
RUN mkdir -p /opt/esp-rom && \
cd /opt/esp-rom && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32-v3-rom.bin && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32-v3-rom-eco3.bin && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32c3-rom.bin && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32s2-rom.bin && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32s3-rom.bin
# Set environment variables for ROM files
ENV ESP_ROM_PATH=/opt/esp-rom
ENV ESP32_ROM=${ESP_ROM_PATH}/esp32-v3-rom.bin
ENV ESP32C3_ROM=${ESP_ROM_PATH}/esp32c3-rom.bin
ENV ESP32S2_ROM=${ESP_ROM_PATH}/esp32s2-rom.bin
ENV ESP32S3_ROM=${ESP_ROM_PATH}/esp32s3-rom.bin
# Create helper script for running QEMU with proper settings
RUN cat > /usr/local/bin/run-qemu-esp32 << 'EOF'
#!/bin/bash
# Helper script to run QEMU with ESP32 firmware
FIRMWARE_PATH=${1:-/workspace/firmware.bin}
MACHINE=${2:-esp32}
FLASH_SIZE=${3:-4}
# Select ROM file based on machine type
case "$MACHINE" in
esp32)
ROM_FILE=$ESP32_ROM
;;
esp32c3)
ROM_FILE=$ESP32C3_ROM
;;
esp32s2)
ROM_FILE=$ESP32S2_ROM
;;
esp32s3)
ROM_FILE=$ESP32S3_ROM
;;
*)
echo "Unknown machine type: $MACHINE"
exit 1
;;
esac
# Create flash image if needed
FLASH_IMAGE=/tmp/flash_${RANDOM}.bin
if [ -f "$FIRMWARE_PATH" ]; then
# Create empty flash image
dd if=/dev/zero of=$FLASH_IMAGE bs=1M count=$FLASH_SIZE 2>/dev/null
dd if=$FIRMWARE_PATH of=$FLASH_IMAGE bs=1 seek=$((0x10000)) conv=notrunc 2>/dev/null
else
echo "Firmware not found: $FIRMWARE_PATH"
exit 1
fi
# Run QEMU
exec qemu-system-xtensa \
-nographic \
-machine $MACHINE \
-bios $ROM_FILE \
-drive file=$FLASH_IMAGE,if=mtd,format=raw \
-global driver=timer.esp32.timg,property=wdt_disable,value=true \
"$@"
EOF
RUN chmod +x /usr/local/bin/run-qemu-esp32
# Default command
CMD ["/usr/local/bin/run-qemu-esp32"]
# Labels
LABEL maintainer="FastLED Team"
LABEL description="ESP32 QEMU emulation environment for FastLED testing"
LABEL version="1.0.0"
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/dockerfiles/Dockerfile.qemu-esp32-lite
+92
View File
@@ -0,0 +1,92 @@
# Lightweight Dockerfile for ESP32 QEMU emulation
# Uses pre-built QEMU binaries without full ESP-IDF
FROM ubuntu:22.04
# Install minimal dependencies
RUN apt-get update && apt-get install -y \
wget \
xz-utils \
python3 \
python3-pip \
&& rm -rf /var/lib/apt/lists/*
# Install esptool for flash image creation
RUN pip3 install --no-cache-dir esptool
# Create workspace
WORKDIR /workspace
# Download and install ESP32 QEMU binaries
ENV QEMU_VERSION=esp-develop-8.2.0-20240122
RUN mkdir -p /opt/qemu && \
cd /opt/qemu && \
wget -q https://github.com/espressif/qemu/releases/download/${QEMU_VERSION}/qemu-xtensa-softmmu-${QEMU_VERSION}-x86_64-linux-gnu.tar.xz && \
tar -xf qemu-xtensa-softmmu-${QEMU_VERSION}-x86_64-linux-gnu.tar.xz && \
rm qemu-xtensa-softmmu-${QEMU_VERSION}-x86_64-linux-gnu.tar.xz && \
mv qemu /usr/local/
# Add QEMU to PATH
ENV PATH=/usr/local/qemu/bin:$PATH
# Download ESP32 ROM files
RUN mkdir -p /opt/esp-rom && \
cd /opt/esp-rom && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32-v3-rom.bin && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32c3-rom.bin && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32s2-rom.bin && \
wget -q https://github.com/espressif/qemu/raw/esp-develop/pc-bios/esp32s3-rom.bin
# Set ROM path environment variable
ENV ESP_ROM_PATH=/opt/esp-rom
# Create entrypoint script
RUN cat > /entrypoint.sh << 'EOF'
#!/bin/bash
set -e
# Default values
FIRMWARE=${FIRMWARE:-/workspace/firmware.bin}
MACHINE=${MACHINE:-esp32}
FLASH_SIZE=${FLASH_SIZE:-4}
TIMEOUT=${TIMEOUT:-30}
# Select ROM based on machine type
case "$MACHINE" in
esp32) ROM_FILE=/opt/esp-rom/esp32-v3-rom.bin ;;
esp32c3) ROM_FILE=/opt/esp-rom/esp32c3-rom.bin ;;
esp32s2) ROM_FILE=/opt/esp-rom/esp32s2-rom.bin ;;
esp32s3) ROM_FILE=/opt/esp-rom/esp32s3-rom.bin ;;
*) echo "Unknown machine: $MACHINE"; exit 1 ;;
esac
# Check if firmware exists
if [ ! -f "$FIRMWARE" ]; then
echo "ERROR: Firmware not found: $FIRMWARE"
exit 1
fi
# Create flash image
FLASH_IMG=/tmp/flash.bin
dd if=/dev/zero of=$FLASH_IMG bs=1M count=$FLASH_SIZE 2>/dev/null
dd if=$FIRMWARE of=$FLASH_IMG bs=1 seek=$((0x10000)) conv=notrunc 2>/dev/null
# Run QEMU with timeout
timeout $TIMEOUT qemu-system-xtensa \
-nographic \
-machine $MACHINE \
-bios $ROM_FILE \
-drive file=$FLASH_IMG,if=mtd,format=raw \
-global driver=timer.esp32.timg,property=wdt_disable,value=true \
"$@"
EOF
RUN chmod +x /entrypoint.sh
# Set entrypoint
ENTRYPOINT ["/entrypoint.sh"]
# Labels
LABEL maintainer="FastLED Team"
LABEL description="Lightweight ESP32 QEMU Docker image"
LABEL version="1.0.0"
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/dockerfiles/README.md
+205
View File
@@ -0,0 +1,205 @@
# Docker-based QEMU ESP32 Testing
This directory contains Docker-based QEMU ESP32 emulation tools for FastLED testing.
## Overview
The Docker QEMU solution provides:
- Consistent testing environment across different platforms
- No need for local QEMU installation
- Isolated execution environment
- Support for multiple ESP32 variants (ESP32, ESP32-C3, ESP32-S2, ESP32-S3)
## Components
### Core Files
- `DockerManager.py` - Docker container management utilities
- `qemu_esp32_docker.py` - Main Docker QEMU runner script
- `qemu_test_integration.py` - Integration with existing test framework
- `Dockerfile.qemu-esp32` - Full ESP-IDF + QEMU environment
- `Dockerfile.qemu-esp32-lite` - Lightweight QEMU-only environment
- `docker-compose.yml` - Docker Compose configuration
## Usage
### Quick Start
1. **Run a firmware test with Docker:**
```bash
uv run ci/docker/qemu_esp32_docker.py path/to/firmware.bin
```
2. **Run with specific timeout and interrupt pattern:**
```bash
uv run ci/docker/qemu_esp32_docker.py firmware.bin \
--timeout 60 \
--interrupt-regex "Test passed"
```
3. **Use docker-compose:**
```bash
cd ci/docker
FIRMWARE_DIR=../../.pio/build/esp32dev docker-compose up qemu-esp32
```
### Integration with Test Framework
The Docker QEMU runner integrates with the existing test framework:
```bash
# Check available runners
uv run ci/docker/qemu_test_integration.py check
# Run test with automatic runner selection
uv run ci/docker/qemu_test_integration.py test --firmware path/to/firmware.bin
# Force Docker runner
uv run ci/docker/qemu_test_integration.py test --firmware firmware.bin --docker
```
### Building Docker Images
```bash
# Build lightweight image
docker build -f Dockerfile.qemu-esp32-lite -t fastled/qemu-esp32:lite .
# Build full ESP-IDF image
docker build -f Dockerfile.qemu-esp32 -t fastled/qemu-esp32:full .
# Or use docker-compose
docker-compose build
```
## Docker Images
### Pre-built Images
The runner can use pre-built images:
- `espressif/qemu:esp-develop-8.2.0-20240122` (Official Espressif image)
- `mluis/qemu-esp32:latest` (Community image)
### Custom Images
Two Dockerfiles are provided:
1. **Dockerfile.qemu-esp32-lite** (Recommended)
- Minimal Ubuntu base
- QEMU binaries only
- ~200MB image size
- Fast startup
2. **Dockerfile.qemu-esp32**
- Full ESP-IDF environment
- Development tools included
- ~2GB image size
- Complete toolchain
## Features
### Automatic Fallback
The integration module automatically selects the best available runner:
1. Native QEMU (if installed)
2. Docker QEMU (if Docker available)
3. Error if neither available
### Platform Support
Works on:
- Linux (native Docker)
- macOS (Docker Desktop)
- Windows (Docker Desktop, WSL2)
### Firmware Formats
Supports:
- Direct `firmware.bin` files
- PlatformIO build directories
- ESP-IDF build directories
## Environment Variables
- `FIRMWARE_DIR` - Directory containing firmware files
- `FIRMWARE_FILE` - Specific firmware file to run
- `MACHINE` - ESP32 variant (esp32, esp32c3, esp32s2, esp32s3)
- `FLASH_SIZE` - Flash size in MB (default: 4)
- `TIMEOUT` - Execution timeout in seconds (default: 30)
## Troubleshooting
### Docker Not Found
Install Docker:
- **Windows/Mac**: Install Docker Desktop
- **Linux**: `sudo apt-get install docker.io` or equivalent
### Permission Denied
Add user to docker group:
```bash
sudo usermod -aG docker $USER
# Log out and back in
```
### Image Pull Failed
Check internet connection and Docker Hub access. The runner will automatically try alternative images if the primary fails.
### QEMU Timeout
Increase timeout value:
```bash
uv run ci/docker/qemu_esp32_docker.py firmware.bin --timeout 120
```
## CI/CD Integration
### GitHub Actions
```yaml
- name: Run QEMU Test in Docker
run: |
uv run ci/docker/qemu_esp32_docker.py \
.pio/build/esp32dev/firmware.bin \
--timeout 60 \
--interrupt-regex "Setup complete"
```
### Local CI
```bash
# Install dependencies
uv pip install -r requirements.txt
# Run tests
uv run ci/docker/qemu_test_integration.py test \
--firmware .pio/build/esp32dev/firmware.bin
```
## Development
### Adding New ESP32 Variants
1. Update Dockerfiles with new ROM files
2. Add machine type to runner scripts
3. Test with sample firmware
### Debugging
Run interactively:
```bash
uv run ci/docker/qemu_esp32_docker.py firmware.bin --interactive
```
Enable verbose output:
```bash
docker-compose up qemu-esp32-interactive
```
## Performance
- Docker adds ~2-5 seconds overhead vs native QEMU
- First run pulls image (one-time ~100MB download)
- Subsequent runs use cached image
- Consider using lightweight image for CI/CD
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/dockerfiles/__init__.py
View File
Whitespace-only changes.
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/dockerfiles/docker-compose.yml
+61
View File
@@ -0,0 +1,61 @@
version: '3.8'
services:
qemu-esp32:
build:
context: .
dockerfile: Dockerfile.qemu-esp32-lite
image: fastled/qemu-esp32:latest
container_name: fastled-qemu-esp32
volumes:
# Mount firmware directory
- ${FIRMWARE_DIR:-./firmware}:/workspace:ro
# Mount output directory for logs
- ${OUTPUT_DIR:-./output}:/output
environment:
- FIRMWARE=${FIRMWARE_FILE:-/workspace/firmware.bin}
- MACHINE=${MACHINE:-esp32}
- FLASH_SIZE=${FLASH_SIZE:-4}
- TIMEOUT=${TIMEOUT:-30}
command: []
stdin_open: false
tty: false
networks:
- qemu-network
qemu-esp32-interactive:
build:
context: .
dockerfile: Dockerfile.qemu-esp32-lite
image: fastled/qemu-esp32:latest
container_name: fastled-qemu-esp32-interactive
volumes:
- ${FIRMWARE_DIR:-./firmware}:/workspace:ro
environment:
- FIRMWARE=${FIRMWARE_FILE:-/workspace/firmware.bin}
- MACHINE=${MACHINE:-esp32}
- FLASH_SIZE=${FLASH_SIZE:-4}
- TIMEOUT=${TIMEOUT:-300}
stdin_open: true
tty: true
networks:
- qemu-network
# Service for building the Docker image
builder:
build:
context: .
dockerfile: Dockerfile.qemu-esp32-lite
image: fastled/qemu-esp32:latest
command: echo "Image built successfully"
networks:
qemu-network:
driver: bridge
# Volume definitions (optional, for persistent storage)
volumes:
firmware:
driver: local
output:
driver: local
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/dockerfiles/qemu_esp32_docker.py
+502
View File
@@ -0,0 +1,502 @@
#!/usr/bin/env python3
"""
Docker-based QEMU ESP32 Runner
Runs ESP32 firmware in QEMU using Docker containers for better isolation
and consistency across different environments.
"""
import argparse
import os
import platform
import re
import shutil
import subprocess
import sys
import tempfile
import time
from pathlib import Path
from typing import Any, Dict, List, Optional, Union
from ci.dockerfiles.DockerManager import DockerManager
from ci.util.running_process import RunningProcess
def get_docker_env() -> Dict[str, str]:
"""Get environment for Docker commands, handling Git Bash/MSYS2 path conversion."""
env = os.environ.copy()
# Set UTF-8 encoding environment variables for Windows
env["PYTHONIOENCODING"] = "utf-8"
env["PYTHONUTF8"] = "1"
# Only set MSYS_NO_PATHCONV if we're in a Git Bash/MSYS2 environment
if (
"MSYSTEM" in os.environ
or os.environ.get("TERM") == "xterm"
or "bash.exe" in os.environ.get("SHELL", "")
):
env["MSYS_NO_PATHCONV"] = "1"
return env
def run_subprocess_safe(
cmd: List[str], **kwargs: Any
) -> subprocess.CompletedProcess[Any]:
"""Run subprocess with safe UTF-8 handling and error replacement."""
kwargs.setdefault("encoding", "utf-8")
kwargs.setdefault("errors", "replace")
kwargs.setdefault("env", get_docker_env())
return subprocess.run(cmd, **kwargs) # type: ignore[misc]
def run_docker_command_streaming(cmd: List[str]) -> int:
"""Run docker command with streaming output using RunningProcess."""
print(f"Executing: {' '.join(cmd)}")
proc = RunningProcess(cmd, env=get_docker_env(), auto_run=True)
# Stream all output to stdout
with proc.line_iter(timeout=None) as it:
for line in it:
print(line)
returncode = proc.wait()
if returncode != 0:
raise subprocess.CalledProcessError(returncode, cmd)
return returncode
def run_docker_command_no_fail(cmd: List[str]) -> int:
"""Run docker command with streaming output, but don't raise exceptions on failure."""
print(f"Executing: {' '.join(cmd)}")
proc = RunningProcess(cmd, env=get_docker_env(), auto_run=True)
# Stream all output to stdout
with proc.line_iter(timeout=None) as it:
for line in it:
print(line)
return proc.wait()
class DockerQEMURunner:
"""Runner for ESP32 QEMU emulation using Docker containers."""
DEFAULT_IMAGE = "mluis/qemu-esp32:latest"
ALTERNATIVE_IMAGE = "espressif/idf:latest"
FALLBACK_IMAGE = "espressif/idf:release-v5.2"
def __init__(self, docker_image: Optional[str] = None):
"""Initialize Docker QEMU runner.
Args:
docker_image: Docker image to use, defaults to espressif/qemu
"""
self.docker_manager = DockerManager()
self.docker_image = docker_image or self.DEFAULT_IMAGE
self.container_name = None
# Use Linux-style paths for all containers since we're using Ubuntu/Alpine
self.firmware_mount_path = "/workspace/firmware"
def check_docker_available(self) -> bool:
"""Check if Docker is available and running."""
try:
proc = RunningProcess(
["docker", "version"], env=get_docker_env(), auto_run=True
)
returncode = proc.wait(timeout=5)
return returncode == 0
except (Exception, FileNotFoundError):
return False
def pull_image(self):
"""Pull the Docker image if not already available."""
print(f"Ensuring Docker image {self.docker_image} is available...")
try:
# Check if image exists locally
proc = RunningProcess(
["docker", "images", "-q", self.docker_image],
env=get_docker_env(),
auto_run=True,
)
stdout_lines: List[str] = []
with proc.line_iter(timeout=None) as it:
for line in it:
stdout_lines.append(line)
result_stdout = "\n".join(stdout_lines)
if not result_stdout.strip():
# Image doesn't exist, pull it directly using docker command
print(f"Pulling Docker image: {self.docker_image}")
run_docker_command_streaming(["docker", "pull", self.docker_image])
print(f"Successfully pulled {self.docker_image}")
else:
print(f"Image {self.docker_image} already available locally")
except subprocess.CalledProcessError as e:
print(f"Warning: Failed to pull image {self.docker_image}: {e}")
if self.docker_image == self.DEFAULT_IMAGE:
print(f"Trying alternative image: {self.ALTERNATIVE_IMAGE}")
self.docker_image = self.ALTERNATIVE_IMAGE
try:
print(f"Pulling alternative Docker image: {self.docker_image}")
run_docker_command_streaming(["docker", "pull", self.docker_image])
print(f"Successfully pulled {self.docker_image}")
except subprocess.CalledProcessError as e2:
print(f"Failed to pull alternative image: {e2}")
print(f"Trying fallback image: {self.FALLBACK_IMAGE}")
self.docker_image = self.FALLBACK_IMAGE
try:
print(f"Pulling fallback Docker image: {self.docker_image}")
run_docker_command_streaming(
["docker", "pull", self.docker_image]
)
print(f"Successfully pulled {self.docker_image}")
except subprocess.CalledProcessError as e3:
print(f"Failed to pull fallback image: {e3}")
raise e3
else:
raise e
def prepare_firmware(self, firmware_path: Path) -> Path:
"""Prepare firmware files for mounting into Docker container.
Args:
firmware_path: Path to firmware.bin or build directory
Returns:
Path to the prepared firmware directory
"""
# Create temporary directory for firmware files
temp_dir = Path(tempfile.mkdtemp(prefix="qemu_firmware_"))
try:
if firmware_path.is_file() and firmware_path.suffix == ".bin":
# Copy single firmware file
shutil.copy2(firmware_path, temp_dir / "firmware.bin")
# Create proper 4MB flash image for QEMU
self._create_flash_image(firmware_path, temp_dir / "flash.bin")
else:
# Copy entire build directory
if firmware_path.is_dir():
# Copy relevant files from build directory
patterns = ["*.bin", "*.elf", "partitions.csv", "flash_args"]
for pattern in patterns:
for file in firmware_path.glob(pattern):
shutil.copy2(file, temp_dir)
# Also check for PlatformIO build structure
pio_build = firmware_path / ".pio" / "build" / "esp32dev"
if pio_build.exists():
for pattern in patterns:
for file in pio_build.glob(pattern):
shutil.copy2(file, temp_dir)
# Create proper flash.bin file from firmware.bin
firmware_bin = temp_dir / "firmware.bin"
if firmware_bin.exists():
# Create proper 4MB flash image for QEMU
self._create_flash_image(firmware_bin, temp_dir / "flash.bin")
else:
# Create a minimal 4MB flash.bin for testing
flash_bin = temp_dir / "flash.bin"
flash_bin.write_bytes(
b"\xff" * (4 * 1024 * 1024)
) # 4MB of 0xFF
else:
raise ValueError(f"Invalid firmware path: {firmware_path}")
return temp_dir
except Exception as e:
# Clean up on error
shutil.rmtree(temp_dir, ignore_errors=True)
raise e
def _create_flash_image(
self, firmware_path: Path, output_path: Path, flash_size_mb: int = 4
):
"""Create a proper flash image for QEMU ESP32.
Args:
firmware_path: Path to the firmware.bin file
output_path: Path where to write the flash.bin
flash_size_mb: Flash size in MB (must be 2, 4, 8, or 16)
"""
if flash_size_mb not in [2, 4, 8, 16]:
raise ValueError(
f"Flash size must be 2, 4, 8, or 16 MB, got {flash_size_mb}"
)
flash_size = flash_size_mb * 1024 * 1024
# Read firmware content
firmware_data = firmware_path.read_bytes()
# Create flash image: firmware at beginning, rest filled with 0xFF
flash_data = firmware_data + b"\xff" * (flash_size - len(firmware_data))
# Ensure we have exactly the right size
if len(flash_data) > flash_size:
raise ValueError(
f"Firmware size ({len(firmware_data)} bytes) exceeds flash size ({flash_size} bytes)"
)
flash_data = flash_data[:flash_size] # Truncate to exact size
output_path.write_bytes(flash_data)
def build_qemu_command(
self,
firmware_name: str = "firmware.bin",
flash_size: int = 4,
machine: str = "esp32",
) -> List[str]:
"""Build QEMU command to run inside Docker container.
Args:
firmware_name: Name of firmware file in mounted directory
flash_size: Flash size in MB
machine: QEMU machine type (esp32, esp32c3, esp32s3, etc.)
Returns:
List of command arguments for QEMU
"""
# For Linux containers, use a workaround to run QEMU through Docker Desktop's Windows integration
firmware_path = f"{self.firmware_mount_path}/flash.bin"
# Determine QEMU system and machine based on target
if machine == "esp32c3":
qemu_system = "/usr/local/bin/qemu-system-riscv32"
qemu_machine = "esp32c3"
echo_target = "ESP32C3"
elif machine == "esp32s3":
qemu_system = "/usr/local/bin/qemu-system-xtensa"
qemu_machine = "esp32s3"
echo_target = "ESP32S3"
else:
# Default to ESP32 (Xtensa)
qemu_system = "/usr/local/bin/qemu-system-xtensa"
qemu_machine = "esp32"
echo_target = "ESP32"
# Use container's QEMU with proper configuration
wrapper_script = f'''#!/bin/bash
set -e
echo "Starting {echo_target} QEMU emulation..."
echo "Firmware: {firmware_path}"
echo "Machine: {qemu_machine}"
echo "QEMU system: {qemu_system}"
echo "Container: $(cat /etc/os-release | head -1)"
# Check if firmware file exists
if [ ! -f "{firmware_path}" ]; then
echo "ERROR: Firmware file not found: {firmware_path}"
exit 1
fi
# Check firmware size
FIRMWARE_SIZE=$(stat -c%s "{firmware_path}")
echo "Firmware size: $FIRMWARE_SIZE bytes"
# Copy firmware to writable location since QEMU needs write access
cp "{firmware_path}" /tmp/flash.bin
echo "Copied firmware to writable location: /tmp/flash.bin"
# Try different QEMU configurations depending on machine type
if [ "{qemu_machine}" = "esp32c3" ]; then
# ESP32C3 uses RISC-V architecture
echo "Running {qemu_system} for {qemu_machine}"
{qemu_system} \\
-nographic \\
-machine {qemu_machine} \\
-drive file="/tmp/flash.bin",if=mtd,format=raw \\
-monitor none \\
-serial mon:stdio
else
# ESP32 uses Xtensa architecture
echo "Running {qemu_system} for {qemu_machine}"
{qemu_system} \\
-nographic \\
-machine {qemu_machine} \\
-drive file="/tmp/flash.bin",if=mtd,format=raw \\
-global driver=timer.esp32.timg,property=wdt_disable,value=true \\
-monitor none \\
-serial mon:stdio
fi
echo "QEMU execution completed"
exit 0
'''
cmd = ["bash", "-c", wrapper_script]
return cmd
def run(
self,
firmware_path: Path,
timeout: int = 30,
flash_size: int = 4,
interrupt_regex: Optional[str] = None,
interactive: bool = False,
output_file: Optional[str] = None,
machine: str = "esp32",
) -> int:
"""Run ESP32/ESP32C3/ESP32S3 firmware in QEMU using Docker.
Args:
firmware_path: Path to firmware.bin or build directory
timeout: Timeout in seconds (timeout is treated as success)
flash_size: Flash size in MB
interrupt_regex: Regex pattern to detect in output (informational only)
interactive: Run in interactive mode (attach to container)
output_file: Optional file path to write QEMU output to
machine: QEMU machine type (esp32, esp32c3, esp32s3, etc.)
Returns:
Exit code: actual QEMU/container exit code, except timeout returns 0
"""
if not self.check_docker_available():
print("ERROR: Docker is not available or not running", file=sys.stderr)
print("Please install Docker and ensure it's running", file=sys.stderr)
return 1
# Pull image if needed
self.pull_image()
# Prepare firmware files
print(f"Preparing firmware from: {firmware_path}")
temp_firmware_dir = None
try:
temp_firmware_dir = self.prepare_firmware(firmware_path)
# Generate unique container name
self.container_name = f"qemu_esp32_{int(time.time())}"
# Convert Windows paths to Docker-compatible paths
def windows_to_docker_path(path_str: Union[str, Path]) -> str:
"""Convert Windows path to Docker volume mount format."""
import os
# Check if we're in Git Bash/MSYS2 environment
is_git_bash = (
"MSYSTEM" in os.environ
or os.environ.get("TERM") == "xterm"
or "bash.exe" in os.environ.get("SHELL", "")
)
if os.name == "nt" and is_git_bash:
# Convert C:\path\to\dir to /c/path/to/dir for Git Bash
path_str = str(path_str).replace("\\", "/")
if len(path_str) > 2 and path_str[1:3] == ":/": # Drive letter
path_str = "/" + path_str[0].lower() + path_str[2:]
else:
# For cmd.exe on Windows, keep native Windows paths
path_str = str(path_str)
return path_str
docker_firmware_path = windows_to_docker_path(temp_firmware_dir)
# Only mount the firmware directory - QEMU is built into the container
volumes = {
docker_firmware_path: {
"bind": self.firmware_mount_path,
"mode": "ro", # Read-only mount
},
}
# Build QEMU command
qemu_cmd = self.build_qemu_command(
firmware_name="firmware.bin", flash_size=flash_size, machine=machine
)
print(f"Running QEMU in Docker container: {self.container_name}")
print(f"QEMU command: {' '.join(qemu_cmd)}")
if interactive:
# Run interactively with streaming
return self.docker_manager.run_container_streaming(
image_name=self.docker_image,
command=qemu_cmd,
volumes=volumes,
name=self.container_name,
timeout=timeout,
interrupt_pattern=interrupt_regex,
output_file=output_file,
)
else:
# Run with streaming output and return actual exit code
return self.docker_manager.run_container_streaming(
image_name=self.docker_image,
command=qemu_cmd,
volumes=volumes,
name=self.container_name,
timeout=timeout,
interrupt_pattern=interrupt_regex,
output_file=output_file,
)
except Exception as e:
print(f"ERROR: {e}", file=sys.stderr)
return 1
finally:
# Cleanup
if temp_firmware_dir and temp_firmware_dir.exists():
shutil.rmtree(temp_firmware_dir, ignore_errors=True)
# Ensure container is stopped and removed
if self.container_name:
try:
# Stop and remove container
run_docker_command_no_fail(["docker", "stop", self.container_name])
run_docker_command_no_fail(["docker", "rm", self.container_name])
except:
pass
def main():
parser = argparse.ArgumentParser(
description="Run ESP32 firmware in QEMU using Docker"
)
parser.add_argument(
"firmware_path", type=Path, help="Path to firmware.bin file or build directory"
)
parser.add_argument(
"--docker-image",
type=str,
help=f"Docker image to use (default: {DockerQEMURunner.DEFAULT_IMAGE})",
)
parser.add_argument(
"--flash-size", type=int, default=4, help="Flash size in MB (default: 4)"
)
parser.add_argument(
"--timeout", type=int, default=30, help="Timeout in seconds (default: 30)"
)
parser.add_argument(
"--interrupt-regex", type=str, help="Regex pattern to interrupt execution"
)
parser.add_argument(
"--interactive", action="store_true", help="Run in interactive mode"
)
parser.add_argument("--output-file", type=str, help="File to write QEMU output to")
args = parser.parse_args()
if not args.firmware_path.exists():
print(
f"ERROR: Firmware path does not exist: {args.firmware_path}",
file=sys.stderr,
)
return 1
runner = DockerQEMURunner(docker_image=args.docker_image)
return runner.run(
firmware_path=args.firmware_path,
timeout=args.timeout,
flash_size=args.flash_size,
interrupt_regex=args.interrupt_regex,
interactive=args.interactive,
output_file=args.output_file,
)
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/dockerfiles/qemu_test_integration.py
+283
View File
@@ -0,0 +1,283 @@
#!/usr/bin/env python3
"""
Integration module for Docker-based QEMU ESP32 testing.
This module provides a bridge between the existing test infrastructure
and the Docker-based QEMU runner.
"""
import os
import platform
import shutil
import subprocess
import sys
from pathlib import Path
from typing import Optional, Union
from ci.dockerfiles.qemu_esp32_docker import DockerQEMURunner
class QEMUTestIntegration:
"""Integration class for QEMU testing with Docker fallback."""
def __init__(self, prefer_docker: bool = False):
"""Initialize QEMU test integration.
Args:
prefer_docker: If True, prefer Docker even if native QEMU is available
"""
self.prefer_docker = prefer_docker
self.docker_available = self._check_docker()
self.native_qemu_available = self._check_native_qemu()
def _check_docker(self) -> bool:
"""Check if Docker is available."""
try:
result = subprocess.run(
["docker", "version"], capture_output=True, timeout=5
)
return result.returncode == 0
except (subprocess.SubprocessError, FileNotFoundError):
return False
def _check_native_qemu(self) -> bool:
"""Check if native QEMU ESP32 is available."""
try:
# Import the native QEMU module to check
from ci.qemu_esp32 import find_qemu_binary # type: ignore[import-untyped]
return find_qemu_binary() is not None
except ImportError:
return False
def select_runner(self) -> str:
"""Select the best available runner.
Returns:
'docker' or 'native' based on availability and preference
"""
if self.prefer_docker and self.docker_available:
return "docker"
elif self.native_qemu_available:
return "native"
elif self.docker_available:
return "docker"
else:
return "none"
def run_qemu_test(
self,
firmware_path: Union[str, Path],
timeout: int = 30,
interrupt_regex: Optional[str] = None,
flash_size: int = 4,
force_runner: Optional[str] = None,
) -> int:
"""Run QEMU test with automatic runner selection.
Args:
firmware_path: Path to firmware or build directory
timeout: Test timeout in seconds
interrupt_regex: Pattern to interrupt on success
flash_size: Flash size in MB
force_runner: Force specific runner ('docker' or 'native')
Returns:
Exit code (0 for success)
"""
firmware_path = Path(firmware_path)
# Determine which runner to use
if force_runner:
runner_type = force_runner
else:
runner_type = self.select_runner()
print(f"Selected QEMU runner: {runner_type}")
if runner_type == "docker":
return self._run_docker_qemu(
firmware_path, timeout, interrupt_regex, flash_size
)
elif runner_type == "native":
return self._run_native_qemu(
firmware_path, timeout, interrupt_regex, flash_size
)
else:
print("ERROR: No QEMU runner available!", file=sys.stderr)
print("Install Docker or native QEMU ESP32", file=sys.stderr)
return 1
def _run_docker_qemu(
self,
firmware_path: Path,
timeout: int,
interrupt_regex: Optional[str],
flash_size: int,
) -> int:
"""Run QEMU test using Docker."""
print("Running QEMU test in Docker container...")
runner = DockerQEMURunner()
return runner.run(
firmware_path=firmware_path,
timeout=timeout,
interrupt_regex=interrupt_regex,
flash_size=flash_size,
)
def _run_native_qemu(
self,
firmware_path: Path,
timeout: int,
interrupt_regex: Optional[str],
flash_size: int,
) -> int:
"""Run QEMU test using native installation."""
print("Running QEMU test with native installation...")
# Import and use the native runner
from ci.qemu_esp32 import QEMURunner # type: ignore[import-untyped]
runner = QEMURunner() # type: ignore[no-untyped-call]
return runner.run( # type: ignore[no-untyped-call]
firmware_path=firmware_path,
timeout=timeout,
interrupt_regex=interrupt_regex,
flash_size=flash_size,
)
def install_qemu(self, use_docker: bool = False) -> bool:
"""Install QEMU (native or pull Docker image).
Args:
use_docker: If True, pull Docker image instead of native install
Returns:
True if installation successful
"""
if use_docker:
print("Pulling Docker QEMU image...")
try:
runner = DockerQEMURunner()
runner.pull_image()
return True
except Exception as e:
print(f"Failed to pull Docker image: {e}", file=sys.stderr)
return False
else:
print("Installing native QEMU...")
try:
# Run the native install script
result = subprocess.run(
[sys.executable, "ci/install-qemu.py"],
cwd=Path(__file__).parent.parent.parent,
)
return result.returncode == 0
except Exception as e:
print(f"Failed to install native QEMU: {e}", file=sys.stderr)
return False
def integrate_with_test_framework():
"""Integrate Docker QEMU with the existing test framework.
This function modifies the test.py to add Docker support.
"""
test_file = Path(__file__).parent.parent.parent / "test.py"
if not test_file.exists():
print(f"ERROR: test.py not found at {test_file}", file=sys.stderr)
return False
print(f"Integrating Docker QEMU support into {test_file}")
# Read the test.py file
content = test_file.read_text()
# Check if already integrated
if "docker.qemu_test_integration" in content:
print("Docker QEMU support already integrated")
return True
# Find the QEMU test section and add Docker support
integration_code = """
# Docker QEMU integration
try:
from ci.dockerfiles.qemu_test_integration import QEMUTestIntegration
DOCKER_QEMU_AVAILABLE = True
except ImportError:
DOCKER_QEMU_AVAILABLE = False
"""
# Add the import at the top of the file after other imports
import_marker = "from ci.qemu_esp32 import"
if import_marker in content:
# Add after the existing QEMU import
content = content.replace(
import_marker, integration_code + "\n" + import_marker
)
# Save the modified file
test_file.write_text(content)
print("Successfully integrated Docker QEMU support")
return True
else:
print("WARNING: Could not find appropriate location to add integration")
return False
def main():
"""Main function for testing the integration."""
import argparse
parser = argparse.ArgumentParser(
description="QEMU Test Integration with Docker support"
)
parser.add_argument(
"command",
choices=["test", "install", "integrate", "check"],
help="Command to execute",
)
parser.add_argument("--firmware", type=Path, help="Firmware path for test command")
parser.add_argument(
"--docker", action="store_true", help="Prefer Docker over native QEMU"
)
parser.add_argument(
"--timeout", type=int, default=30, help="Test timeout in seconds"
)
args = parser.parse_args()
integration = QEMUTestIntegration(prefer_docker=args.docker)
if args.command == "check":
print(f"Docker available: {integration.docker_available}")
print(f"Native QEMU available: {integration.native_qemu_available}")
print(f"Selected runner: {integration.select_runner()}")
return 0
elif args.command == "install":
success = integration.install_qemu(use_docker=args.docker)
return 0 if success else 1
elif args.command == "integrate":
success = integrate_with_test_framework()
return 0 if success else 1
elif args.command == "test":
if not args.firmware:
print("ERROR: --firmware required for test command", file=sys.stderr)
return 1
return integration.run_qemu_test(
firmware_path=args.firmware,
timeout=args.timeout,
force_runner="docker" if args.docker else None,
)
return 0
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/docs.py
+186
View File
@@ -0,0 +1,186 @@
"""
Work in progress to generate doxygen via a script instead of a GitHub action.
"""
import os
import platform
import shutil
import subprocess
import warnings
from pathlib import Path
from typing import Optional, Tuple
from download import download # type: ignore
# Configs
DOXYGEN_VERSION = (
"1.13.2" # DOXYGEN_AWESOME styler is has certain restrictions with doxygen version
)
DOXYGEN_AWESOME_VERSION = "2.3.4" # deprecating
DOXYFILE_PATH = Path("docs/Doxyfile")
HTML_OUTPUT_DIR = Path("docs/html")
DOXYGEN_CSS_REPO = "https://github.com/jothepro/doxygen-awesome-css" # deprecating
HERE = Path(__file__).parent.resolve()
PROJECT_ROOT = HERE.parent
DOCS_ROOT = PROJECT_ROOT / "docs"
DOCS_TOOL_PATH = PROJECT_ROOT / ".tools_cache"
DOCS_OUTPUT_PATH = DOCS_ROOT / "html"
def run(
cmd: str,
cwd: Optional[str] = None,
shell: bool = True,
check: bool = True,
capture: bool = True,
) -> str:
print(f"Running: {cmd}")
result = subprocess.run(
cmd, shell=shell, cwd=cwd, check=False, capture_output=capture, text=False
)
if capture:
stdout = result.stdout.decode("utf-8") if result.stdout else ""
stderr = result.stderr.decode("utf-8") if result.stderr else ""
else:
stdout = ""
stderr = ""
if result.returncode != 0:
msg = f"Command failed with exit code {result.returncode}:\nstdout:\n{stdout}\n\nstderr:\n{stderr}"
warnings.warn(msg)
if check:
raise subprocess.CalledProcessError(
result.returncode, cmd, output=result.stdout
)
return stdout.strip()
def get_git_info() -> Tuple[str, str]:
release_tag = os.environ.get("RELEASE_TAG", "")
try:
latest_tag = run("git tag | grep -E '^[0-9]' | sort -V | tail -1")
latest_tag = latest_tag if latest_tag else ""
except subprocess.CalledProcessError:
latest_tag = ""
git_sha_short = run("git rev-parse --short HEAD")
full_sha = run("git rev-parse HEAD")
project_number = release_tag or latest_tag or git_sha_short
commit_message = (
f"{project_number} ({full_sha})"
if project_number != git_sha_short
else project_number
)
print(f"Project number: {project_number}")
print(f"Commit message: {commit_message}")
return project_number, commit_message
def install_doxygen_windows() -> Path:
print("Installing Doxygen...")
doxygen_url = (
f"https://www.doxygen.nl/files/doxygen-{DOXYGEN_VERSION}.windows.x64.bin.zip"
)
zip_path = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}.zip"
extract_dir = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}"
# Create tool path if it doesn't exist
DOCS_TOOL_PATH.mkdir(exist_ok=True, parents=True)
download(doxygen_url, zip_path)
shutil.unpack_archive(str(zip_path), extract_dir)
bin_path = next(extract_dir.glob("**/doxygen.exe"), None)
if not bin_path:
raise FileNotFoundError("Doxygen executable not found after extraction.")
print(f"Doxygen installed at: {bin_path}")
return bin_path
def install_doxygen_unix() -> Path:
print("Installing Doxygen...")
archive = f"doxygen-{DOXYGEN_VERSION}.linux.bin.tar.gz"
url = f"https://www.doxygen.nl/files/{archive}"
# Create tool path if it doesn't exist
DOCS_TOOL_PATH.mkdir(exist_ok=True, parents=True)
# Change to tool directory for download and extraction
original_dir = os.getcwd()
os.chdir(str(DOCS_TOOL_PATH))
try:
run(f"wget -q {url}")
run(f"tar -xf {archive}")
bin_dir = DOCS_TOOL_PATH / f"doxygen-{DOXYGEN_VERSION}"
return bin_dir / "bin" / "doxygen"
finally:
os.chdir(original_dir)
def install_theme() -> Path:
print("Installing Doxygen Awesome Theme...")
theme_path = DOCS_ROOT / "doxygen-awesome-css"
if theme_path.exists():
return theme_path
run(
f"git clone --depth 1 -b v{DOXYGEN_AWESOME_VERSION} {DOXYGEN_CSS_REPO}",
cwd=str(DOCS_ROOT),
)
return theme_path
def generate_docs(doxygen_bin: Path) -> None:
print("Generating documentation...")
cmd_str = f'"{doxygen_bin}" {DOXYFILE_PATH.name}'
run(cmd_str, cwd=str(DOCS_ROOT), capture=False)
# def install_graphviz() -> None:
# url: str = get_latest_release_for_platform()
# print(url)
def main() -> None:
is_windows = platform.system() == "Windows"
# is_macos = platform.system() == "Darwin"
_, commit_msg = get_git_info()
if is_windows:
doxygen_bin = install_doxygen_windows()
# add to path C:\Program Files\Graphviz\bin\
os.environ["PATH"] += os.pathsep + r"C:\Program Files\Graphviz\bin"
else:
doxygen_bin = install_doxygen_unix()
# install_theme()
# install_graphviz() # Work in progress
# Verify Graphviz installation
try:
dot_version = run("dot -V", check=False)
print(f"Graphviz detected: {dot_version}")
except Exception:
warnings.warn(
"Graphviz (dot) not found in PATH. Diagrams may not be generated."
)
# Check it graphviz is installed
# if linux
if not is_windows:
run("dot -Tsvg -Kneato -Grankdir=LR", check=True)
generate_docs(doxygen_bin=doxygen_bin)
print(f"\n✅ Docs generated in: {HTML_OUTPUT_DIR}")
print(f"📄 Commit message: {commit_msg}")
print("✨ You can now manually deploy to GitHub Pages or automate this step.")
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/esp32_symbol_analysis_runner.py
+100
View File
@@ -0,0 +1,100 @@
#!/usr/bin/env python3
"""
ESP32 Symbol Analysis Runner for GitHub Actions
Wrapper script to run ESP32 symbol analysis with proper error handling
"""
import argparse
import os
import sys
from pathlib import Path
from ci.util.esp32_symbol_analysis import main as esp32_symbol_analysis
def is_esp32_board(board_name: str) -> bool:
"""Check if the board is ESP32-based"""
esp32_boards = [
"esp32dev",
"esp32",
"esp32s2",
"esp32s3",
"esp32c3",
"esp32c6",
"esp32h2",
"esp32p4",
"esp32c2",
]
return any(esp32_board in board_name.lower() for esp32_board in esp32_boards)
def main():
parser = argparse.ArgumentParser(
description="Run ESP32 symbol analysis for GitHub Actions"
)
parser.add_argument("--board", required=True, help="Board name being analyzed")
parser.add_argument("--example", default="Blink", help="Example that was compiled")
parser.add_argument(
"--skip-on-failure",
action="store_true",
help="Don't fail the build if symbol analysis fails",
)
args = parser.parse_args()
print(
f"ESP32 Symbol Analysis Runner - Board: {args.board}, Example: {args.example}"
)
# Only run for ESP32 boards
if not is_esp32_board(args.board):
print(f"Skipping symbol analysis - {args.board} is not an ESP32-based board")
return 0
# Check if build_info.json exists
build_info_path = Path(".build") / args.board / "build_info.json"
if not build_info_path.exists():
message = (
f"Build info not found at {build_info_path}. Skipping symbol analysis."
)
if args.skip_on_failure:
print(f"Warning: {message}")
return 0
else:
print(f"Error: {message}")
return 1
try:
# TODO - Remove this.
# Change to the ci/util directory since the script expects to be run from there
original_cwd = os.getcwd()
ci_ci_dir = Path(__file__).parent / "ci"
os.chdir(ci_ci_dir)
# Import and run the ESP32 symbol analysis
try:
print("Running ESP32 symbol analysis...")
esp32_symbol_analysis()
print("ESP32 symbol analysis completed successfully!")
return 0
except ImportError as e:
print(f"Failed to import esp32_symbol_analysis module: {e}")
if args.skip_on_failure:
return 0
return 1
except Exception as e:
print(f"ESP32 symbol analysis failed: {e}")
if args.skip_on_failure:
return 0
return 1
finally:
os.chdir(original_cwd)
except Exception as e:
print(f"Unexpected error in ESP32 symbol analysis runner: {e}")
if args.skip_on_failure:
return 0
return 1
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/examples/fingerprint_cache_demo.py
+232
View File
@@ -0,0 +1,232 @@
#!/usr/bin/env python3
"""
Demonstration of the Fingerprint Cache Feature
This script shows how to use the fingerprint cache for efficient file change detection
in build systems and development workflows.
"""
import os
import tempfile
import time
from pathlib import Path
from typing import List
from ci.ci.fingerprint_cache import FingerprintCache, has_changed
def demo_basic_usage() -> None:
"""Demonstrate basic fingerprint cache usage."""
print("=== Basic Fingerprint Cache Demo ===")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
cache_file = temp_path / "demo_cache.json"
cache = FingerprintCache(cache_file)
# Create a source file
source_file = temp_path / "example.cpp"
with open(source_file, "w") as f:
f.write("#include <iostream>\nint main() { return 0; }")
baseline_time = time.time() - 3600 # 1 hour ago
current_modtime = os.path.getmtime(source_file)
print(f"Source file: {source_file}")
print(f"Baseline time: {baseline_time}")
print(f"Current modtime: {current_modtime}")
# First check - file is newer than baseline
changed = cache.has_changed(source_file, baseline_time)
print(f"First check (vs baseline): {changed} (expected: True)")
# Second check - same modtime
changed = cache.has_changed(source_file, current_modtime)
print(f"Second check (same modtime): {changed} (expected: False)")
# Touch file (change modtime but not content)
time.sleep(0.01)
source_file.touch()
# Third check - modtime changed but content same
changed = cache.has_changed(source_file, current_modtime)
print(
f"Third check (touched file): {changed} (expected: False - content unchanged)"
)
# Actually modify content
time.sleep(0.01)
with open(source_file, "w") as f:
f.write(
'#include <iostream>\nint main() { std::cout << "Hello"; return 0; }'
)
# Fourth check - content actually changed
changed = cache.has_changed(source_file, current_modtime)
print(f"Fourth check (content changed): {changed} (expected: True)")
def demo_build_system_workflow() -> None:
"""Demonstrate build system integration workflow."""
print("\n=== Build System Workflow Demo ===")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
cache_file = temp_path / "build_cache.json"
cache = FingerprintCache(cache_file)
# Simulate multiple source files
source_files = [
temp_path / "main.cpp",
temp_path / "utils.cpp",
temp_path / "config.h",
]
# Create source files
contents = [
'#include "config.h"\n#include "utils.h"\nint main() { return 0; }',
'#include "utils.h"\nvoid utility_function() {}',
'#pragma once\n#define VERSION "1.0"',
]
for src_file, content in zip(source_files, contents):
with open(src_file, "w") as f:
f.write(content)
print(f"Created {len(source_files)} source files")
# Simulate first build
print("\n--- First Build (all files new) ---")
last_build_time = time.time() - 3600 # 1 hour ago
changed_files: List[str] = []
for src_file in source_files:
if cache.has_changed(src_file, last_build_time):
changed_files.append(src_file.name)
print(
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
)
# Update last build time
current_modtimes = [os.path.getmtime(f) for f in source_files]
# Simulate second build (no changes)
print("\n--- Second Build (no changes) ---")
changed_files: List[str] = []
for src_file, modtime in zip(source_files, current_modtimes):
if cache.has_changed(src_file, modtime):
changed_files.append(src_file.name)
print(
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
)
# Modify one file
print("\n--- Third Build (one file modified) ---")
time.sleep(0.01)
with open(source_files[0], "a") as f:
f.write("\n// Added comment")
changed_files: List[str] = []
for src_file, modtime in zip(source_files, current_modtimes):
if cache.has_changed(src_file, modtime):
changed_files.append(src_file.name)
print(
f"Files to rebuild: {changed_files} ({len(changed_files)}/{len(source_files)})"
)
def demo_performance() -> None:
"""Demonstrate cache performance characteristics."""
print("\n=== Performance Demo ===")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
cache_file = temp_path / "perf_cache.json"
cache = FingerprintCache(cache_file)
# Create multiple test files
num_files = 20
test_files: List[Path] = []
for i in range(num_files):
test_file = temp_path / f"file_{i:03d}.cpp"
with open(test_file, "w") as f:
f.write(
f"// File {i}\n#include <iostream>\nint func_{i}() {{ return {i}; }}"
)
test_files.append(test_file)
print(f"Created {num_files} test files")
# Measure cache miss performance (first check)
start_time = time.time()
baseline_time = time.time() - 3600
for test_file in test_files:
cache.has_changed(test_file, baseline_time)
miss_time = time.time() - start_time
print(
f"Cache miss time: {miss_time * 1000:.1f}ms ({miss_time * 1000 / num_files:.2f}ms per file)"
)
# Measure cache hit performance (second check with same modtime)
modtimes = [os.path.getmtime(f) for f in test_files]
start_time = time.time()
for test_file, modtime in zip(test_files, modtimes):
cache.has_changed(test_file, modtime)
hit_time = time.time() - start_time
print(
f"Cache hit time: {hit_time * 1000:.1f}ms ({hit_time * 1000 / num_files:.2f}ms per file)"
)
print(f"Speedup: {miss_time / hit_time:.1f}x faster")
# Show cache statistics
stats = cache.get_cache_stats()
print(f"Cache entries: {stats['total_entries']}")
print(f"Cache file size: {stats['cache_file_size_bytes']} bytes")
def demo_convenience_function() -> None:
"""Demonstrate the convenience has_changed function."""
print("\n=== Convenience Function Demo ===")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
cache_file = temp_path / "convenience_cache.json"
# Create test file
test_file = temp_path / "test.h"
with open(test_file, "w") as f:
f.write("#pragma once\n#define API_VERSION 1")
baseline_time = time.time() - 1800 # 30 minutes ago
# Use convenience function
changed = has_changed(test_file, baseline_time, cache_file)
print(f"Convenience function result: {changed}")
print(f"Cache file created: {cache_file.exists()}")
if __name__ == "__main__":
print("Fingerprint Cache Feature Demonstration")
print("=" * 50)
demo_basic_usage()
demo_build_system_workflow()
demo_performance()
demo_convenience_function()
print("\n" + "=" * 50)
print("Demo completed successfully!")
print("\nKey Benefits:")
print("- Fast modification time checks avoid expensive MD5 computation")
print("- Content-based verification prevents false positives")
print("- Persistent cache survives across tool invocations")
print("- Significant speedup for incremental builds and testing")
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/inspect_binary.py
+205
View File
@@ -0,0 +1,205 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
import argparse
import json
import re
import subprocess
import sys
from pathlib import Path
from tempfile import TemporaryDirectory
from typing import Any, Dict
from ci.util.bin_2_elf import bin_to_elf
from ci.util.elf import dump_symbol_sizes
from ci.util.map_dump import map_dump
def cpp_filt(cpp_filt_path: Path, input_text: str) -> str:
"""
Demangle C++ symbols using c++filt.
Args:
cpp_filt_path (Path): Path to c++filt executable.
input_text (str): Text to demangle.
Returns:
str: Demangled text.
"""
if not cpp_filt_path.exists():
raise FileNotFoundError(f"cppfilt not found at '{cpp_filt_path}'")
command = [str(cpp_filt_path), "-t", "-n"]
process = subprocess.Popen(
command,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
text=True,
)
stdout, stderr = process.communicate(input=input_text)
if process.returncode != 0:
raise RuntimeError(f"Error running c++filt: {stderr}")
return stdout
def demangle_gnu_linkonce_symbols(cpp_filt_path: Path, map_text: str) -> str:
"""
Demangle .gnu.linkonce.t symbols in the map file.
Args:
cpp_filt_path (Path): Path to c++filt executable.
map_text (str): Content of the map file.
Returns:
str: Map file content with demangled symbols.
"""
# Extract all .gnu.linkonce.t symbols
pattern = r"\.gnu\\.linkonce\\.t\\.(.+?)\\s"
matches = re.findall(pattern, map_text)
if not matches:
return map_text
# Create a block of text with the extracted symbols
symbols_block = "\n".join(matches)
# Demangle the symbols
demangled_block = cpp_filt(cpp_filt_path, symbols_block)
# Create a dictionary of mangled to demangled symbols
demangled_dict = dict(zip(matches, demangled_block.strip().split("\n")))
# Replace the mangled symbols with demangled ones in the original text
for mangled, demangled in demangled_dict.items():
map_text = map_text.replace(
f".gnu.linkonce.t.{mangled}", f".gnu.linkonce.t.{demangled}"
)
return map_text
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Convert a binary file to ELF using map file."
)
parser.add_argument("--first", action="store_true", help="Inspect the first board")
parser.add_argument("--cwd", type=Path, help="Custom working directory")
return parser.parse_args()
def load_build_info(build_info_path: Path) -> Dict[str, Any]:
"""
Load build information from a JSON file.
Args:
build_info_path (Path): Path to the build_info.json file.
Returns:
dict: Parsed JSON data.
"""
if not build_info_path.exists():
raise FileNotFoundError(f"Build info JSON not found at '{build_info_path}'")
return json.loads(build_info_path.read_text())
def main() -> int:
args = parse_args()
if args.cwd:
root_build_dir = args.cwd / ".build"
else:
root_build_dir = Path(".build")
# Support nested PlatformIO structure: .build/pio/<board>
nested_pio_dir = root_build_dir / "pio"
if nested_pio_dir.is_dir():
root_build_dir = nested_pio_dir
board_dirs = [d for d in root_build_dir.iterdir() if d.is_dir()]
if not board_dirs:
print(f"No board directories found in {root_build_dir.absolute()}")
return 1
print("Available boards:")
for i, board_dir in enumerate(board_dirs):
print(f"[{i}]: {board_dir.name}")
which = (
0
if args.first
else int(input("Enter the number of the board you want to inspect: "))
)
board_dir = board_dirs[which]
build_info_json = board_dir / "build_info.json"
build_info = load_build_info(build_info_json)
board = board_dir.name
board_info = build_info.get(board) or build_info[next(iter(build_info))]
# Validate paths from build_info.json
elf_path = Path(board_info.get("prog_path", ""))
if not elf_path.exists():
print(
f"Error: ELF path '{elf_path}' does not exist. Check the 'prog_path' in build_info.json."
)
return 1
bin_file = elf_path.with_suffix(".bin")
if not bin_file.exists():
# use .hex or .uf2 if .bin doesn't exist
bin_file = elf_path.with_suffix(".hex")
if not bin_file.exists():
bin_file = elf_path.with_suffix(".uf2")
if not bin_file.exists():
print(f"Error: Binary file not found for '{elf_path}'")
return 1
cpp_filt_path = Path(board_info["aliases"]["c++filt"])
ld_path = Path(board_info["aliases"]["ld"])
as_path = Path(board_info["aliases"]["as"])
nm_path = Path(board_info["aliases"]["nm"])
objcopy_path = Path(board_info["aliases"]["objcopy"])
nm_path = Path(board_info["aliases"]["nm"])
map_file = board_dir / "firmware.map"
if not map_file.exists():
# Search for the map file
map_file = bin_file.with_suffix(".map")
if not map_file.exists():
possible_map_files = list(board_dir.glob("**/firmware.map"))
if possible_map_files:
map_file = possible_map_files[0]
else:
print("Error: firmware.map file not found")
return 1
try:
with TemporaryDirectory() as temp_dir:
temp_dir_path = Path(temp_dir)
output_elf = bin_to_elf(
bin_file,
map_file,
as_path,
ld_path,
objcopy_path,
temp_dir_path / "output.elf",
)
out = dump_symbol_sizes(nm_path, cpp_filt_path, output_elf)
print(out)
except Exception as e:
print(
f"Error while converting binary to ELF, binary analysis will not work on this build: {e}"
)
map_dump(map_file)
# Demangle .gnu.linkonce.t symbols and print map file
print("\n##################################################")
print("# Map file dump:")
print("##################################################\n")
map_text = map_file.read_text()
demangled_map_text = demangle_gnu_linkonce_symbols(cpp_filt_path, map_text)
print(demangled_map_text)
return 0
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/inspect_elf.py
+78
View File
@@ -0,0 +1,78 @@
import argparse
import json
from pathlib import Path
from ci.util.elf import dump_symbol_sizes
HERE = Path(__file__).resolve().parent
PROJECT_ROOT = HERE.parent
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(description="Inspect a compiled binary")
parser.add_argument("--first", action="store_true", help="Inspect the first board")
parser.add_argument("--cwd", type=str, help="Custom working directory")
parser.add_argument("--elf", type=str, help="Path to the ELF file to inspect")
return parser.parse_args()
def main() -> int:
args = parse_args()
if args.elf:
firmware_path = Path(args.elf)
if not firmware_path.exists():
print(f"ELF file not found: {firmware_path}")
return 1
if args.cwd:
# os.chdir(args.cwd)
root_build_dir = Path(args.cwd) / ".build"
else:
root_build_dir = Path(".build")
# Support nested PlatformIO structure: .build/pio/<board>
nested_pio_dir = root_build_dir / "pio"
if nested_pio_dir.is_dir():
root_build_dir = nested_pio_dir
# Find the first board directory
board_dirs = [d for d in root_build_dir.iterdir() if d.is_dir()]
if not board_dirs:
# print("No board directories found in .build")
print(f"No board directories found in {root_build_dir.absolute()}")
return 1
# display all the boards to the user and ask them to select which one they want by number
print("Available boards:")
for i, board_dir in enumerate(board_dirs):
print(f"[{i}]: {board_dir.name}")
if args.first:
which = 0
else:
which = int(input("Enter the number of the board you want to inspect: "))
board_dir = board_dirs[which]
board = board_dir.name
build_info_json = board_dir / "build_info.json"
build_info = json.loads(build_info_json.read_text())
board_info = build_info.get(board) or build_info[next(iter(build_info))]
firmware_path = Path(board_info["prog_path"])
cpp_filt_path = Path(board_info["aliases"]["c++filt"])
print(f"Dumping symbol sizes for {board} firmware: {firmware_path}")
try:
nm_path = Path(board_info["aliases"]["nm"])
symbol_sizes = dump_symbol_sizes(nm_path, cpp_filt_path, firmware_path)
print(symbol_sizes)
except Exception as e:
print(f"Error while dumping symbol sizes: {e}")
return 0
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/inspect_obj.py
+124
View File
@@ -0,0 +1,124 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
import os
import subprocess
import sys
from pathlib import Path
from typing import List
from ci.util.paths import BUILD
from ci.util.tools import load_tools
def _list_builds() -> List[Path]:
str_paths = os.listdir(BUILD)
paths = [BUILD / p for p in str_paths]
dirs = [p for p in paths if p.is_dir()]
return dirs
def _check_build(build: Path) -> bool:
# 1. should contain a build_info.json file
# 2. should contain a .pio/build directory
has_build_info = (build / "build_info.json").exists()
has_pio_build = (build / ".pio" / "build").exists()
return has_build_info and has_pio_build
def _prompt_build() -> Path:
builds = _list_builds()
if not builds:
print("Error: No builds found", file=sys.stderr)
sys.exit(1)
print("Select a build:")
for i, build in enumerate(builds):
print(f" [{i}]: {build}")
while True:
try:
which = int(input("Enter the number of the build to use: "))
if 0 <= which < len(builds):
valid = _check_build(BUILD / builds[which])
if valid:
return BUILD / builds[which]
print("Error: Invalid build", file=sys.stderr)
else:
print("Error: Invalid selection", file=sys.stderr)
continue
except ValueError:
print("Error: Invalid input", file=sys.stderr)
continue
def _prompt_object_file(build: Path) -> Path:
# Look for object files in .pio/build directory
build_dir = build / ".pio" / "build"
object_files: List[Path] = list(build_dir.rglob("*.o"))
if not object_files:
print("Error: No object files found", file=sys.stderr)
sys.exit(1)
print("\nSelect an object file:")
for i, obj_file in enumerate(object_files):
print(f" [{i}]: {obj_file.relative_to(build_dir)}")
while True:
try:
which = int(input("Enter the number of the object file to use: "))
if 0 <= which < len(object_files):
return object_files[which]
print("Error: Invalid selection", file=sys.stderr)
except ValueError:
print("Error: Invalid input", file=sys.stderr)
continue
def cli() -> None:
import argparse
parser = argparse.ArgumentParser(
description="Dump object file information using build tools"
)
parser.add_argument(
"build_path",
type=Path,
nargs="?",
help="Path to build directory containing build info JSON file",
)
args = parser.parse_args()
build_path = args.build_path
# Check if object file was provided and exists
if build_path is None:
build_path = _prompt_build()
else:
if not _check_build(build_path):
print("Error: Invalid build directory", file=sys.stderr)
sys.exit(1)
assert build_path is not None
assert build_path
build_info_path = build_path / "build_info.json"
assert build_info_path.exists(), f"File not found: {build_info_path}"
tools = load_tools(build_info_path)
object_file = _prompt_object_file(build_path)
cmd = [str(tools.objdump_path), "--syms", str(object_file)]
if sys.platform == "win32":
cmd = ["cmd", "/c"] + cmd
cmd_str = subprocess.list2cmdline(cmd)
subprocess.run(cmd, check=True)
print("\nDone. Command used:", cmd_str)
if __name__ == "__main__":
try:
cli()
except KeyboardInterrupt:
print("Exiting...")
sys.exit(1)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/install-qemu.py
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env python3
"""
Standalone QEMU Installation Script
This is a convenience wrapper around install_qemu_esp32.py
that can be called directly to install QEMU for ESP32 emulation.
Usage:
uv run ci/install-qemu.py
"""
import os
import sys
from install_qemu_esp32 import main as install_qemu_main
def main():
"""Run the QEMU installation script."""
print("=== FastLED QEMU Installation ===")
print("Starting QEMU installation process...")
print(f"Python executable: {sys.executable}")
print(f"Working directory: {os.getcwd()}")
print()
try:
# Call the main function directly
print("Calling install_qemu_esp32.main()...")
install_qemu_main()
print("install_qemu_esp32.main() completed successfully")
except KeyboardInterrupt:
print("\nInstallation cancelled by user")
sys.exit(130)
except SystemExit as e:
print(f"SystemExit caught with code: {e.code}")
# Re-raise SystemExit to preserve exit codes
raise
except Exception as e:
print(f"ERROR: Installation failed: {e}")
import traceback
print("Full traceback:")
traceback.print_exc()
sys.exit(1)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/install_qemu_esp32.py
+732
View File
@@ -0,0 +1,732 @@
#!/usr/bin/env python3
"""
ESP32 QEMU Installation Script
Installs ESP32-compatible QEMU emulator using ESP-IDF tools.
Cross-platform support for Linux, macOS, and Windows.
WARNING: Never use sys.stdout.flush() in this file!
It causes blocking issues on Windows that hang QEMU processes.
Python's default buffering behavior works correctly across platforms.
"""
import hashlib
import os
import platform
import shutil
import subprocess
import sys
import tarfile
import tempfile
import threading
from pathlib import Path
from typing import Dict, List, Optional, Union
from download import download # type: ignore
from ci.util.running_process import RunningProcess
def get_platform_info():
"""Get platform-specific information for QEMU installation."""
print("IMMEDIATE DEBUG: get_platform_info() called")
print("Detecting platform information...")
system = platform.system().lower()
print(f"IMMEDIATE DEBUG: platform.system() = {system}")
machine = platform.machine().lower()
print(f"IMMEDIATE DEBUG: platform.machine() = {machine}")
print(f"System: {system}, Machine: {machine}")
# Normalize machine architecture
if machine in ["x86_64", "amd64"]:
arch = "x86_64"
elif machine in ["aarch64", "arm64"]:
arch = "arm64"
elif machine.startswith("arm"):
arch = "arm"
else:
arch = machine
return system, arch
def install_system_dependencies():
"""Install system dependencies required for QEMU."""
system, _ = get_platform_info()
print("Installing system dependencies...")
if system == "linux":
# Ubuntu/Debian dependencies
deps = [
"libgcrypt20",
"libglib2.0-0",
"libpixman-1-0",
"libsdl2-2.0-0",
"libslirp0",
]
try:
# Check if we can install packages (requires sudo)
print("Updating package list...")
process = RunningProcess(["sudo", "apt-get", "update"])
result = process.wait(echo=True)
if result == 0:
print("Installing system dependencies:", " ".join(deps))
process = RunningProcess(["sudo", "apt-get", "install", "-y"] + deps)
result = process.wait(echo=True)
if result != 0:
raise subprocess.CalledProcessError(
result if result is not None else 1,
"apt-get install",
)
print("System dependencies installed")
else:
print("WARNING: Could not install system dependencies (sudo required)")
print("Please install manually:", " ".join(deps))
except subprocess.CalledProcessError:
print("WARNING: Failed to install system dependencies")
elif system == "darwin":
# macOS - dependencies usually available via system
print("macOS system dependencies should be available")
elif system == "windows":
# Windows - QEMU binaries are self-contained
print("Windows - no additional dependencies required")
def download_esp_idf_tools():
"""Download and setup ESP-IDF tools for QEMU installation."""
tools_dir = Path.home() / ".espressif"
tools_dir.mkdir(exist_ok=True)
# ESP-IDF tools script URL
tools_url = (
"https://raw.githubusercontent.com/espressif/esp-idf/master/tools/idf_tools.py"
)
tools_script = tools_dir / "idf_tools.py"
if not tools_script.exists():
print("Downloading ESP-IDF tools...")
print(f"Source: {tools_url}")
print(f"Destination: {tools_script}")
download(tools_url, tools_script)
print(f"Downloaded ESP-IDF tools to {tools_script}")
else:
print(f"ESP-IDF tools already exist at {tools_script}")
return tools_script
def check_command_available(command: str) -> bool:
"""Check if a command is available in PATH."""
print(f"Checking if {command} is available...")
try:
result = subprocess.run(
[command, "--version"], capture_output=True, text=True, timeout=10
)
available = result.returncode == 0
print(f"{command} {'found' if available else 'not found'}")
return available
except (subprocess.TimeoutExpired, FileNotFoundError, subprocess.SubprocessError):
print(f"{command} not found or timed out")
return False
def verify_file_hash(file_path: Path, expected_hash: str) -> bool:
"""Verify SHA256 hash of a downloaded file."""
print(f"Verifying SHA256 hash of {file_path}...")
try:
with open(file_path, "rb") as f:
file_hash = hashlib.sha256(f.read()).hexdigest()
if file_hash == expected_hash:
print(f"SHA256 verification passed: {file_hash}")
return True
else:
print(f"SHA256 verification failed!")
print(f"Expected: {expected_hash}")
print(f"Got: {file_hash}")
return False
except Exception as e:
print(f"Error verifying hash: {e}")
return False
def try_espressif_binary_install():
"""Download and install QEMU using Espressif official precompiled binaries."""
try:
print("\n--- Espressif Official Binary Installation ---")
print("Downloading QEMU from Espressif official releases...")
# Create local QEMU directory (project-local)
qemu_dir = Path(".cache") / "qemu"
qemu_dir.mkdir(parents=True, exist_ok=True)
# Check if already installed
qemu_xtensa_binary = qemu_dir / "qemu-system-xtensa.exe"
qemu_riscv32_binary = qemu_dir / "qemu-system-riscv32.exe"
print(f"Checking for existing QEMU at {qemu_xtensa_binary}...")
if qemu_xtensa_binary.exists() and qemu_riscv32_binary.exists():
print(
f"QEMU already installed at {qemu_xtensa_binary} and {qemu_riscv32_binary}"
)
return True
else:
print("QEMU not found, proceeding with download...")
# Espressif official QEMU binaries for Windows x64
binaries = [
{
"name": "QEMU-Xtensa (ESP32/ESP32-S2/ESP32-S3)",
"url": "https://github.com/espressif/qemu/releases/download/esp-develop-9.2.2-20250817/qemu-xtensa-softmmu-esp_develop_9.2.2_20250817-x86_64-w64-mingw32.tar.xz",
"sha256": "ef550b912726997f3c1ff4a4fb13c1569e2b692efdc5c9f9c3c926a8f7c540fa",
"binary_name": "qemu-system-xtensa.exe",
},
{
"name": "QEMU-RISCV32 (ESP32-C3/ESP32-H2/ESP32-C2)",
"url": "https://github.com/espressif/qemu/releases/download/esp-develop-9.2.2-20250817/qemu-riscv32-softmmu-esp_develop_9.2.2_20250817-x86_64-w64-mingw32.tar.xz",
"sha256": "9474015f24d27acb7516955ec932e5307226bd9d6652cdc870793ed36010ab73",
"binary_name": "qemu-system-riscv32.exe",
},
]
for binary_info in binaries:
print(f"\nDownloading {binary_info['name']}...")
print("This may take a few minutes depending on your internet connection.")
archive_path = qemu_dir / f"{binary_info['binary_name']}.tar.xz"
print(f"Downloading from: {binary_info['url']}")
print(f"Saving to: {archive_path}")
try:
# Download the archive
download(binary_info["url"], archive_path)
print(f"Download completed: {archive_path}")
# Verify SHA256 hash
if not verify_file_hash(archive_path, binary_info["sha256"]):
print(f"Hash verification failed for {binary_info['name']}")
archive_path.unlink() # Remove corrupted file
return False
# Extract the archive
print(f"Extracting {binary_info['name']}...")
with tarfile.open(archive_path, "r:xz") as tar:
# List contents to understand structure
members = tar.getnames()
print(f"Archive contains {len(members)} files")
# Extract to temporary directory first
temp_extract_dir = qemu_dir / "temp_extract"
temp_extract_dir.mkdir(exist_ok=True)
tar.extractall(temp_extract_dir)
print(f"Extracted to temporary directory: {temp_extract_dir}")
# Find the binary in the extracted contents
found_binary = None
for root, dirs, files in os.walk(temp_extract_dir):
for file in files:
if file == binary_info["binary_name"]:
found_binary = Path(root) / file
break
if found_binary:
break
if found_binary:
target_binary = qemu_dir / binary_info["binary_name"]
print(
f"Copying {binary_info['binary_name']} to {target_binary}"
)
shutil.copy2(found_binary, target_binary)
# Make sure it's executable (though Windows doesn't use execute bit)
target_binary.chmod(0o755)
print(f"Successfully installed {binary_info['binary_name']}")
else:
print(
f"ERROR: Could not find {binary_info['binary_name']} in extracted archive"
)
return False
# Clean up temporary extraction directory
shutil.rmtree(temp_extract_dir)
# Clean up downloaded archive
archive_path.unlink()
print(f"Cleaned up {archive_path}")
except Exception as e:
print(f"Error downloading/extracting {binary_info['name']}: {e}")
# Clean up on error
if archive_path.exists():
archive_path.unlink()
return False
print("Espressif binary installation completed successfully")
return True
except Exception as e:
print(f"Espressif binary installation error: {e}")
return False
def try_direct_download_install():
"""Download and install QEMU directly from the official source."""
print("\n--- Legacy Installer Method (Requires Admin) ---")
print("WARNING: This method requires administrator privileges and may fail")
print("Consider using the Espressif binary method instead")
try:
print("Downloading QEMU directly from official source...")
# Create local QEMU directory (project-local)
qemu_dir = Path(".cache") / "qemu"
qemu_dir.mkdir(parents=True, exist_ok=True)
# Check if already installed
qemu_binary = qemu_dir / "qemu-system-xtensa.exe"
print(f"Checking for existing QEMU at {qemu_binary}...")
if qemu_binary.exists():
print(f"QEMU already installed at {qemu_binary}")
return True
else:
print("QEMU not found, proceeding with download...")
# Download QEMU from official source
print("Downloading QEMU for Windows...")
print("This may take several minutes depending on your internet connection.")
# QEMU download URL from the official Windows builds
qemu_url = "https://qemu.weilnetz.de/w64/2024/qemu-w64-setup-20241220.exe"
installer_path = qemu_dir / "qemu-installer.exe"
print(f"Downloading from: {qemu_url}")
print(f"Saving to: {installer_path}")
try:
# Download the installer
download(qemu_url, installer_path)
print(f"Download completed: {installer_path}")
# Run the installer in silent mode
print("Running QEMU installer in silent mode...")
print("Note: This may require administrator privileges")
# Try different installation approaches
install_attempts = [
# Attempt 1: Install to custom directory (may require admin)
[
str(installer_path),
"/VERYSILENT", # Very silent install
"/SUPPRESSMSGBOXES", # Suppress message boxes
f"/DIR={qemu_dir.absolute()}", # Install directory
],
# Attempt 2: Install to default location (may require admin)
[
str(installer_path),
"/VERYSILENT", # Very silent install
"/SUPPRESSMSGBOXES", # Suppress message boxes
],
# Attempt 3: Use basic silent install
[
str(installer_path),
"/S", # Silent install
],
]
result = None
process = None
for i, install_cmd in enumerate(install_attempts, 1):
print(
f"Installation attempt {i}/{len(install_attempts)}: {' '.join(install_cmd)}"
)
try:
process = RunningProcess(
install_cmd,
timeout=300, # 5 minute timeout
)
result = process.wait(echo=True)
if result == 0:
print(f"Installation attempt {i} succeeded")
break
else:
print(
f"Installation attempt {i} failed with exit code: {result}"
)
if i < len(install_attempts):
print("Trying next installation method...")
except Exception as e:
print(f"Installation attempt {i} failed with error: {e}")
if i < len(install_attempts):
print("Trying next installation method...")
if result == 0:
print("QEMU installer completed successfully")
# Look for the installed qemu-system-xtensa.exe
for qemu_path in qemu_dir.rglob("qemu-system-xtensa.exe"):
if qemu_path.is_file():
print(f"Found QEMU binary at: {qemu_path}")
# If it's not in the expected location, copy it there
if qemu_path != qemu_binary:
print(
f"Copying QEMU binary to standard location: {qemu_binary}"
)
shutil.copy2(qemu_path, qemu_binary)
# Clean up installer
if installer_path.exists():
installer_path.unlink()
print("Direct download installation completed successfully")
return True
print("QEMU installer completed but qemu-system-xtensa.exe not found")
else:
print(f"QEMU installer failed with exit code: {result}")
if process:
stderr_output = process.stdout
if stderr_output:
print(f"Error output: {stderr_output[:200]}")
except Exception as download_error:
print(f"Download/install error: {download_error}")
return False
except Exception as e:
print(f"Direct download installation error: {e}")
return False
def try_windows_installation():
"""Try installing QEMU using Espressif binaries for Windows."""
print("Attempting automatic QEMU installation on Windows...")
# Try Espressif official binaries first (recommended method)
print("Using Espressif official binaries...")
if try_espressif_binary_install():
print("Espressif binary installation completed, verifying...")
# Verify installation worked
if find_qemu_binary():
print("SUCCESS: QEMU installed via Espressif official binaries")
return True
else:
print(
"WARNING: Espressif binary installation reported success but QEMU binary not found"
)
else:
print("Espressif binary installation failed")
# Fall back to legacy installer method (requires admin privileges)
print("Falling back to legacy installer method...")
if try_direct_download_install():
print("Legacy installer completed, verifying...")
# Verify installation worked
if find_qemu_binary():
print("SUCCESS: QEMU installed via legacy installer")
return True
else:
print(
"WARNING: Legacy installer reported success but QEMU binary not found"
)
else:
print("Legacy installer failed")
print("Windows QEMU installation failed")
return False
def install_qemu_esp32():
"""Install ESP32 QEMU using ESP-IDF tools."""
print("\n=== Starting ESP32 QEMU Installation ===")
system, arch = get_platform_info()
print(f"\nChecking for ESP32 QEMU for {system}-{arch}...")
# Check if already installed
print("Searching for existing QEMU installation...")
qemu_binary = find_qemu_binary()
print(f"IMMEDIATE DEBUG: find_qemu_binary returned: {qemu_binary}")
if qemu_binary:
print("ESP32 QEMU already installed")
return True
else:
print("No existing QEMU installation found")
# Try automated installation
print("Attempting automatic installation...")
if system == "linux":
# Check if we're in CI first for sudo operations
if os.getenv("CI") == "true" or os.getenv("GITHUB_ACTIONS") == "true":
try:
# Try to install QEMU from package manager in CI
print("Installing QEMU via apt-get...")
print("Updating package list...")
process = RunningProcess(["sudo", "apt-get", "update"])
result = process.wait(echo=True)
if result == 0:
print("Package list updated successfully")
print("Installing QEMU packages...")
print("Installing: qemu-system-misc qemu-system-xtensa")
process = RunningProcess(
[
"sudo",
"apt-get",
"install",
"-y",
"qemu-system-misc",
"qemu-system-xtensa",
]
)
result = process.wait(echo=True)
if result == 0:
print("QEMU packages installed successfully via apt-get")
return True
else:
print("Failed to install QEMU packages via apt-get")
print("Return code:", result)
except Exception as e:
print(f"Automatic installation failed: {e}")
else:
print(
"Local Linux environment - automatic installation requires sudo privileges"
)
elif system == "windows":
print("Windows platform detected, using direct download...")
# Try direct download installation (works in both CI and local environments)
if try_windows_installation():
print("Windows QEMU installation successful")
return True
else:
print("Windows installation failed")
elif system == "darwin":
# Try macOS installation methods
print("macOS platform detected")
print("Attempting macOS installation...")
print("Note: Homebrew installation could be added here in the future")
# Could add homebrew installation here in the future
# If not found, provide installation instructions
print("ESP32 QEMU not found. Please install using one of these methods:")
print()
if system == "windows":
print("Windows Installation Options:")
print()
print("Option 1: Espressif Official Binaries (Recommended)")
print(
" This is what the automatic installer uses - no administrator privileges required"
)
print(
" Espressif QEMU Xtensa: https://github.com/espressif/qemu/releases/download/esp-develop-9.2.2-20250817/qemu-xtensa-softmmu-esp_develop_9.2.2_20250817-x86_64-w64-mingw32.tar.xz"
)
print(
" Espressif QEMU RISC-V32: https://github.com/espressif/qemu/releases/download/esp-develop-9.2.2-20250817/qemu-riscv32-softmmu-esp_develop_9.2.2_20250817-x86_64-w64-mingw32.tar.xz"
)
print(" Extract to .cache/qemu/ directory")
print()
print("Option 2: Official QEMU Installer (Requires Administrator)")
print(
" Download QEMU from: https://qemu.weilnetz.de/w64/2024/qemu-w64-setup-20241220.exe"
)
print(" Run the installer and ensure qemu-system-xtensa.exe is in your PATH")
print()
print("Option 3: ESP-IDF (includes QEMU)")
print(
" Download ESP-IDF from: https://docs.espressif.com/projects/esp-idf/en/latest/esp32/get-started/windows-setup.html"
)
print()
print("Option 4: Manual Installation")
print(" Download QEMU from: https://www.qemu.org/download/")
print(" Make sure qemu-system-xtensa.exe is in your PATH")
elif system == "linux":
print("Option 1: Install via ESP-IDF")
print(" git clone --recursive https://github.com/espressif/esp-idf.git")
print(" cd esp-idf && ./install.sh esp32s3")
print(" source ./export.sh")
print()
print("Option 2: Install QEMU from package manager")
print(" sudo apt-get install qemu-system-misc qemu-system-xtensa")
elif system == "darwin":
print("Option 1: Install via ESP-IDF")
print(" git clone --recursive https://github.com/espressif/esp-idf.git")
print(" cd esp-idf && ./install.sh esp32s3")
print(" source ./export.sh")
print()
print("Option 2: Install via Homebrew")
print(" brew install qemu")
print()
print("For CI/GitHub Actions, QEMU will be cached automatically once installed.")
return False
def find_qemu_binary() -> Optional[Path]:
"""Find the installed QEMU binary."""
print("IMMEDIATE DEBUG: find_qemu_binary() called")
system, _ = get_platform_info()
print(f"IMMEDIATE DEBUG: got system: {system}")
# Look for both Xtensa and RISC-V32 binaries
binary_names = []
if system == "windows":
binary_names = ["qemu-system-xtensa.exe", "qemu-system-riscv32.exe"]
else:
binary_names = ["qemu-system-xtensa", "qemu-system-riscv32"]
print(f"IMMEDIATE DEBUG: looking for binaries: {binary_names}")
# ESP-IDF installation paths and portable installation
esp_paths = [
Path(".cache") / "qemu", # Project-local portable installation directory
Path.home()
/ ".cache"
/ "qemu", # Legacy user cache location (for backward compatibility)
Path.home() / ".espressif" / "tools" / "qemu-xtensa",
Path.home() / ".espressif" / "python_env",
Path("/opt/espressif/tools/qemu-xtensa"),
]
# Search in ESP-IDF paths first
for base_path in esp_paths:
print(f"Searching in: {base_path}")
try:
if base_path.exists():
print(f" Directory exists, searching for QEMU binaries...")
for binary_name in binary_names:
print(
f" Starting recursive search for {binary_name} in {base_path}..."
)
for qemu_path in base_path.rglob(binary_name):
print(f" Checking: {qemu_path}")
if qemu_path.is_file():
print(f"Found QEMU binary at: {qemu_path}")
return qemu_path
print(f" No QEMU binaries found in {base_path}")
else:
print(f" Directory does not exist")
except Exception as e:
print(f" Error searching {base_path}: {e}")
# Check system PATH for each binary
print("Checking system PATH...")
for binary_name in binary_names:
try:
cmd = ["where" if system == "windows" else "which", binary_name]
print(f"Running: {' '.join(cmd)}")
print("About to call subprocess.run...")
result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
print("subprocess.run completed")
if result.returncode == 0:
# Take first line in case of multiple results (Windows)
first_line = result.stdout.strip().split("\n")[0]
qemu_path = Path(first_line)
print(f"Found QEMU binary in PATH: {qemu_path}")
return qemu_path
else:
print(f"Command returned code {result.returncode}")
except subprocess.TimeoutExpired:
print(f"PATH check for {binary_name} timed out after 10 seconds")
except (subprocess.SubprocessError, FileNotFoundError) as e:
print(f"Error checking PATH for {binary_name}: {e}")
print("ERROR: Could not find any QEMU binary")
return None
def create_qemu_wrapper():
"""Create a wrapper script for easy QEMU access."""
# DISABLED: This function was overwriting the actual qemu-esp32.py runner script
# The qemu-esp32.py script already exists and handles all the necessary
# functionality for running ESP32 firmware in QEMU with proper argument parsing
print("Skipping wrapper creation - using existing qemu-esp32.py runner script")
return True
def main():
"""Main installation routine."""
print("IMMEDIATE DEBUG: main() function called")
print("=== ESP32 QEMU Installation ===")
print("Step 1/5: Starting installation process...")
# Show platform info
print("Step 2/5: Detecting platform...")
system, arch = get_platform_info()
print(f"IMMEDIATE DEBUG: get_platform_info completed, got: {system}, {arch}")
print(f"Platform: {system}-{arch}")
# Install system dependencies
print("Step 3/5: Installing system dependencies...")
install_system_dependencies()
print("IMMEDIATE DEBUG: install_system_dependencies completed")
print("System dependencies step completed")
# Install ESP32 QEMU
print("Step 4/5: Installing ESP32 QEMU...")
qemu_installed = install_qemu_esp32()
print(
f"IMMEDIATE DEBUG: install_qemu_esp32 completed with result: {qemu_installed}"
)
print(f"ESP32 QEMU installation step completed (success: {qemu_installed})")
if qemu_installed:
# Create wrapper script
print("Step 5/5: Creating wrapper scripts...")
if not create_qemu_wrapper():
print("WARNING: Could not create wrapper script, but QEMU is available")
else:
print("QEMU installation required for ESP32 emulation")
sys.exit(1)
print("=== Installation Complete ===")
print("ESP32 QEMU is ready to use!")
# Test installation
print("\n=== Verifying Installation ===")
print("Searching for QEMU binary...")
qemu_binary = find_qemu_binary()
if qemu_binary:
print(f"Testing QEMU binary: {qemu_binary}")
try:
print("Running version check...")
result = subprocess.run(
[str(qemu_binary), "--version"],
capture_output=True,
text=True,
timeout=10,
)
if result.returncode == 0:
print(f"QEMU version: {result.stdout.strip()}")
print("Installation verification successful!")
sys.exit(0) # Explicit success exit
else:
print("WARNING: QEMU installed but version check failed")
print(f"Version check return code: {result.returncode}")
sys.exit(1)
except Exception as e:
print(f"WARNING: Could not verify QEMU installation: {e}")
sys.exit(1)
else:
print("ERROR: QEMU installation verification failed")
sys.exit(1)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/iwyu/fastled.imp
+57
View File
@@ -0,0 +1,57 @@
# FastLED Include What You Use Mapping File
# This file tells IWYU about FastLED's specific include patterns and structure
[
# FastLED main headers - these are the public API
{ include: ['"FastLED.h"', 'public', '"FastLED.h"', 'public'] },
{ include: ['"fastled.h"', 'public', '"FastLED.h"', 'public'] },
# Core FastLED headers that should be included directly
{ include: ['"lib8tion.h"', 'public', '"lib8tion.h"', 'public'] },
{ include: ['"pixeltypes.h"', 'public', '"pixeltypes.h"', 'public'] },
{ include: ['"hsv2rgb.h"', 'public', '"hsv2rgb.h"', 'public'] },
{ include: ['"colorutils.h"', 'public', '"colorutils.h"', 'public'] },
{ include: ['"colorpalettes.h"', 'public', '"colorpalettes.h"', 'public'] },
# Platform-specific headers should not be suggested directly
{ include: ['"platforms/*/fastled_*.h"', 'private', '"FastLED.h"', 'public'] },
{ include: ['"platforms/*/clockless_*.h"', 'private', '"FastLED.h"', 'public'] },
{ include: ['"platforms/*/*"', 'private', '"FastLED.h"', 'public'] },
# Internal headers should redirect to public API
{ include: ['"controller.h"', 'private', '"FastLED.h"', 'public'] },
{ include: ['"led_sysdefs.h"', 'private', '"FastLED.h"', 'public'] },
{ include: ['"fastled_config.h"', 'private', '"FastLED.h"', 'public'] },
{ include: ['"fastled_delay.h"', 'private', '"FastLED.h"', 'public'] },
{ include: ['"fastled_progmem.h"', 'private', '"FastLED.h"', 'public'] },
# fl/ namespace headers - the modern FastLED API
{ include: ['"fl/namespace.h"', 'public', '"fl/namespace.h"', 'public'] },
{ include: ['"fl/ptr.h"', 'public', '"fl/ptr.h"', 'public'] },
{ include: ['"fl/vector.h"', 'public', '"fl/vector.h"', 'public'] },
{ include: ['"fl/allocator.h"', 'public', '"fl/allocator.h"', 'public'] },
{ include: ['"fl/math.h"', 'public', '"fl/math.h"', 'public'] },
{ include: ['"fl/force_inline.h"', 'public', '"fl/force_inline.h"', 'public'] },
# FX headers - effects system
{ include: ['"fx/fx.h"', 'public', '"fx/fx.h"', 'public'] },
{ include: ['"fx/fx_engine.h"', 'public', '"fx/fx_engine.h"', 'public'] },
{ include: ['"fx/frame.h"', 'public', '"fx/frame.h"', 'public'] },
{ include: ['"fx/detail/*"', 'private', '"fx/fx.h"', 'public'] },
# Third-party headers should not be suggested for user code
{ include: ['"third_party/*/*"', 'private', '"FastLED.h"', 'public'] },
# Arduino-specific headers
{ include: ['<Arduino.h>', 'public', '<Arduino.h>', 'public'] },
{ include: ['<avr/pgmspace.h>', 'public', '<avr/pgmspace.h>', 'public'] },
# Chipset-specific headers redirect to main header
{ include: ['"chipsets.h"', 'private', '"FastLED.h"', 'public'] },
{ include: ['"ws2812_controller.h"', 'private', '"FastLED.h"', 'public'] },
{ include: ['"apa102_controller.h"', 'private', '"FastLED.h"', 'public'] },
# Testing headers - keep as-is for test files
{ include: ['"test_*.h"', 'public', '"test_*.h"', 'public'] },
{ include: ['"doctest.h"', 'public', '"doctest.h"', 'public'] },
]
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/iwyu/stdlib.imp
+67
View File
@@ -0,0 +1,67 @@
# Standard Library Include What You Use Mapping File
# This file provides standard library header mappings for better IWYU suggestions
[
# C++ standard library headers
{ include: ['<cstdint>', 'public', '<cstdint>', 'public'] },
{ include: ['<cstddef>', 'public', '<cstddef>', 'public'] },
{ include: ['<cstring>', 'public', '<cstring>', 'public'] },
{ include: ['<cstdlib>', 'public', '<cstdlib>', 'public'] },
{ include: ['<cmath>', 'public', '<cmath>', 'public'] },
{ include: ['<cassert>', 'public', '<cassert>', 'public'] },
{ include: ['<climits>', 'public', '<climits>', 'public'] },
{ include: ['<cfloat>', 'public', '<cfloat>', 'public'] },
# C++ containers and algorithms
{ include: ['<vector>', 'public', '<vector>', 'public'] },
{ include: ['<array>', 'public', '<array>', 'public'] },
{ include: ['<string>', 'public', '<string>', 'public'] },
{ include: ['<algorithm>', 'public', '<algorithm>', 'public'] },
{ include: ['<memory>', 'public', '<memory>', 'public'] },
{ include: ['<functional>', 'public', '<functional>', 'public'] },
{ include: ['<utility>', 'public', '<utility>', 'public'] },
{ include: ['<type_traits>', 'public', '<type_traits>', 'public'] },
# I/O headers
{ include: ['<iostream>', 'public', '<iostream>', 'public'] },
{ include: ['<iomanip>', 'public', '<iomanip>', 'public'] },
{ include: ['<sstream>', 'public', '<sstream>', 'public'] },
{ include: ['<fstream>', 'public', '<fstream>', 'public'] },
# Common type mappings
{ symbol: ['std::uint8_t', 'public', '<cstdint>', 'public'] },
{ symbol: ['std::uint16_t', 'public', '<cstdint>', 'public'] },
{ symbol: ['std::uint32_t', 'public', '<cstdint>', 'public'] },
{ symbol: ['std::uint64_t', 'public', '<cstdint>', 'public'] },
{ symbol: ['std::int8_t', 'public', '<cstdint>', 'public'] },
{ symbol: ['std::int16_t', 'public', '<cstdint>', 'public'] },
{ symbol: ['std::int32_t', 'public', '<cstdint>', 'public'] },
{ symbol: ['std::int64_t', 'public', '<cstdint>', 'public'] },
{ symbol: ['std::size_t', 'public', '<cstddef>', 'public'] },
{ symbol: ['std::ptrdiff_t', 'public', '<cstddef>', 'public'] },
# Common functions
{ symbol: ['std::memcpy', 'public', '<cstring>', 'public'] },
{ symbol: ['std::memset', 'public', '<cstring>', 'public'] },
{ symbol: ['std::strlen', 'public', '<cstring>', 'public'] },
{ symbol: ['std::malloc', 'public', '<cstdlib>', 'public'] },
{ symbol: ['std::free', 'public', '<cstdlib>', 'public'] },
{ symbol: ['std::abs', 'public', '<cmath>', 'public'] },
{ symbol: ['std::sin', 'public', '<cmath>', 'public'] },
{ symbol: ['std::cos', 'public', '<cmath>', 'public'] },
{ symbol: ['std::sqrt', 'public', '<cmath>', 'public'] },
# C standard library compatibility (prefer C++ versions)
{ include: ['<stdint.h>', 'public', '<cstdint>', 'public'] },
{ include: ['<stddef.h>', 'public', '<cstddef>', 'public'] },
{ include: ['<string.h>', 'public', '<cstring>', 'public'] },
{ include: ['<stdlib.h>', 'public', '<cstdlib>', 'public'] },
{ include: ['<math.h>', 'public', '<cmath>', 'public'] },
{ include: ['<assert.h>', 'public', '<cassert>', 'public'] },
{ include: ['<limits.h>', 'public', '<climits>', 'public'] },
{ include: ['<float.h>', 'public', '<cfloat>', 'public'] },
# Platform-specific but common headers
{ include: ['<sys/types.h>', 'public', '<sys/types.h>', 'public'] },
{ include: ['<unistd.h>', 'public', '<unistd.h>', 'public'] },
]
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/js/README.md
+56
View File
@@ -0,0 +1,56 @@
# FastLED JavaScript Linting Tools
This directory contains JavaScript linting tools for the FastLED project, using Node.js + ESLint for **fast** JavaScript validation.
### Via Main Lint Script (Recommended)
```bash
# JavaScript linting is included by default (fast only!)
bash lint
```
## Setup
The fast JavaScript linter is automatically installed during project setup:
```bash
# Installs everything including fast JS linting
./install
```
Or manually install just the JavaScript linter:
```bash
uv run ci/setup-js-linting-fast.py
```
This will:
- Download Node.js binary to `.js-tools/node/`
- Install ESLint to `.js-tools/node_modules/`
- Create ESLint configuration in `.js-tools/.eslintrc.js`
- Create fast linting script `.cache/js-tools/lint-js-fast`
## Philosophy: Fast Only
**JavaScript linting follows a "fast only" policy:**
- ✅ **Fast linting available**: Uses Node.js + ESLint (~0.9 seconds)
- ⚠️ **Fast linting unavailable**: Skips JavaScript linting entirely
- ❌ **No slow fallback**: Fast linting or no linting at all
## Configuration
The fast linter only checks for **critical runtime issues**:
- `no-debugger` - Prevents debugger statements in production
- `no-eval` - Prevents eval() security vulnerabilities
**No style enforcement** - focuses on critical issues only for maximum speed.
## Integration
JavaScript linting is included by default in `bash lint` when fast linting is available.
## Performance
- **Fast linting**: ~0.9 seconds (53x faster than previous solution)
- **Total lint time**: ~1.6 seconds (Python + C++ + JavaScript)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/kitchensink/kitchensink.ino.cpp
+16
View File
@@ -0,0 +1,16 @@
#include "FastLED.h"
// Let's include a bunch of stuff and see if it breaks the build.
#include <WiFi.h>
#include <ESPmDNS.h>
#include <NetworkUdp.h>
#include <ArduinoOTA.h>
#include <ArduinoJson.h>
void setup() {
}
void loop() {
}
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/kitchensink/platformio.ini
+17
View File
@@ -0,0 +1,17 @@
[platformio]
src_dir = symlink://../../../src
[env:dev]
platform = https://github.com/pioarduino/platform-espressif32/releases/download/51.03.04/platform-espressif32.zip
board = esp32-s3-devkitc-1
framework = arduino
lib_deps =
https://github.com/dvarrel/AsyncTCP
https://github.com/mathieucarbou/ESPAsyncWebServer
ArduinoJson
FS
ArduinoOTA
ESPmDNS
lib_ldf_mode=deep
build_flags=-DFASTLED_STUB_MAIN_INCLUDE_INO="../ci/kitchensink/kitchensink.ino.cpp"
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/lint-js-fast
+98
View File
@@ -0,0 +1,98 @@
#!/bin/bash
# FastLED JavaScript Linting Script (Node.js + ESLint + JSDoc Type Checking - FAST!)
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
echo -e "${BLUE}FAST FastLED JavaScript Linting (Node.js + ESLint + JSDoc Type Checking - FAST!)${NC}"
# Check if ESLint is installed
if [ ! -f ".cache/js-tools/node_modules/.bin/eslint.cmd" ]; then
echo -e "${RED}ERROR: ESLint not found. Run: uv run ci/setup-js-linting-fast.py${NC}"
exit 1
fi
# Find JavaScript files in WASM platform
JS_FILES=$(find src/platforms/wasm -name "*.js" -type f 2>/dev/null)
if [ -z "$JS_FILES" ]; then
echo -e "${YELLOW}WARNING: No JavaScript files found in src/platforms/wasm/${NC}"
exit 0
fi
echo -e "${BLUE}Found JavaScript files:${NC}"
echo "$JS_FILES" | sed 's/^/ /'
# Run ESLint using config from ci/ directory
echo -e "${BLUE}Running ESLint...${NC}"
cd .cache/js-tools
if "./node_modules/.bin/eslint.cmd" --no-eslintrc --no-inline-config -c ../../ci/.eslintrc.js ../../src/platforms/wasm/compiler/*.js ../../src/platforms/wasm/compiler/modules/*.js; then
echo -e "${GREEN}✅ ESLint completed successfully${NC}"
else
echo -e "${RED}ERROR: ESLint failed${NC}"
exit 1
fi
# Run JSDoc Type Checking using TypeScript
echo -e "${BLUE}Running JSDoc Type Checking...${NC}"
cd ../../src/platforms/wasm/compiler
# Check if TypeScript is available (local installation first, then global)
if [ -f "../../../../.cache/js-tools/node_modules/.bin/tsc.cmd" ]; then
echo -e "${BLUE}Using local TypeScript for JSDoc type checking...${NC}"
# Run TypeScript and filter out Node.js events library errors
TSC_OUTPUT=$(../../../../.cache/js-tools/node_modules/.bin/tsc.cmd --noEmit --project jsconfig.json 2>&1)
TSC_EXIT_CODE=$?
# Filter out the specific Node.js events error
FILTERED_OUTPUT=$(echo "$TSC_OUTPUT" | grep -v "node_modules/events/events.js.*Property 'context' does not exist on type 'Error'" || true)
if [ -n "$FILTERED_OUTPUT" ]; then
echo "$FILTERED_OUTPUT"
if [ $TSC_EXIT_CODE -ne 0 ]; then
echo -e "${RED}❌ JSDoc type checking failed with errors${NC}"
exit 1
fi
fi
echo -e "${GREEN}✅ JSDoc type checking completed successfully${NC}"
elif [ -f "../../../../.cache/js-tools/node_modules/.bin/tsc" ]; then
echo -e "${BLUE}Using local TypeScript for JSDoc type checking...${NC}"
# Run TypeScript and filter out Node.js events library errors
TSC_OUTPUT=$(../../../../.cache/js-tools/node_modules/.bin/tsc --noEmit --project jsconfig.json 2>&1)
TSC_EXIT_CODE=$?
# Filter out the specific Node.js events error
FILTERED_OUTPUT=$(echo "$TSC_OUTPUT" | grep -v "node_modules/events/events.js.*Property 'context' does not exist on type 'Error'" || true)
if [ -n "$FILTERED_OUTPUT" ]; then
echo "$FILTERED_OUTPUT"
if [ $TSC_EXIT_CODE -ne 0 ]; then
echo -e "${RED}❌ JSDoc type checking failed with errors${NC}"
exit 1
fi
fi
echo -e "${GREEN}✅ JSDoc type checking completed successfully${NC}"
elif command -v npx >/dev/null 2>&1 && npx tsc --version >/dev/null 2>&1; then
echo -e "${BLUE}Using global TypeScript for JSDoc type checking...${NC}"
# Run TypeScript and filter out Node.js events library errors
TSC_OUTPUT=$(npx tsc --noEmit --project jsconfig.json 2>&1)
TSC_EXIT_CODE=$?
# Filter out the specific Node.js events error
FILTERED_OUTPUT=$(echo "$TSC_OUTPUT" | grep -v "node_modules/events/events.js.*Property 'context' does not exist on type 'Error'" || true)
if [ -n "$FILTERED_OUTPUT" ]; then
echo "$FILTERED_OUTPUT"
if [ $TSC_EXIT_CODE -ne 0 ]; then
echo -e "${RED}❌ JSDoc type checking failed with errors${NC}"
exit 1
fi
fi
echo -e "${GREEN}✅ JSDoc type checking completed successfully${NC}"
else
echo -e "${YELLOW}⚠️ TypeScript not found, skipping JSDoc type checking${NC}"
echo -e "${BLUE}💡 To enable JSDoc checking: Run the ./install script or npm install -g typescript${NC}"
fi
echo -e "${GREEN}SUCCESS: JavaScript linting and type checking completed successfully${NC}"
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/lint-js-fast.backup
+38
View File
@@ -0,0 +1,38 @@
#!/bin/bash
# FastLED JavaScript Linting Script (Node.js + ESLint - FAST!)
# Colors for output
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m' # No Color
echo -e "${BLUE}FAST FastLED JavaScript Linting (Node.js + ESLint - FAST!)${NC}"
# Check if ESLint is installed
if [ ! -f ".cache/js-tools/node_modules/.bin/eslint.cmd" ]; then
echo -e "${RED}ERROR: ESLint not found. Run: uv run ci/setup-js-linting-fast.py${NC}"
exit 1
fi
# Find JavaScript files in WASM platform
JS_FILES=$(find src/platforms/wasm -name "*.js" -type f 2>/dev/null)
if [ -z "$JS_FILES" ]; then
echo -e "${YELLOW}WARNING: No JavaScript files found in src/platforms/wasm/${NC}"
exit 0
fi
echo -e "${BLUE}Found JavaScript files:${NC}"
echo "$JS_FILES" | sed 's/^/ /'
# Run ESLint using config from ci/ directory
echo -e "${BLUE}Running ESLint...${NC}"
cd .cache/js-tools
if "./node_modules/.bin/eslint.cmd" --no-eslintrc --no-inline-config -c ../../ci/.eslintrc.js ../../src/platforms/wasm/compiler/*.js ../../src/platforms/wasm/compiler/modules/*.js; then
echo -e "${GREEN}SUCCESS: JavaScript linting completed successfully${NC}"
else
echo -e "${RED}ERROR: JavaScript linting failed${NC}"
exit 1
fi
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/migration_example.py
+252
View File
@@ -0,0 +1,252 @@
#!/usr/bin/env python3
"""
FastLED Build System Migration Example
This shows how to migrate from the old scattered build logic
to the new unified BuildAPI.
"""
from pathlib import Path
from typing import List
from ci.build_api import create_example_builder, create_unit_test_builder
def old_way_unit_test_building():
"""
OLD WAY: Scattered logic with manual PCH and library building
This is how unit test building worked before the refactor:
- Manual PCH building in multiple places
- Manual library building with duplicated logic
- Mixed build flags between unit tests and examples
- Manual directory management
"""
# OLD CODE (before refactor) - DO NOT USE
"""
# Manual PCH building (scattered in multiple files)
from ci.compiler.test_example_compilation import create_fastled_compiler
from ci.compiler.cpp_test_compile import create_unit_test_fastled_library
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
# Manual library building (duplicated logic)
lib_file = create_unit_test_fastled_library(clean=False, use_pch=True)
# Manual target compilation loop
for test_file in test_files:
success = compiler.compile_cpp_file(test_file, output_obj)
if success:
link_executable(output_obj, lib_file, output_exe)
"""
def new_way_unit_test_building():
"""
NEW WAY: Unified BuildAPI with automatic PCH and library building
This is how unit test building works after the refactor:
- Automatic PCH building (once, reused by all targets)
- Automatic library building (once, reused by all targets)
- Clean separation of unit test build flags
- Automatic directory organization
"""
# NEW CODE (after refactor) - USE THIS
unit_builder = create_unit_test_builder(
build_dir=".build/unit", use_pch=True, parallel=True
)
test_files = [
Path("tests/test_json.cpp"),
Path("tests/test_color.cpp"),
Path("tests/test_math.cpp"),
]
# Single call handles everything:
# - PCH building (automatically, once)
# - Library building (automatically, once)
# - Individual target compilation and linking
results = unit_builder.build_targets(test_files)
# Clean result handling
for result in results:
if result.success:
print(f"✓ {result.target.name} built successfully")
else:
print(f"✗ {result.target.name} failed: {result.message}")
def old_way_example_building():
"""
OLD WAY: Mixed build configuration and manual management
Examples used the same build_flags.toml as unit tests, causing:
- Wrong defines (FASTLED_FORCE_NAMESPACE=1 for examples)
- Mixed configuration between unit tests and examples
- Manual PCH and library building
"""
# OLD CODE (before refactor) - DO NOT USE
"""
# Used same build_flags.toml as unit tests (wrong!)
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
# Manual example compilation with wrong defines
for example_file in example_files:
success = compiler.compile_ino_file(example_file, output_obj)
# ... manual linking logic ...
"""
def new_way_example_building():
"""
NEW WAY: Clean separation with dedicated build configuration
Examples now have their own build_example.toml with:
- Correct defines (no FASTLED_FORCE_NAMESPACE=1)
- Arduino compatibility defines
- Automatic PCH and library building
"""
# NEW CODE (after refactor) - USE THIS
example_builder = create_example_builder(
build_dir=".build/examples", use_pch=True, parallel=True
)
example_files = [
Path("examples/Blink/Blink.ino"),
Path("examples/DemoReel100/DemoReel100.ino"),
]
# Single call handles everything with correct configuration
results = example_builder.build_targets(example_files)
for result in results:
if result.success:
print(f"✓ {result.target.name} built successfully")
else:
print(f"✗ {result.target.name} failed: {result.message}")
def migration_benefits():
"""
Benefits of the new unified BuildAPI:
"""
print("MIGRATION BENEFITS:")
print("=" * 50)
print("1. UNIFIED API:")
print(
" Before: create_fastled_compiler() + create_unit_test_fastled_library() + manual loops"
)
print(" After: create_unit_test_builder().build_targets(files)")
print("\n2. CLEAN SEPARATION:")
print(" Before: build_flags.toml used by both unit tests and examples")
print(" After: build_unit.toml vs build_example.toml")
print("\n3. AUTOMATIC ARTIFACT BUILDING:")
print(" Before: Manual PCH and library building in callers")
print(" After: Automatic PCH and library building in BuildAPI")
print("\n4. ORGANIZED DIRECTORIES:")
print(" Before: Mixed artifacts in .build/")
print(" After: Organized .build/unit/ and .build/examples/ with subdirectories")
print("\n5. MULTIPLE TARGET EFFICIENCY:")
print(" Before: PCH and library built for each target")
print(" After: PCH and library built once, reused by all targets")
print("\n6. SIMPLE ERROR HANDLING:")
print(" Before: Manual error handling in compilation loops")
print(" After: Structured SketchResult objects with clear success/failure")
def migration_checklist():
"""
Checklist for migrating existing code to the new BuildAPI:
"""
print("\nMIGRATION CHECKLIST:")
print("=" * 50)
print("✓ COMPLETED:")
print(" - Split ci/build_flags.toml into build_unit.toml and build_example.toml")
print(" - Created unified BuildAPI class")
print(" - Implemented automatic PCH and library building")
print(" - Implemented organized build directory structure")
print(
" - Created convenience functions create_unit_test_builder() and create_example_builder()"
)
print("\n⚠ TODO:")
print(
" - Update existing unit test compilation code to use create_unit_test_builder()"
)
print(
" - Update existing example compilation code to use create_example_builder()"
)
print(" - Remove old scattered PCH and library building logic")
print(" - Update test scripts to use new API")
print(" - Deprecate old build_flags.toml (marked as legacy)")
def show_api_simplification():
"""
Show how the API is dramatically simplified:
"""
print("\nAPI SIMPLIFICATION:")
print("=" * 50)
print("BEFORE (scattered across multiple files):")
print("""
# In ci/compiler/test_example_compilation.py
compiler = create_fastled_compiler(use_pch=True, use_sccache=False, parallel=True)
# In ci/compiler/cpp_test_compile.py
lib_file = create_unit_test_fastled_library(clean=False, use_pch=True)
# In test runner
for test_file in test_files:
obj_file = build_dir / f"{test_file.stem}.o"
success = compiler.compile_cpp_file(test_file, obj_file)
if success:
exe_file = build_dir / f"{test_file.stem}.exe"
link_success = link_executable(obj_file, lib_file, exe_file)
""")
print("AFTER (unified API):")
print("""
# Single unified API
builder = create_unit_test_builder()
results = builder.build_targets(test_files)
# That's it! Everything else is automatic:
# - PCH building
# - Library building
# - Individual compilation
# - Linking
# - Directory organization
# - Error handling
""")
if __name__ == "__main__":
print("FastLED Build System Migration Example")
print("=" * 60)
migration_benefits()
migration_checklist()
show_api_simplification()
print("\n" + "=" * 60)
print("NEXT STEPS:")
print("1. Update existing unit test compilation to use create_unit_test_builder()")
print("2. Update existing example compilation to use create_example_builder()")
print("3. Remove old scattered build logic")
print("4. Test the new unified system")
print("=" * 60)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/native/platformio.ini
+17
View File
@@ -0,0 +1,17 @@
[platformio]
src_dir = symlink://../../../src
[env:dev]
platform = platformio/native
lib_deps =
FastLED
build_flags =
-DFASTLED_STUB_IMPL
-DFASTLED_USE_STUB_ARDUINO
-DFASTLED_STUB_MAIN_INCLUDE_INO="../examples/Blink/Blink.ino"
-std=c++17
-I../../src/platforms
-isystem ../../src/platforms/stub ; for Arduino.h
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/optimization_report.py
+48
View File
@@ -0,0 +1,48 @@
import argparse
import sys
from pathlib import Path
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description="Convert a binary file to ELF using map file."
)
parser.add_argument("--first", action="store_true", help="Inspect the first board")
parser.add_argument("--cwd", type=Path, help="Custom working directory")
return parser.parse_args()
def main() -> int:
args = parse_args()
if args.cwd:
root_build_dir = args.cwd / ".build"
else:
root_build_dir = Path(".build")
# Support nested PlatformIO structure: .build/pio/<board>
nested_pio_dir = root_build_dir / "pio"
if nested_pio_dir.is_dir():
root_build_dir = nested_pio_dir
board_dirs = [d for d in root_build_dir.iterdir() if d.is_dir()]
if not board_dirs:
print(f"No board directories found in {root_build_dir.absolute()}")
return 1
print("Available boards:")
for i, board_dir in enumerate(board_dirs):
print(f"[{i}]: {board_dir.name}")
which = (
0
if args.first
else int(input("Enter the number of the board you want to inspect: "))
)
board_dir = board_dirs[which]
# build_info_json = board_dir / "build_info.json"
optimization_report = board_dir / "optimization_report.txt"
text = optimization_report.read_text(encoding="utf-8", errors="replace")
print(text)
return 0
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/pyright-cached.py
+151
View File
@@ -0,0 +1,151 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
"""
Caching wrapper for pyright that only runs if files have changed.
This significantly speeds up repeated pyright runs during development.
"""
import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path
from typing import Dict, List, Optional
def get_file_hash(file_path: Path) -> str:
"""Get SHA256 hash of a file."""
try:
with open(file_path, "rb") as f:
return hashlib.sha256(f.read()).hexdigest()
except (OSError, IOError):
return ""
def get_directory_hash(directory: Path, extensions: Optional[List[str]] = None) -> str:
"""Get combined hash of all files in a directory."""
if extensions is None:
extensions = [".py"]
hashes: List[str] = []
if directory.exists():
for ext in extensions:
for file_path in directory.rglob(f"*{ext}"):
if file_path.is_file():
hashes.append(
f"{file_path.relative_to(directory)}:{get_file_hash(file_path)}"
)
hashes.sort() # Ensure consistent ordering
return hashlib.sha256("\n".join(hashes).encode()).hexdigest()
def get_cache_file() -> Path:
"""Get the cache file path."""
cache_dir = Path(".cache/pyright")
cache_dir.mkdir(parents=True, exist_ok=True)
return cache_dir / "file_hashes.json"
def load_cache() -> Dict[str, str]:
"""Load the cache file."""
cache_file = get_cache_file()
if cache_file.exists():
try:
with open(cache_file, "r") as f:
return json.load(f)
except (json.JSONDecodeError, IOError):
pass
return {}
def save_cache(cache: Dict[str, str]) -> None:
"""Save the cache file."""
cache_file = get_cache_file()
try:
with open(cache_file, "w") as f:
json.dump(cache, f, indent=2)
except IOError:
pass # Ignore cache save errors
def has_files_changed() -> bool:
"""Check if any relevant files have changed since last run."""
cache = load_cache()
changed = False
# Files and directories to check
files_to_check = [
Path("pyrightconfig.json"),
Path(".pyrightconfig.json"),
Path("pyproject.toml"),
]
directories_to_check = [
Path("ci"),
]
# Check individual files
for file_path in files_to_check:
if file_path.exists():
current_hash = get_file_hash(file_path)
if cache.get(str(file_path)) != current_hash:
cache[str(file_path)] = current_hash
changed = True
# Check directories
for dir_path in directories_to_check:
if dir_path.exists():
current_hash = get_directory_hash(dir_path)
cache_key = f"dir:{dir_path}"
if cache.get(cache_key) != current_hash:
cache[cache_key] = current_hash
changed = True
# Save cache if any changes were detected
if changed:
save_cache(cache)
return changed
def run_pyright(args: List[str]) -> int:
"""Run pyright with the given arguments."""
env = os.environ.copy()
# env["PYRIGHT_PYTHON_FORCE_VERSION"] = "latest"
env["PYRIGHT_PYTHON_CACHE_DIR"] = ".cache/pyright"
cmd = ["uv", "run", "pyright"] + args
try:
result = subprocess.run(cmd, env=env)
return result.returncode
except KeyboardInterrupt:
return 1
except Exception as e:
print(f"Error running pyright: {e}", file=sys.stderr)
return 1
def main() -> int:
"""Main entry point."""
args = sys.argv[1:] # Remove script name
# If --force is passed, skip cache check
if "--force" in args:
args.remove("--force")
print("Forcing pyright run (--force flag detected)")
return run_pyright(args)
# Check if files have changed
if not has_files_changed():
print("No changes detected, skipping pyright (use --force to override)")
return 0
print("Changes detected, running pyright...")
return run_pyright(args)
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/qemu-esp32.py
+548
View File
@@ -0,0 +1,548 @@
#!/usr/bin/env python3
"""
ESP32 QEMU Runner
Runs ESP32 firmware in QEMU with logging and validation.
Designed to replace the tobozo/esp32-qemu-sim GitHub Action.
"""
import argparse
import platform
import re
import subprocess
import sys
import threading
import time
from pathlib import Path
from typing import List, Optional, TextIO
from ci.util.running_process import RunningProcess
def get_binary_name() -> str:
"""Get platform-specific QEMU binary name."""
return (
"qemu-system-xtensa.exe"
if platform.system().lower() == "windows"
else "qemu-system-xtensa"
)
def find_executable(binary_name: str) -> Optional[Path]:
"""Find executable in PATH using platform-appropriate command."""
try:
cmd = [
"where" if platform.system().lower() == "windows" else "which",
binary_name,
]
result = subprocess.run(cmd, capture_output=True, text=True)
if result.returncode == 0:
# Take first line in case of multiple results (Windows 'where')
return Path(result.stdout.strip().split("\n")[0])
except (subprocess.SubprocessError, FileNotFoundError):
pass
return None
def find_qemu_binary() -> Optional[Path]:
"""Find QEMU ESP32 binary in common locations."""
binary_name = get_binary_name()
# ESP-IDF installation paths and portable installation
esp_paths = [
Path(".cache") / "qemu", # Project-local portable installation directory
Path.home() / ".espressif" / "tools" / "qemu-xtensa",
Path.home() / ".espressif" / "python_env",
]
for base_path in esp_paths:
if base_path.exists():
for qemu_path in base_path.rglob(binary_name):
if qemu_path.is_file():
return qemu_path
# Check system PATH
return find_executable(binary_name)
def show_installation_help():
"""Display platform-specific installation instructions."""
binary_name = get_binary_name()
print(f"ERROR: QEMU ESP32 emulator ({binary_name}) not found!", file=sys.stderr)
print()
if platform.system().lower() == "windows":
print("To install QEMU on Windows:")
print("1. Chocolatey: choco install qemu -y")
print("2. winget: winget install SoftwareFreedomConservancy.QEMU --scope user")
print("3. Manual: Download from https://qemu.weilnetz.de/w64/")
print("4. ESP-IDF: uv run ci/install-qemu.py")
else:
print("Run: uv run ci/install-qemu.py")
class QEMURunner:
def __init__(self, qemu_binary: Optional[Path] = None):
self.qemu_binary = qemu_binary or self._get_qemu_binary()
self.output_lines: List[str] = []
self.interrupt_met = False
def _get_qemu_binary(self) -> Path:
"""Get QEMU binary path or raise error."""
qemu_path = find_qemu_binary()
if not qemu_path:
raise FileNotFoundError(
f"Could not find {get_binary_name()}. Please run install-qemu.py first."
)
return qemu_path
def build_qemu_command(self, firmware_path: Path, flash_size: int = 4) -> List[str]:
"""Build QEMU command line arguments.
Args:
firmware_path: Path to firmware.bin or directory containing build artifacts
flash_size: Flash size in MB
"""
# Validate that flash_size is used correctly internally only
if not isinstance(flash_size, int) or flash_size <= 0:
raise ValueError(
f"Invalid flash_size: {flash_size}. Must be a positive integer."
)
# Determine if we have a direct firmware.bin or a build directory
if firmware_path.is_file() and firmware_path.suffix == ".bin":
# Direct firmware.bin path
flash_image_path = firmware_path.parent / "flash.bin"
print(f"Creating flash image for firmware: {firmware_path}")
self._create_flash_image_from_firmware(
firmware_path, flash_image_path, flash_size
)
else:
# Build directory path
flash_image_path = firmware_path / "flash.bin"
print(
f"Creating combined flash image from build directory: {firmware_path}"
)
self._create_flash_image_from_build_dir(
firmware_path, flash_image_path, flash_size
)
# Check for ROM file in multiple locations
rom_paths = [
Path(".cache") / "qemu" / "esp32-v3-rom.bin",
Path(".cache") / "qemu" / "qemu" / "share" / "qemu" / "esp32-v3-rom.bin",
Path(".cache") / "qemu" / "extracted" / "share" / "esp32-v3-rom.bin",
]
rom_path = None
for path in rom_paths:
if path.exists():
rom_path = path
break
# Ensure flash image path is absolute and exists
flash_image_path = flash_image_path.resolve()
if not flash_image_path.exists():
raise FileNotFoundError(f"Flash image not found: {flash_image_path}")
cmd = [
str(self.qemu_binary),
"-nographic",
"-machine",
"esp32",
"-drive",
f"file={str(flash_image_path)},if=mtd,format=raw",
"-global",
"driver=timer.esp32.timg,property=wdt_disable,value=true",
]
# Set QEMU data directory if ROM files are available
if rom_path and rom_path.exists():
qemu_data_dir = rom_path.parent
cmd.extend(["-L", str(qemu_data_dir)])
print(f"Setting QEMU data directory: {qemu_data_dir}")
# Note: ESP32 ROM is built into QEMU machine, no need to load external ROM
# ESP32 QEMU machine has the ROM integrated, so we don't need to specify it
if rom_path and rom_path.exists():
print(
f"ROM file available at: {rom_path} (but not needed for ESP32 machine)"
)
else:
print("Note: ROM file not found, but ESP32 machine has integrated ROM")
# Validate that no custom script arguments are accidentally included
custom_args = ["--flash-size", "--timeout", "--interrupt-regex"]
for arg in cmd:
if any(custom_arg in str(arg) for custom_arg in custom_args):
raise ValueError(
f"Custom script argument {arg} should not be passed to QEMU"
)
return cmd
def _create_flash_image_from_firmware(
self, firmware_bin: Path, flash_bin: Path, flash_size: int
):
"""Create flash image from a direct firmware.bin file."""
if flash_bin.exists():
print(f"Using existing flash image: {flash_bin}")
return
print(f"Creating flash image from firmware: {firmware_bin}")
# Create empty flash image
flash_size_bytes = flash_size * 1024 * 1024
flash_bin.write_bytes(b"\xff" * flash_size_bytes)
try:
esptool_cmd = self._find_esptool()
if esptool_cmd:
# Use esptool to create proper flash image
self._merge_with_esptool_firmware(esptool_cmd, firmware_bin, flash_bin)
else:
# Simple manual merge - just write firmware at 0x10000
self._merge_manually_firmware(firmware_bin, flash_bin)
except Exception as e:
print(f"Warning: Could not create proper flash image: {e}")
def _create_flash_image_from_build_dir(
self, build_folder: Path, flash_bin: Path, flash_size: int
):
"""Create combined flash image from ESP32 build artifacts in a directory."""
if flash_bin.exists():
print(f"Using existing flash image: {flash_bin}")
return
print("Creating combined flash image from build directory...")
# Create empty flash image
flash_size_bytes = flash_size * 1024 * 1024
flash_bin.write_bytes(b"\xff" * flash_size_bytes)
try:
esptool_cmd = self._find_esptool()
if esptool_cmd:
self._merge_with_esptool(esptool_cmd, build_folder, flash_bin)
else:
self._merge_manually(build_folder, flash_bin)
except Exception as e:
print(f"Warning: Could not create proper flash image: {e}")
def _find_esptool(self) -> Optional[str]:
"""Find esptool.py executable."""
# Try direct esptool.py
if find_executable("esptool.py"):
return "esptool.py"
# Try python -m esptool
try:
result = subprocess.run(
[sys.executable, "-m", "esptool", "--help"],
capture_output=True,
timeout=5,
)
if result.returncode == 0:
return f"{sys.executable} -m esptool"
except (subprocess.SubprocessError, subprocess.TimeoutExpired):
pass
return None
def _merge_with_esptool(
self, esptool_cmd: str, build_folder: Path, flash_bin: Path
):
"""Merge using esptool.py merge_bin command."""
cmd = esptool_cmd.split() + [
"--chip",
"esp32", # Use esp32 chip type for esp32dev machine
"merge-bin", # Updated command name
"-o",
str(flash_bin),
"--flash-mode", # Updated option name
"dio",
"--flash-freq", # Updated option name
"40m", # Standard frequency for esp32
"--flash-size", # Updated option name
"4MB",
]
# Add binaries at their offsets - check if we're in the actual build directory
if (build_folder / "firmware.bin").exists():
firmware_path = build_folder / "firmware.bin"
else:
firmware_path = (
build_folder / ".pio" / "build" / "esp32dev" / "firmware.bin"
)
if not firmware_path.exists():
print(
f"FATAL: ESP32dev firmware not found at {firmware_path}",
file=sys.stderr,
)
print("Please build for esp32dev target first.", file=sys.stderr)
sys.exit(1)
binaries = [
("0x1000", build_folder / "bootloader.bin"), # ESP32 bootloader offset
("0x8000", build_folder / "partitions.bin"),
("0x10000", firmware_path),
]
for offset, bin_path in binaries:
if bin_path.exists():
cmd.extend([offset, str(bin_path)])
# Add padding to 4MB at the end
cmd.extend(["--pad-to-size", "4MB"])
print(f"Running esptool command: {' '.join(cmd)}")
esptool_proc = RunningProcess(cmd, timeout=60, auto_run=True)
# Stream esptool output and check for errors
has_errors = False
with esptool_proc.line_iter(timeout=None) as it:
for line in it:
print(f"[esptool] {line}")
# Check for common esptool error patterns
if any(
error_pattern in line.lower()
for error_pattern in ["error", "failed", "exception"]
):
has_errors = True
returncode = esptool_proc.wait()
if returncode != 0:
raise subprocess.CalledProcessError(returncode, cmd)
if has_errors:
print("Warning: Errors detected in esptool output")
print("Flash image created with esptool")
def _merge_with_esptool_firmware(
self, esptool_cmd: str, firmware_bin: Path, flash_bin: Path
):
"""Merge firmware.bin using esptool with minimal bootloader."""
cmd = esptool_cmd.split() + [
"--chip",
"esp32",
"merge-bin", # Updated command name
"-o",
str(flash_bin),
"--flash-mode", # Updated option name
"dio",
"--flash-freq", # Updated option name
"40m",
"--flash-size", # Updated option name
"4MB",
"0x10000", # Application offset
str(firmware_bin),
]
# Note: Without bootloader and partition table, QEMU may not boot properly
# but we'll try with just the app for now
print(f"Running esptool firmware command: {' '.join(cmd)}")
esptool_proc = RunningProcess(cmd, timeout=60, auto_run=True)
# Stream esptool output and check for errors
has_errors = False
with esptool_proc.line_iter(timeout=None) as it:
for line in it:
print(f"[esptool] {line}")
# Check for common esptool error patterns
if any(
error_pattern in line.lower()
for error_pattern in ["error", "failed", "exception"]
):
has_errors = True
returncode = esptool_proc.wait()
if returncode != 0:
raise subprocess.CalledProcessError(returncode, cmd)
if has_errors:
print("Warning: Errors detected in esptool output")
print("Flash image created with esptool (firmware only)")
def _merge_manually_firmware(self, firmware_bin: Path, flash_bin: Path):
"""Manually write firmware.bin to flash image."""
print("WARNING: Manually merging firmware (esptool not found)")
print("Note: Without bootloader, QEMU may not boot properly")
with open(flash_bin, "r+b") as flash:
# Write firmware at application offset
flash.seek(0x10000)
flash.write(firmware_bin.read_bytes())
print(f"Firmware written to flash image at offset 0x10000")
def _merge_manually(self, build_folder: Path, flash_bin: Path):
"""Manually merge binary files at correct offsets."""
print("WARNING: Manually merging binaries (esptool not found)")
# Find firmware binary - check if we're in the actual build directory
if (build_folder / "firmware.bin").exists():
firmware_path = build_folder / "firmware.bin"
else:
firmware_path = (
build_folder / ".pio" / "build" / "esp32dev" / "firmware.bin"
)
if not firmware_path.exists():
print(
f"FATAL: ESP32dev firmware not found at {firmware_path}",
file=sys.stderr,
)
print("Please build for esp32dev target first.", file=sys.stderr)
sys.exit(1)
binaries = [
("bootloader.bin", 0x1000), # ESP32 bootloader offset
("partitions.bin", 0x8000),
(firmware_path, 0x10000),
]
with open(flash_bin, "r+b") as flash:
for filename, offset in binaries:
bin_path = build_folder / filename
if bin_path.exists():
flash.seek(offset)
flash.write(bin_path.read_bytes())
def run(
self,
firmware_path: Path,
timeout: int = 30,
interrupt_regex: Optional[str] = None,
flash_size: int = 4,
) -> int:
"""Run ESP32 firmware in QEMU.
Args:
firmware_path: Path to firmware.bin file or build directory
timeout: Timeout in seconds
interrupt_regex: Regex pattern to interrupt execution
flash_size: Flash size in MB
"""
print("=== Running ESP32 firmware in QEMU ===")
print(f"Firmware path: {firmware_path}")
print(f"Timeout: {timeout}s")
print(f"Flash size: {flash_size}MB")
if interrupt_regex:
print(f"Interrupt pattern: {interrupt_regex}")
if not firmware_path.exists():
print(f"ERROR: Path does not exist: {firmware_path}", file=sys.stderr)
return 1
try:
cmd = self.build_qemu_command(firmware_path, flash_size)
print(f"QEMU command: {' '.join(cmd)}")
except Exception as e:
print(f"ERROR: Failed to build QEMU command: {e}", file=sys.stderr)
return 1
try:
qemu_proc = RunningProcess(
cmd,
timeout=timeout,
auto_run=True,
)
self.interrupt_met = False
interrupt_pattern = re.compile(interrupt_regex) if interrupt_regex else None
self.output_lines = []
try:
with qemu_proc.line_iter(timeout=timeout) as it:
for line in it:
print(line, flush=True)
self.output_lines.append(line)
if interrupt_pattern and interrupt_pattern.search(line):
print(f"SUCCESS: Interrupt condition met: {line}")
self.interrupt_met = True
time.sleep(2)
except TimeoutError:
print(
f"ERROR: QEMU timed out after {timeout} seconds of no output.",
file=sys.stderr,
)
qemu_proc.kill()
return 1
return_code = qemu_proc.wait()
log_file = Path("qemu_output.log")
log_file.write_text("\n".join(self.output_lines))
print(f"QEMU output saved to: {log_file}")
print(f"Total output lines: {len(self.output_lines)}")
if self.interrupt_met:
return 0
return return_code
except Exception as e:
print(f"ERROR: Error running QEMU: {e}", file=sys.stderr)
return 1
def main():
parser = argparse.ArgumentParser(description="Run ESP32 firmware in QEMU")
parser.add_argument(
"firmware_path",
type=Path,
help="Path to firmware.bin file or build directory containing firmware files",
)
parser.add_argument(
"--flash-size", type=int, default=4, help="Flash size in MB (default: 4)"
)
parser.add_argument(
"--timeout", type=int, default=30, help="Timeout in seconds (default: 30)"
)
parser.add_argument(
"--interrupt-regex", type=str, help="Regex pattern to interrupt execution"
)
args = parser.parse_args()
# Validate arguments to prevent issues
if args.flash_size <= 0:
print(
f"ERROR: Invalid flash size: {args.flash_size}. Must be positive.",
file=sys.stderr,
)
return 1
if args.timeout <= 0:
print(
f"ERROR: Invalid timeout: {args.timeout}. Must be positive.",
file=sys.stderr,
)
return 1
# Ensure we're not accidentally being called as a wrapper for qemu-system-xtensa
if len(sys.argv) > 1 and any("qemu-system-xtensa" in arg for arg in sys.argv):
print(
"ERROR: This script should not be used as a wrapper for qemu-system-xtensa",
file=sys.stderr,
)
print(
"Use this script directly: python qemu-esp32.py <build_folder> [options]",
file=sys.stderr,
)
return 1
# Check if QEMU is available
if not find_qemu_binary():
show_installation_help()
return 1
runner = QEMURunner()
return runner.run(
firmware_path=args.firmware_path,
timeout=args.timeout,
interrupt_regex=args.interrupt_regex,
flash_size=args.flash_size,
)
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/run-clang-format.py
+435
View File
@@ -0,0 +1,435 @@
#!/usr/bin/env python
# pyright: reportUnknownMemberType=false
"""A wrapper script around clang-format, suitable for linting multiple files
and to use for continuous integration.
This is an alternative API for the clang-format command line.
It runs over multiple files and directories in parallel.
A diff output is produced and a sensible exit code is returned.
"""
from __future__ import print_function, unicode_literals
import argparse
import codecs
import difflib
import errno
import fnmatch
import io
import multiprocessing
import os
import signal
import subprocess
import sys
import traceback
from functools import partial
from typing import Any, Dict, Generator, List, Optional, Tuple, Union
try:
from subprocess import DEVNULL # py3k
except ImportError:
DEVNULL = open(os.devnull, "wb") # type: ignore
DEFAULT_EXTENSIONS = "c,h,C,H,cpp,hpp,cc,hh,c++,h++,cxx,hxx"
DEFAULT_CLANG_FORMAT_IGNORE = ".clang-format-ignore"
class ExitStatus:
SUCCESS = 0
DIFF = 1
TROUBLE = 2
def excludes_from_file(ignore_file: str) -> List[str]:
excludes: List[str] = []
try:
with io.open(ignore_file, "r", encoding="utf-8") as f:
for line in f:
if line.startswith("#"):
# ignore comments
continue
pattern = line.rstrip()
if not pattern:
# allow empty lines
continue
excludes.append(pattern)
except EnvironmentError as e:
if e.errno != errno.ENOENT:
raise
return excludes
def list_files(
files: List[str],
recursive: bool = False,
extensions: Optional[List[str]] = None,
exclude: Optional[List[str]] = None,
) -> List[str]:
if extensions is None:
extensions = []
if exclude is None:
exclude = []
out: List[str] = []
for file in files:
if recursive and os.path.isdir(file):
for dirpath, dnames, fnames in os.walk(file):
fpaths = [os.path.join(dirpath, fname) for fname in fnames]
for pattern in exclude:
# os.walk() supports trimming down the dnames list
# by modifying it in-place,
# to avoid unnecessary directory listings.
dnames[:] = [
x
for x in dnames
if not fnmatch.fnmatch(os.path.join(dirpath, x), pattern)
]
fpaths = [x for x in fpaths if not fnmatch.fnmatch(x, pattern)]
for f in fpaths:
ext = os.path.splitext(f)[1][1:]
if ext in extensions:
out.append(f)
else:
out.append(file)
return out
def make_diff(file: str, original: List[str], reformatted: List[str]) -> List[str]:
return list(
difflib.unified_diff(
original,
reformatted,
fromfile="{}\t(original)".format(file),
tofile="{}\t(reformatted)".format(file),
n=3,
)
)
class DiffError(Exception):
def __init__(self, message: str, errs: Optional[List[str]] = None):
super(DiffError, self).__init__(message)
self.errs: List[str] = errs or []
class UnexpectedError(Exception):
def __init__(self, message: str, exc: Optional[Exception] = None):
super(UnexpectedError, self).__init__(message)
self.formatted_traceback = traceback.format_exc()
self.exc = exc
def run_clang_format_diff_wrapper(args: Any, file: str) -> Tuple[List[str], List[str]]:
try:
ret = run_clang_format_diff(args, file)
return ret
except DiffError:
raise
except Exception as e:
raise UnexpectedError("{}: {}: {}".format(file, e.__class__.__name__, e), e)
def run_clang_format_diff(args: Any, file: str) -> Tuple[List[str], List[str]]:
try:
with io.open(file, "r", encoding="utf-8") as f:
original = f.readlines()
except IOError as exc:
raise DiffError(str(exc))
if args.in_place:
invocation = [args.clang_format_executable, "-i", file]
else:
invocation = [args.clang_format_executable, file]
if args.style:
invocation.extend(["--style", args.style])
if args.dry_run:
print(" ".join(invocation))
return [], []
# Use of utf-8 to decode the process output.
#
# Hopefully, this is the correct thing to do.
#
# It's done due to the following assumptions (which may be incorrect):
# - clang-format will returns the bytes read from the files as-is,
# without conversion, and it is already assumed that the files use utf-8.
# - if the diagnostics were internationalized, they would use utf-8:
# > Adding Translations to Clang
# >
# > Not possible yet!
# > Diagnostic strings should be written in UTF-8,
# > the client can translate to the relevant code page if needed.
# > Each translation completely replaces the format string
# > for the diagnostic.
# > -- http://clang.llvm.org/docs/InternalsManual.html#internals-diag-translation
#
# It's not pretty, due to Python 2 & 3 compatibility.
encoding_py3: Dict[str, str] = {}
if sys.version_info[0] >= 3:
encoding_py3["encoding"] = "utf-8"
try:
if sys.version_info[0] >= 3:
proc: subprocess.Popen[str] = subprocess.Popen(
invocation,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True,
encoding="utf-8",
)
else:
proc: subprocess.Popen[str] = subprocess.Popen(
invocation,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
universal_newlines=True,
)
except OSError as exc:
raise DiffError(
"Command '{}' failed to start: {}".format(
subprocess.list2cmdline(invocation), exc
)
)
proc_stdout = proc.stdout
proc_stderr = proc.stderr
assert proc_stdout is not None
assert proc_stderr is not None
if sys.version_info[0] < 3:
# make the pipes compatible with Python 3,
# reading lines should output unicode
encoding = "utf-8"
proc_stdout = codecs.getreader(encoding)(proc_stdout)
proc_stderr = codecs.getreader(encoding)(proc_stderr)
# hopefully the stderr pipe won't get full and block the process
outs = list(proc_stdout.readlines())
errs = list(proc_stderr.readlines())
proc.wait()
if proc.returncode:
raise DiffError(
"Command '{}' returned non-zero exit status {}".format(
subprocess.list2cmdline(invocation), proc.returncode
),
errs,
)
if args.in_place:
return [], errs
return make_diff(file, original, outs), errs
def bold_red(s: str) -> str:
return "\x1b[1m\x1b[31m" + s + "\x1b[0m"
def colorize(diff_lines: List[str]) -> Generator[str, None, None]:
def bold(s: str) -> str:
return "\x1b[1m" + s + "\x1b[0m"
def cyan(s: str) -> str:
return "\x1b[36m" + s + "\x1b[0m"
def green(s: str) -> str:
return "\x1b[32m" + s + "\x1b[0m"
def red(s: str) -> str:
return "\x1b[31m" + s + "\x1b[0m"
for line in diff_lines:
if line[:4] in ["--- ", "+++ "]:
yield bold(line)
elif line.startswith("@@ "):
yield cyan(line)
elif line.startswith("+"):
yield green(line)
elif line.startswith("-"):
yield red(line)
else:
yield line
def print_diff(diff_lines: List[str], use_color: bool) -> None:
if use_color:
diff_lines = list(colorize(diff_lines))
if sys.version_info[0] < 3:
sys.stdout.writelines((line.encode("utf-8") for line in diff_lines))
else:
sys.stdout.writelines(diff_lines)
def print_trouble(prog: str, message: str, use_colors: bool) -> None:
error_text = "error:"
if use_colors:
error_text = bold_red(error_text)
print("{}: {} {}".format(prog, error_text, message), file=sys.stderr)
def main() -> int:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument(
"--clang-format-executable",
metavar="EXECUTABLE",
help="path to the clang-format executable",
default="clang-format",
)
parser.add_argument(
"--extensions",
help="comma separated list of file extensions (default: {})".format(
DEFAULT_EXTENSIONS
),
default=DEFAULT_EXTENSIONS,
)
parser.add_argument(
"-r",
"--recursive",
action="store_true",
help="run recursively over directories",
)
parser.add_argument(
"-d", "--dry-run", action="store_true", help="just print the list of files"
)
parser.add_argument(
"-i",
"--in-place",
action="store_true",
help="format file instead of printing differences",
)
parser.add_argument("files", metavar="file", nargs="+")
parser.add_argument(
"-q",
"--quiet",
action="store_true",
help="disable output, useful for the exit code",
)
parser.add_argument(
"-j",
metavar="N",
type=int,
default=0,
help="run N clang-format jobs in parallel (default number of cpus + 1)",
)
parser.add_argument(
"--color",
default="auto",
choices=["auto", "always", "never"],
help="show colored diff (default: auto)",
)
parser.add_argument(
"-e",
"--exclude",
metavar="PATTERN",
action="append",
default=[],
help="exclude paths matching the given glob-like pattern(s)"
" from recursive search",
)
parser.add_argument(
"--style",
help="formatting style to apply (LLVM, Google, Chromium, Mozilla, WebKit)",
)
args = parser.parse_args()
# use default signal handling, like diff return SIGINT value on ^C
# https://bugs.python.org/issue14229#msg156446
signal.signal(signal.SIGINT, signal.SIG_DFL)
try:
signal.SIGPIPE # type: ignore
except AttributeError:
# compatibility, SIGPIPE does not exist on Windows
pass
else:
signal.signal(signal.SIGPIPE, signal.SIG_DFL) # type: ignore
colored_stdout = False
colored_stderr = False
if args.color == "always":
colored_stdout = True
colored_stderr = True
elif args.color == "auto":
colored_stdout = sys.stdout.isatty()
colored_stderr = sys.stderr.isatty()
version_invocation = [args.clang_format_executable, str("--version")]
try:
subprocess.check_call(version_invocation, stdout=DEVNULL)
except subprocess.CalledProcessError as e:
print_trouble(parser.prog, str(e), use_colors=colored_stderr)
return ExitStatus.TROUBLE
except OSError as e:
print_trouble(
parser.prog,
"Command '{}' failed to start: {}".format(
subprocess.list2cmdline(version_invocation), e
),
use_colors=colored_stderr,
)
return ExitStatus.TROUBLE
retcode = ExitStatus.SUCCESS
excludes = excludes_from_file(DEFAULT_CLANG_FORMAT_IGNORE)
excludes.extend(args.exclude)
files = list_files(
args.files,
recursive=args.recursive,
exclude=excludes,
extensions=args.extensions.split(","),
)
if not files:
return retcode
njobs = args.j
if njobs == 0:
njobs = multiprocessing.cpu_count() + 1
njobs = min(len(files), njobs)
if njobs == 1:
# execute directly instead of in a pool,
# less overhead, simpler stacktraces
it = (run_clang_format_diff_wrapper(args, file) for file in files)
pool = None
else:
pool = multiprocessing.Pool(njobs)
it = pool.imap_unordered(partial(run_clang_format_diff_wrapper, args), files)
pool.close()
while True:
try:
outs, errs = next(it)
except StopIteration:
break
except DiffError as e:
print_trouble(parser.prog, str(e), use_colors=colored_stderr)
retcode = ExitStatus.TROUBLE
sys.stderr.writelines(e.errs)
except UnexpectedError as e:
print_trouble(parser.prog, str(e), use_colors=colored_stderr)
sys.stderr.write(e.formatted_traceback)
retcode = ExitStatus.TROUBLE
# stop at the first unexpected error,
# something could be very wrong,
# don't process all files unnecessarily
if pool:
pool.terminate()
break
else:
sys.stderr.writelines(errs)
if outs == []:
continue
if not args.quiet:
print_diff(outs, use_color=colored_stdout)
if retcode == ExitStatus.SUCCESS:
retcode = ExitStatus.DIFF
if pool:
pool.join()
return retcode
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/run_test_wasm_local_compile.py
+30
View File
@@ -0,0 +1,30 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false
"""
Unit test file.
"""
import unittest
class ApiTester(unittest.TestCase):
"""Main tester class."""
def test_build_all_examples(self) -> None:
"""Test command line interface (CLI)."""
from fastled import Api, Test # type: ignore
with Api.server(auto_updates=True) as server:
exception_map = Test.test_examples(host=server)
if len(exception_map) > 0:
exception: Exception
msg: str = ""
for example, exception in exception_map.items():
msg += f"Failed to compile example: {example}, error: {exception}\n"
self.fail(msg)
# self.assertEqual(0, len(out), f"Failed tests: {out}")
if __name__ == "__main__":
unittest.main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/run_tests.py
+285
View File
@@ -0,0 +1,285 @@
#!/usr/bin/env python3
"""
Simple test runner for FastLED unit tests.
Discovers and runs test executables in parallel with clean output handling.
"""
import argparse
import os
import subprocess
import sys
import time
from concurrent.futures import ThreadPoolExecutor, as_completed
from dataclasses import dataclass
from pathlib import Path
from typing import List, Optional
from ci.util.running_process import RunningProcess
from ci.util.test_exceptions import (
TestExecutionFailedException,
TestFailureInfo,
)
_HERE = Path(__file__).parent
_PROJECT_ROOT = _HERE.parent
@dataclass
class TestResult:
"""Result of a test execution"""
name: str
success: bool
duration: float
output: str
def _is_test_executable(f: Path) -> bool:
"""Check if a file is a valid test executable"""
return (
f.is_file() and f.suffix not in [".o", ".obj", ".pdb"] and os.access(f, os.X_OK)
)
def _get_test_patterns() -> List[str]:
"""Get test patterns based on platform"""
# On Windows, check both .exe and no extension (Clang generates without .exe)
return ["test_*.exe"] if sys.platform == "win32" else ["test_*"]
def discover_tests(build_dir: Path, specific_test: Optional[str] = None) -> List[Path]:
"""Find test executables in the build directory"""
# Check test directory
possible_test_dirs = [
_PROJECT_ROOT / "tests" / "bin", # Python build system
]
test_dir = None
for possible_dir in possible_test_dirs:
if possible_dir.exists():
# Check if this directory has actual executable test files
executable_tests = [
f
for pattern in _get_test_patterns()
for f in possible_dir.glob(pattern)
if _is_test_executable(f)
]
if executable_tests:
test_dir = possible_dir
break
if not test_dir:
print(f"Error: No test directory found. Checked: {possible_test_dirs}")
sys.exit(1)
test_files: List[Path] = []
for pattern in _get_test_patterns():
for test_file in test_dir.glob(pattern):
if not _is_test_executable(test_file):
continue
if specific_test:
# Support both "test_name" and "name" formats (case-insensitive)
test_stem = test_file.stem
test_name = test_stem.replace("test_", "")
if (
test_stem.lower() == specific_test.lower()
or test_name.lower() == specific_test.lower()
):
test_files.append(test_file)
else:
test_files.append(test_file)
return test_files
def run_test(test_file: Path, verbose: bool = False) -> TestResult:
"""Run a single test and capture its output"""
start_time = time.time()
# Build command with doctest flags
# Convert to absolute path for Windows compatibility
test_executable = test_file.resolve()
cmd = [str(test_executable)]
if not verbose:
cmd.append("--minimal") # Only show output for failures
try:
process = RunningProcess(
command=cmd,
cwd=Path(".").absolute(),
check=False,
shell=False,
auto_run=True,
)
with process.line_iter(timeout=30) as it:
for line in it:
print(line)
# Wait for process to complete
process.wait()
success = process.poll() == 0
output = process.stdout
except Exception as e:
success = False
output = f"Error running test: {e}"
duration = time.time() - start_time
return TestResult(
name=test_file.stem, success=success, duration=duration, output=output
)
def main() -> None:
try:
parser = argparse.ArgumentParser(description="Run FastLED unit tests")
parser.add_argument("--test", help="Run specific test (without test_ prefix)")
parser.add_argument(
"--verbose", "-v", action="store_true", help="Show all test output"
)
parser.add_argument("--jobs", "-j", type=int, help="Number of parallel jobs")
args = parser.parse_args()
# Find build directory
build_dir = Path.cwd() / ".build"
if not build_dir.exists():
print("Error: Build directory not found. Run compilation first.")
sys.exit(1)
# Discover tests
test_files = discover_tests(build_dir, args.test)
if not test_files:
if args.test:
print(f"Error: No test found matching '{args.test}'")
else:
print("Error: No tests found")
sys.exit(1)
print(f"Running {len(test_files)} tests...")
if args.test:
print(f"Filter: {args.test}")
# Determine number of parallel jobs
if os.environ.get("NO_PARALLEL"):
max_workers = 1
print(
"NO_PARALLEL environment variable set - forcing sequential test execution"
)
elif args.jobs:
max_workers = args.jobs
else:
import multiprocessing
max_workers = max(1, multiprocessing.cpu_count() - 1)
# Run tests in parallel
start_time = time.time()
results: List[TestResult] = []
failed_tests: List[TestResult] = []
with ThreadPoolExecutor(max_workers=max_workers) as executor:
future_to_test = {
executor.submit(run_test, test_file, args.verbose): test_file
for test_file in test_files
}
completed = 0
for future in as_completed(future_to_test):
test_file = future_to_test[future]
completed += 1
try:
result = future.result()
results.append(result)
# Show progress
status = "PASS" if result.success else "FAIL"
print(
f"[{completed}/{len(test_files)}] {status} {result.name} ({result.duration:.2f}s)"
)
# Show output for failures or in verbose mode
if not result.success or args.verbose:
if result.output:
print(result.output)
if not result.success:
failed_tests.append(result)
# Early abort after 3 failures
if len(failed_tests) >= 3:
print(
"Reached failure threshold (3). Aborting remaining unit tests."
)
# Cancel remaining futures and stop accepting new work
try:
executor.shutdown(wait=False, cancel_futures=True)
except TypeError:
# Python < 3.9: cancel_futures not available
pass
break
except Exception as e:
print(f"Error running {test_file.name}: {e}")
failed_tests.append(
TestResult(
name=test_file.stem,
success=False,
duration=0.0,
output=str(e),
)
)
if len(failed_tests) >= 3:
print(
"Reached failure threshold (3) due to errors. Aborting remaining unit tests."
)
try:
executor.shutdown(wait=False, cancel_futures=True)
except TypeError:
pass
break
# Show summary
total_duration = time.time() - start_time
success_count = len(results) - len(failed_tests)
print("\nTest Summary:")
print(f"Total tests: {len(results)}")
print(f"Passed: {success_count}")
print(f"Failed: {len(failed_tests)}")
print(f"Total time: {total_duration:.2f}s")
if failed_tests:
print("\nFailed tests:")
failures: list[TestFailureInfo] = []
for test in failed_tests:
print(f" {test.name}")
failures.append(
TestFailureInfo(
test_name=test.name,
command=f"test_{test.name}",
return_code=1, # Failed tests get return code 1
output=test.output,
error_type="test_execution_failure",
)
)
raise TestExecutionFailedException(
f"{len(failed_tests)} test(s) failed", failures
)
except TestExecutionFailedException as e:
# Print detailed failure information
print("\n" + "=" * 60)
print("FASTLED TEST EXECUTION FAILURE DETAILS")
print("=" * 60)
print(e.get_detailed_failure_info())
print("=" * 60)
# Exit with appropriate code
sys.exit(1)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/setup-js-linting-fast.py
+284
View File
@@ -0,0 +1,284 @@
#!/usr/bin/env -S uv run --script
"""
Fast JavaScript linting setup for FastLED using Node.js + ESLint
Downloads Node.js binary and sets up fast linting - much faster than Deno
Uses uv run --script for dynamic package management
"""
import io
import json
import os
import platform
import shutil
import subprocess
import sys
import tarfile
import zipfile
from pathlib import Path
# Import httpx for HTTP requests (dynamically managed by uv)
import httpx
# Force UTF-8 output for Windows consoles
if sys.platform.startswith("win"):
sys.stdout = io.TextIOWrapper(sys.stdout.buffer, encoding="utf-8", errors="replace")
sys.stderr = io.TextIOWrapper(sys.stderr.buffer, encoding="utf-8", errors="replace")
# Configuration
NODE_VERSION = "20.11.0" # LTS version
TOOLS_DIR = Path(".cache/js-tools")
NODE_DIR = TOOLS_DIR / "node"
def get_node_download_info():
"""Get Node.js download URL and filename based on platform"""
system = platform.system().lower()
machine = platform.machine().lower()
# Map architecture names
if machine in ["x86_64", "amd64"]:
arch = "x64"
elif machine in ["aarch64", "arm64"]:
arch = "arm64"
else:
raise ValueError(f"Unsupported architecture: {machine}")
if system == "windows":
filename = f"node-v{NODE_VERSION}-win-{arch}.zip"
is_zip = True
elif system == "darwin":
filename = f"node-v{NODE_VERSION}-darwin-{arch}.tar.gz"
is_zip = False
elif system == "linux":
filename = f"node-v{NODE_VERSION}-linux-{arch}.tar.xz"
is_zip = False
else:
raise ValueError(f"Unsupported platform: {system}")
url = f"https://nodejs.org/dist/v{NODE_VERSION}/{filename}"
return url, filename, is_zip
def download_and_extract_node():
"""Download and extract Node.js binary"""
url, filename, is_zip = get_node_download_info()
download_path = TOOLS_DIR / filename
# Get architecture for extraction path
machine = platform.machine().lower()
if machine in ["x86_64", "amd64"]:
arch = "x64"
elif machine in ["aarch64", "arm64"]:
arch = "arm64"
else:
raise ValueError(f"Unsupported architecture: {machine}")
print(f"Downloading Node.js v{NODE_VERSION}...")
TOOLS_DIR.mkdir(exist_ok=True)
NODE_DIR.mkdir(exist_ok=True)
if not download_path.exists():
print(f"Downloading from: {url}")
with httpx.stream("GET", url, follow_redirects=True) as response:
response.raise_for_status()
with open(download_path, "wb") as f:
for chunk in response.iter_bytes(chunk_size=8192):
f.write(chunk)
print(f"SUCCESS: Downloaded {filename}")
# Extract Node.js
if platform.system() == "Windows":
node_exe = NODE_DIR / "node.exe"
else:
node_exe = NODE_DIR / "bin" / "node"
if not node_exe.exists():
print("Extracting Node.js...")
if is_zip:
with zipfile.ZipFile(download_path, "r") as zip_ref:
zip_ref.extractall(NODE_DIR)
# Move contents from nested folder to NODE_DIR
nested_dir = NODE_DIR / f"node-v{NODE_VERSION}-win-{arch}"
if nested_dir.exists():
for item in nested_dir.iterdir():
if item.is_dir():
# For directories, move contents recursively
target_dir = NODE_DIR / item.name
if target_dir.exists():
shutil.rmtree(target_dir)
shutil.move(str(item), str(target_dir))
else:
# For files, move directly
target_file = NODE_DIR / item.name
if target_file.exists():
target_file.unlink()
item.rename(target_file)
nested_dir.rmdir()
else:
with tarfile.open(download_path, "r:*") as tar_ref:
tar_ref.extractall(NODE_DIR)
# Move contents from nested folder to NODE_DIR
nested_dir = (
NODE_DIR / f"node-v{NODE_VERSION}-{platform.system().lower()}-x64"
)
if nested_dir.exists():
for item in nested_dir.iterdir():
item.rename(NODE_DIR / item.name)
nested_dir.rmdir()
print("SUCCESS: Node.js extracted")
# Clean up download file
if download_path.exists():
download_path.unlink()
return node_exe
def setup_eslint():
"""Install ESLint and create configuration"""
if platform.system() == "Windows":
node_exe = NODE_DIR / "node.exe"
npm_exe = NODE_DIR / "npm.cmd"
else:
node_exe = NODE_DIR / "bin" / "node"
npm_exe = NODE_DIR / "bin" / "npm"
print("Installing ESLint...")
# Create package.json
package_json = {
"name": "fastled-js-linting",
"version": "1.0.0",
"private": True,
"dependencies": {"eslint": "^8.56.0"},
}
with open(TOOLS_DIR / "package.json", "w") as f:
json.dump(package_json, f, indent=2)
# Install ESLint
try:
# Use absolute path for npm executable
npm_exe_abs = (
(TOOLS_DIR / "node" / "npm.cmd").resolve()
if platform.system() == "Windows"
else (TOOLS_DIR / "node" / "bin" / "npm").resolve()
)
# On Windows, use shell=True to properly execute .cmd files
if platform.system() == "Windows":
result = subprocess.run(
[str(npm_exe_abs), "install"],
cwd=TOOLS_DIR,
check=True,
capture_output=True,
text=True,
shell=True,
)
else:
result = subprocess.run(
[str(npm_exe_abs), "install"],
cwd=TOOLS_DIR,
check=True,
capture_output=True,
text=True,
)
# npm install succeeded
except subprocess.CalledProcessError as e:
# Show stderr output for debugging
print(f"npm install stderr: {e.stderr}")
print(f"npm install stdout: {e.stdout}")
raise
# ESLint config is now in ci/.eslintrc.js (tracked in git)
# No need to create it here since it's already in version control
print("SUCCESS: ESLint configured")
def create_fast_lint_script():
"""Create fast linting script"""
if platform.system() == "Windows":
node_exe = NODE_DIR / "node.exe"
eslint_exe = TOOLS_DIR / "node_modules" / ".bin" / "eslint.cmd"
else:
node_exe = NODE_DIR / "bin" / "node"
eslint_exe = TOOLS_DIR / "node_modules" / ".bin" / "eslint"
script_content = f"""#!/bin/bash
# FastLED JavaScript Linting Script (Node.js + ESLint - FAST!)
# Colors for output
RED='\\033[0;31m'
GREEN='\\033[0;32m'
YELLOW='\\033[1;33m'
BLUE='\\033[0;34m'
NC='\\033[0m' # No Color
echo -e "${{BLUE}}FAST FastLED JavaScript Linting (Node.js + ESLint - FAST!)${{NC}}"
# Check if ESLint is installed
if [ ! -f ".cache/js-tools/node_modules/.bin/eslint{".cmd" if platform.system() == "Windows" else ""}" ]; then
echo -e "${{RED}}ERROR: ESLint not found. Run: uv run ci/setup-js-linting-fast.py${{NC}}"
exit 1
fi
# Find JavaScript files in WASM platform
JS_FILES=$(find src/platforms/wasm -name "*.js" -type f 2>/dev/null)
if [ -z "$JS_FILES" ]; then
echo -e "${{YELLOW}}WARNING: No JavaScript files found in src/platforms/wasm/${{NC}}"
exit 0
fi
echo -e "${{BLUE}}Found JavaScript files:${{NC}}"
echo "$JS_FILES" | sed 's/^/ /'
# Run ESLint
echo -e "${{BLUE}}Running ESLint...${{NC}}"
cd .cache/js-tools
if "./node_modules/.bin/eslint{".cmd" if platform.system() == "Windows" else ""}" --no-eslintrc --no-inline-config -c .eslintrc.js ../../src/platforms/wasm/compiler/*.js ../../src/platforms/wasm/compiler/modules/*.js; then
echo -e "${{GREEN}}SUCCESS: JavaScript linting completed successfully${{NC}}"
else
echo -e "${{RED}}ERROR: JavaScript linting failed${{NC}}"
exit 1
fi
"""
# Lint script is now in ci/lint-js-fast (tracked in git)
# No need to create it here since it's already in version control
script_path = Path("ci/lint-js-fast")
# Script already exists in ci/ and is tracked in git
# Just ensure it's executable
if platform.system() != "Windows" and script_path.exists():
os.chmod(script_path, 0o755)
print("SUCCESS: Fast lint script created")
def main():
"""Main setup function"""
print("Setting up fast JavaScript linting (Node.js + ESLint)...")
try:
node_exe = download_and_extract_node()
setup_eslint()
create_fast_lint_script()
print("\\nFast JavaScript linting setup complete!")
print("\\nUsage:")
print(" bash ci/lint-js-fast # Fast linting with ESLint")
print(" For more info: ci/js/README.md")
except Exception as e:
print(f"Setup failed: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/setup-llvm.py
+969
View File
@@ -0,0 +1,969 @@
#!/usr/bin/env python3
import _thread
import atexit
import json
import lzma
import os
import platform
import shutil
import subprocess
import sys
import tarfile
import tempfile
import urllib.error
import urllib.request
import zipfile
from pathlib import Path
from typing import Any, Dict, List, Optional, Protocol, Tuple, cast
from ci.util.resumable_downloader import ResumableDownloader
class _SysConf(Protocol):
def short_os_name_and_version(self) -> str: ...
architecture: str
import sys_detection
from llvm_installer import LlvmInstaller
_HERE = Path(__file__).parent
_PROJECT_ROOT = _HERE.parent
# Global download directory - lazily initialized
_download_dir: Optional[Path] = None
def _get_download_dir() -> Path:
"""Get or create the download directory lazily."""
global _download_dir
if _download_dir is None:
_download_dir = _PROJECT_ROOT / ".cache" / "downloads"
_download_dir.mkdir(parents=True, exist_ok=True)
print(f"Created download directory: {_download_dir}")
# Register cleanup on exit as fallback
atexit.register(_cleanup_download_dir)
return _download_dir
def _cleanup_download_dir() -> None:
"""Clean up the download directory."""
global _download_dir
if _download_dir and _download_dir.exists():
try:
shutil.rmtree(_download_dir)
print(f"Cleaned up download directory: {_download_dir}")
except Exception as e:
print(
f"Warning: Failed to clean up download directory {_download_dir}: {e}"
)
_download_dir = None
def _download(url: str, filename: str) -> Path:
"""Download to the managed download directory.
Args:
url: URL to download
filename: Filename to save as (not full path)
Returns:
Path to the downloaded file
"""
download_dir = _get_download_dir()
file_path = download_dir / filename
print(f"download {url} to {file_path}")
# Use resumable downloader for large files
downloader = ResumableDownloader(chunk_size=1024 * 1024) # 1MB chunks
try:
downloader.download(url, file_path)
except Exception as e:
# Print download failure banner
url_truncated = url[:100]
banner_width = max(35, len(url_truncated) + 6)
banner_line = "#" * banner_width
print(f"\n{banner_line}")
print(
f"# DOWNLOAD FAILED FOR {url_truncated}{'...' if len(url) > 100 else ''} #"
)
print(f"{banner_line}")
print(f"Error: {e}")
print()
raise
return file_path
def is_linux_or_unix() -> Tuple[bool, str]:
"""Check if running on Linux or Unix-like system."""
system = platform.system().lower()
if system == "linux":
return True, "linux"
elif system in ["darwin", "freebsd", "openbsd", "netbsd"]:
return True, system
else:
return False, system
def get_tool_version(tool_name: str) -> Optional[str]:
"""Get version of a tool if it exists on system PATH."""
try:
result = subprocess.run(
[tool_name, "--version"], capture_output=True, text=True, timeout=5
)
if result.returncode == 0:
return result.stdout.strip()
return None
except (subprocess.TimeoutExpired, FileNotFoundError, subprocess.SubprocessError):
return None
def get_clang_version_number(version_output: str) -> Optional[int]:
"""Extract major version number from clang version output."""
try:
# Look for "clang version X.Y.Z" pattern
import re
match = re.search(r"clang version (\d+)\.", version_output)
if match:
return int(match.group(1))
return None
except (ValueError, AttributeError):
return None
def find_tool_path(tool_name: str) -> Optional[Path]:
"""Find the full path to a tool if it exists."""
tool_path = shutil.which(tool_name)
return Path(tool_path) if tool_path else None
def check_required_tools() -> Tuple[
Dict[str, Optional[Path]], Dict[str, Optional[Path]]
]:
"""Check for essential and extra LLVM/Clang tools on system PATH.
On Windows, prefer lld-link as the linker front-end.
"""
system_name = platform.system().lower()
essential_tools: List[str] = [
"clang",
"clang++",
"llvm-ar",
"llvm-nm",
"llvm-objdump",
"llvm-addr2line",
"llvm-strip",
"llvm-objcopy",
]
# Linker tools differ across platforms
if system_name == "windows":
essential_tools.append("lld-link")
else:
essential_tools.append("lld")
# Useful developer tools (not strictly required for building)
extra_tools: List[str] = [
"clangd",
"clang-format",
"clang-tidy",
"lldb",
]
found_essential: Dict[str, Optional[Path]] = {}
for tool in essential_tools:
found_essential[tool] = find_tool_path(tool)
found_extras: Dict[str, Optional[Path]] = {}
for tool in extra_tools:
found_extras[tool] = find_tool_path(tool)
return found_essential, found_extras
def create_hard_links(source_tools: Dict[str, Path], target_dir: Path) -> None:
"""Create hard links from system tools to target directory."""
bin_dir = target_dir / "bin"
bin_dir.mkdir(parents=True, exist_ok=True)
linked_count: int = 0
copied_count: int = 0
failed_tools: list[str] = []
for tool_name, source_path in source_tools.items():
if source_path and source_path.exists():
# Preserve original filename to keep extensions on Windows (e.g., .exe)
target_filename = source_path.name
target_path = bin_dir / target_filename
try:
# Remove existing file if it exists
if target_path.exists():
target_path.unlink()
# Create hard link
target_path.hardlink_to(source_path)
print(f"[OK] Linked {tool_name}: {source_path} -> {target_path}")
linked_count += 1
except (OSError, PermissionError) as e:
print(f"[WARN] Failed to link {tool_name}: {e}")
# Fallback to copy if hard link fails
try:
shutil.copy2(source_path, target_path)
target_path.chmod(0o755)
print(f"[OK] Copied {tool_name}: {source_path} -> {target_path}")
copied_count += 1
except (OSError, PermissionError):
print(f"[ERROR] Failed to copy {tool_name}")
failed_tools.append(tool_name)
total_prepared: int = linked_count + copied_count
print(
f"Summary: prepared {total_prepared} tools (linked: {linked_count}, copied: {copied_count})"
)
if failed_tools:
print(f"ERROR: Failed to mirror tools: {', '.join(failed_tools)}")
def find_tools_in_directories(
tool_names: List[str], directories: List[Path]
) -> Dict[str, Optional[Path]]:
"""Search for tools across a list of directories, returning first matches.
On Windows, will match executables with .exe suffix.
"""
found: Dict[str, Optional[Path]] = {}
for tool in tool_names:
found[tool] = None
for directory in directories:
if not directory.exists():
continue
for tool in tool_names:
if found[tool] is not None:
continue
candidate: Path = directory / tool
if candidate.exists():
found[tool] = candidate
continue
# Try platform-specific suffix
if platform.system().lower() == "windows":
exe_candidate: Path = directory / f"{tool}.exe"
if exe_candidate.exists():
found[tool] = exe_candidate
return found
def get_breadcrumb_file(target_dir: Path) -> Path:
"""Get the path to the installation completion breadcrumb file."""
cache_dir = target_dir / ".cache" / "cc"
cache_dir.mkdir(parents=True, exist_ok=True)
return cache_dir / "done.txt"
def create_breadcrumb(target_dir: Path) -> None:
"""Create breadcrumb file to mark successful installation."""
breadcrumb = get_breadcrumb_file(target_dir)
breadcrumb.write_text(f"Installation completed at {os.path.basename(target_dir)}\n")
print(f"Created installation breadcrumb: {breadcrumb}")
def check_existing_installation(target_dir: Path) -> Tuple[bool, bool]:
"""Check if there's an existing installation and if it's complete.
Returns:
(has_artifacts, has_breadcrumb) - bool tuple indicating installation state
"""
breadcrumb = get_breadcrumb_file(target_dir)
has_breadcrumb = breadcrumb.exists()
# Check for any installation artifacts
bin_dir = target_dir / "bin"
has_artifacts = bin_dir.exists() and any(bin_dir.iterdir())
return has_artifacts, has_breadcrumb
def verify_clang_installation(target_dir: Path) -> bool:
"""Test if clang is properly installed in target directory."""
bin_dir = target_dir / "bin"
clang_exe = (
bin_dir / "clang.exe"
if platform.system().lower() == "windows"
else bin_dir / "clang"
)
if not clang_exe.exists():
return False
# Test that clang can run and report version
try:
result = subprocess.run(
[str(clang_exe), "--version"], capture_output=True, text=True, timeout=5
)
return result.returncode == 0
except (subprocess.TimeoutExpired, FileNotFoundError, subprocess.SubprocessError):
return False
def get_windows_default_llvm_bins() -> List[Path]:
"""Return common Windows LLVM bin directories to probe."""
roots: List[Path] = []
program_files: Optional[str] = os.environ.get("ProgramFiles")
program_files_x86: Optional[str] = os.environ.get("ProgramFiles(x86)")
chocolatey_lib: Optional[str] = os.environ.get("ChocolateyInstall")
scoop: Optional[str] = os.environ.get("SCOOP")
if program_files:
roots.append(Path(program_files) / "LLVM" / "bin")
if program_files_x86:
roots.append(Path(program_files_x86) / "LLVM" / "bin")
if chocolatey_lib:
roots.append(Path(chocolatey_lib) / "lib" / "llvm" / "tools" / "llvm" / "bin")
if scoop:
roots.append(Path(scoop) / "apps" / "llvm" / "current" / "bin")
# Also consider PATH entries explicitly
path_entries: List[str] = os.environ.get("PATH", "").split(os.pathsep)
for entry in path_entries:
# Only consider entries that look like LLVM bin directories
if "llvm" in entry.lower() and "bin" in entry.lower():
roots.append(Path(entry))
return roots
def _run_subprocess(args: List[str], timeout_seconds: int) -> tuple[int, str, str]:
"""Run a subprocess command and return (returncode, stdout, stderr)."""
try:
result = subprocess.run(
args,
capture_output=True,
text=True,
timeout=timeout_seconds,
)
return result.returncode, result.stdout, result.stderr
except subprocess.TimeoutExpired as e:
return 124, "", f"Timeout after {timeout_seconds}s: {e}"
except subprocess.SubprocessError as e:
return 1, "", f"Subprocess error: {e}"
def install_llvm_windows_direct(target_dir: Path) -> Path:
"""Download and extract official LLVM Windows tar.xz archive into target_dir.
Returns the extracted LLVM bin directory path on success.
Fails fast with a descriptive error on failure.
"""
# Prefer dynamically discovered releases; as fallback, try a small curated list
preferred_versions: List[str] = [
"20.1.8",
"20.1.7",
"20.1.6",
"19.1.8",
"19.1.7",
"19.1.6",
]
download_errors: List[str] = []
# First, try to discover the latest available Windows tar.xz for majors 20 and 19
for probe_major in [20, 19]:
print(f"Attempting discovery for LLVM major version {probe_major}...")
discovered_url = discover_latest_github_llvm_win_zip(probe_major)
if discovered_url is not None:
print(f"Discovered URL for LLVM {probe_major}: {discovered_url}")
# Put discovered URL at the front of attempts
target_dir.mkdir(parents=True, exist_ok=True)
extract_root: Path = target_dir
tmp_archive_path: Optional[Path] = None
try:
print(f"Attempting to download and extract LLVM {probe_major}...")
# Download to managed directory
tmp_archive_path = _download(
discovered_url, f"llvm-discovered-{probe_major}.tar.xz"
)
extract_root = target_dir
with lzma.open(tmp_archive_path, "rb") as xz_file:
with tarfile.open(fileobj=xz_file, mode="r") as tar:
tar.extractall(extract_root, filter="data")
except KeyboardInterrupt:
print("Operation interrupted by user")
_thread.interrupt_main()
raise
except Exception as e:
print(f"Failed to download/extract LLVM {probe_major}: {e}")
download_errors.append(f"discovered-{probe_major}: {e}")
finally:
# Clean up the downloaded file immediately
if tmp_archive_path and tmp_archive_path.exists():
try:
tmp_archive_path.unlink()
print(f"Cleaned up: {tmp_archive_path}")
except Exception as e:
print(f"Warning: Failed to clean up {tmp_archive_path}: {e}")
extracted_bin: Optional[Path] = None
for entry in extract_root.iterdir():
if not entry.is_dir():
continue
candidate = entry / "bin" / "clang.exe"
if candidate.exists():
extracted_bin = entry / "bin"
break
if extracted_bin is not None:
print(
f"SUCCESS: LLVM {probe_major} installed successfully via discovery!"
)
print("OK: Successfully installed LLVM (Windows tar.xz, discovered)")
print(f" Installation directory: {extracted_bin.parent}")
print(f" Bin directory: {extracted_bin}")
create_breadcrumb(target_dir)
return extracted_bin
else:
print(
f"WARNING: LLVM {probe_major} downloaded but clang.exe not found in expected location"
)
else:
print(f"No URL discovered for LLVM major version {probe_major}")
# If discovery did not succeed, try curated versions list
print("\n" + "=" * 60)
print("Discovery failed for all major versions. Falling back to curated list...")
print("=" * 60 + "\n")
for version in preferred_versions:
base_url = (
"https://github.com/llvm/llvm-project/releases/download/"
f"llvmorg-{version}/clang+llvm-{version}-x86_64-pc-windows-msvc.tar.xz"
)
print(f"Attempting direct LLVM download for Windows: {base_url}")
try:
target_dir.mkdir(parents=True, exist_ok=True)
tmp_archive_path: Optional[Path] = None
try:
# Download to managed directory
tmp_archive_path = _download(base_url, f"llvm-{version}.tar.xz")
except KeyboardInterrupt:
print("Operation interrupted by user")
_thread.interrupt_main()
raise
except Exception as e:
download_errors.append(f"{version}: {e}")
# Clean up on download error
if tmp_archive_path and tmp_archive_path.exists():
try:
tmp_archive_path.unlink()
print(f"Cleaned up after error: {tmp_archive_path}")
except Exception:
pass
continue
extract_root = target_dir
try:
with lzma.open(tmp_archive_path, "rb") as xz_file:
with tarfile.open(fileobj=xz_file, mode="r") as tar:
tar.extractall(extract_root, filter="data")
finally:
# Clean up the downloaded file immediately
if tmp_archive_path and tmp_archive_path.exists():
try:
tmp_archive_path.unlink()
print(f"Cleaned up: {tmp_archive_path}")
except Exception as e:
print(f"Warning: Failed to clean up {tmp_archive_path}: {e}")
# Find extracted directory that contains bin/clang.exe
extracted_bin: Optional[Path] = None
for entry in extract_root.iterdir():
if not entry.is_dir():
continue
candidate = entry / "bin" / "clang.exe"
if candidate.exists():
extracted_bin = entry / "bin"
break
if extracted_bin is None:
print(
"ERROR: LLVM downloaded but clang.exe not found in expected location"
)
# Try next version
continue
print("OK: Successfully installed LLVM (Windows tar.xz)")
print(f" Installation directory: {extracted_bin.parent}")
print(f" Bin directory: {extracted_bin}")
create_breadcrumb(target_dir)
return extracted_bin
except KeyboardInterrupt:
print("Operation interrupted by user")
_thread.interrupt_main()
raise
except Exception as e:
download_errors.append(f"{version}: {e}")
continue
# If we reach here, all attempts failed
print("CRITICAL: Failed to download LLVM for Windows from official releases.")
if download_errors:
for err in download_errors:
print(f" - {err}")
print("Please ensure internet access is available and retry.")
sys.exit(1)
def discover_latest_github_llvm_win_zip(major: int) -> Optional[str]:
"""Find the latest LLVM Windows tar.xz asset for a given major version via GitHub API.
Returns a browser_download_url string if found, otherwise None.
"""
try:
per_page: int = 50
page: int = 1
max_pages: int = 3
while page <= max_pages:
api_url: str = (
"https://api.github.com/repos/llvm/llvm-project/releases"
f"?per_page={per_page}&page={page}"
)
req = urllib.request.Request(
api_url, headers={"User-Agent": "fastled-setup-llvm"}
)
with urllib.request.urlopen(req, timeout=20) as resp:
raw: str = resp.read().decode("utf-8")
releases_any: Any = json.loads(raw)
releases: list[dict[str, Any]] = []
if isinstance(releases_any, list):
tmp_list: list[Any] = cast(list[Any], releases_any)
filtered: list[dict[str, Any]] = []
k: int = 0
while k < len(tmp_list):
item: Any = tmp_list[k]
k += 1
if isinstance(item, dict):
filtered.append(cast(dict[str, Any], item))
releases = filtered
# Iterate releases
idx: int = 0
while idx < len(releases):
rel: dict[str, Any] = releases[idx]
idx += 1
tag_val: Any = rel.get("tag_name")
tag_name: Optional[str] = tag_val if isinstance(tag_val, str) else None
if tag_name is None:
continue
# Expect tags like llvmorg-19.1.6
prefix: str = f"llvmorg-{major}."
if not tag_name.startswith(prefix):
continue
assets_val: Any = rel.get("assets")
assets: list[dict[str, Any]] = (
cast(list[dict[str, Any]], assets_val)
if isinstance(assets_val, list)
else []
)
filtered_assets: list[dict[str, Any]] = [
a for a in assets if isinstance(a, dict)
]
j: int = 0
while j < len(filtered_assets):
asset: dict[str, Any] = filtered_assets[j]
j += 1
name_val: Any = asset.get("name")
url_val: Any = asset.get("browser_download_url")
name: Optional[str] = (
name_val if isinstance(name_val, str) else None
)
url: Optional[str] = url_val if isinstance(url_val, str) else None
if name is not None and url is not None:
# Match e.g., clang+llvm-19.1.6-x86_64-pc-windows-msvc.tar.xz
if (
name.startswith(f"clang+llvm-{major}.")
and "x86_64-pc-windows-msvc.tar.xz" in name
):
return url
page += 1
return None
except urllib.error.URLError:
return None
def download_and_install_llvm(target_dir: Path) -> None:
"""Download and install LLVM using llvm-installer package."""
print("Detecting system configuration...")
try:
sys_conf_obj2: Any = sys_detection.local_sys_conf()
local_conf2: _SysConf = cast(_SysConf, sys_conf_obj2)
detected_os: str = local_conf2.short_os_name_and_version()
architecture: str = local_conf2.architecture
print(f"Detected OS: {detected_os}")
print(f"Architecture: {architecture}")
# Use Ubuntu 24.04 as fallback since it's known to work
# This package seems to be from YugabyteDB and has limited OS support
working_os: str = "ubuntu24.04"
if detected_os.startswith("ubuntu24") or detected_os.startswith("ubuntu22"):
working_os = detected_os
print(f"Using OS variant: {working_os}")
llvm_installer: Any = LlvmInstaller(
short_os_name_and_version=working_os, architecture=architecture
)
# Prefer most recent supported major release for linux tarball path
llvm_url: str = ""
selected_version: Optional[int] = None
for version in [20, 19]:
try:
print(f"Installing LLVM version {version}...")
llvm_url = llvm_installer.get_llvm_url(major_llvm_version=version)
print(f"Found LLVM {version} at: {llvm_url}")
selected_version = version
break
except KeyboardInterrupt:
print("Operation interrupted by user")
_thread.interrupt_main()
raise
except Exception as e:
print(f"WARN: Could not resolve LLVM {version}: {e}")
if not llvm_url:
raise RuntimeError(
"Could not resolve a suitable LLVM download URL for Linux"
)
# Download and extract
print("Downloading LLVM package...")
# Download to managed directory
tmp_archive_path: Path = _download(
llvm_url, f"llvm-linux-{selected_version or 'unknown'}.tar.gz"
)
try:
print("Extracting LLVM package...")
with tarfile.open(tmp_archive_path, "r:gz") as tar:
tar.extractall(target_dir, filter="data")
finally:
# Clean up immediately after extraction
try:
tmp_archive_path.unlink()
print(f"Cleaned up: {tmp_archive_path}")
except Exception as e:
print(f"Warning: Failed to clean up {tmp_archive_path}: {e}")
# Find the extracted directory and verify clang exists
for extracted_dir in target_dir.iterdir():
if extracted_dir.is_dir():
clang_path = extracted_dir / "bin" / "clang"
if clang_path.exists():
print(
f"OK: Successfully installed LLVM {selected_version if selected_version is not None else 'unknown'}"
)
print(f" Installation directory: {extracted_dir}")
print(f" Clang binary: {clang_path}")
# List available tools (first few) and provide PATH guidance
bin_dir = extracted_dir / "bin"
if bin_dir.exists():
tools = [
f.name
for f in bin_dir.iterdir()
if f.is_file() and not f.name.endswith(".dll")
]
print(
f" Available tools: {', '.join(sorted(tools[:10]))}{'...' if len(tools) > 10 else ''}"
)
print("")
print("Next steps:")
print(f" - Add to PATH: {bin_dir}")
print(" - Example:")
print(f' export PATH="{bin_dir}:$PATH"')
print(" - Verify:")
print(" clang --version")
return
print("ERROR: LLVM downloaded but clang not found in expected location")
sys.exit(1)
except KeyboardInterrupt:
print("Operation interrupted by user")
_thread.interrupt_main()
raise
except Exception as e:
print(f"Error during installation: {e}")
import traceback
traceback.print_exc()
sys.exit(1)
def main() -> None:
"""Setup LLVM toolchain - check system first, fallback to package installation."""
# Get target directory from command line (optional)
if len(sys.argv) > 2:
print("Usage: uv run python ci/setup-llvm.py [target_directory]")
print(" target_directory: Optional. Defaults to .cache/cc in project root")
sys.exit(1)
if len(sys.argv) == 2:
target_dir = Path(sys.argv[1])
else:
# Default to project root .cache/cc directory
target_dir = _PROJECT_ROOT / ".cache" / "cc"
print(f"Using default target directory: {target_dir}")
target_dir.mkdir(parents=True, exist_ok=True)
# Check platform compatibility
is_unix_like, platform_name = is_linux_or_unix()
if platform_name == "windows":
print("Checking for existing LLVM/Clang tools on system (Windows)...")
# Check for existing installation in target directory
has_artifacts, has_breadcrumb = check_existing_installation(target_dir)
if has_artifacts and not has_breadcrumb:
print("WARN: Found installation artifacts without completion marker")
print(" This suggests an incomplete or corrupted installation")
print(" Re-downloading LLVM to ensure proper installation...")
elif has_artifacts and has_breadcrumb:
if verify_clang_installation(target_dir):
print("OK: Valid LLVM installation found in target directory")
print(" Skipping download - clang installation verified")
return
else:
print(
"WARN: Installation breadcrumb exists but clang verification failed"
)
print(" Re-downloading LLVM to fix installation...")
clang_version_output = get_tool_version("clang")
if clang_version_output:
clang_version = get_clang_version_number(clang_version_output)
print(
f"Found system clang: version {clang_version if clang_version else 'unknown'}"
)
else:
clang_version = None
found_essential, found_extras = check_required_tools()
available_essential = {
name: path for name, path in found_essential.items() if path
}
missing_essential = [name for name, path in found_essential.items() if not path]
if len(available_essential) != len(found_essential):
# Probe common install directories (e.g., C:\Program Files\LLVM\bin)
probe_dirs = get_windows_default_llvm_bins()
print(
f"Probing default Windows LLVM locations: {', '.join(str(p) for p in probe_dirs)}"
)
probed = find_tools_in_directories(list(found_essential.keys()), probe_dirs)
for name, p in probed.items():
if p is not None and found_essential.get(name) is None:
found_essential[name] = p
available_essential = {
name: path for name, path in found_essential.items() if path
}
missing_essential = [
name for name, path in found_essential.items() if not path
]
if len(available_essential) == len(found_essential) and (
clang_version is None or clang_version < 19
):
print(
"Note: Detected toolchain but clang < 19. Continuing; ensure compatibility with your build settings."
)
# On Windows, require core tools including llvm-addr2line for stack traces
core_required = ["clang", "clang++", "lld-link", "llvm-addr2line"]
missing_core = [name for name in core_required if not found_essential.get(name)]
if not missing_core:
print("OK: Core tools present - proceeding with available tools")
all_available_tools: Dict[str, Path] = {}
for name, path in found_essential.items():
if path is not None:
all_available_tools[name] = cast(Path, path)
for name, path in found_extras.items():
if path is not None:
all_available_tools[name] = cast(Path, path)
print(f"Creating hard links in {target_dir}...")
create_hard_links(all_available_tools, target_dir)
version_file = target_dir / "VERSION"
version_file.write_text(
f"system-clang-{clang_version if clang_version else 'unknown'}\n"
)
bin_dir = target_dir / "bin"
existing = [p for p in bin_dir.iterdir() if p.is_file()]
print(f"OK: Tools available in {bin_dir}: {len(existing)} files")
create_breadcrumb(target_dir)
# Warn about missing non-core tools
missing_noncore = [
name
for name in list(found_essential.keys())
if name not in core_required and not found_essential.get(name)
]
if missing_noncore:
print(f"WARN: Missing non-core tools: {', '.join(missing_noncore)}")
missing_extras = [name for name, path in found_extras.items() if not path]
if missing_extras:
print(f"WARN: Missing extra tools: {', '.join(missing_extras)}")
return
# Try to auto-install on Windows via direct GitHub download
print(f"ERROR: Missing core tools: {', '.join(missing_core)}")
print("Downloading LLVM for Windows from GitHub releases...")
extracted_bin_dir = install_llvm_windows_direct(target_dir)
# Collect tools from the extracted bin directory and link/copy them
tool_names: List[str] = [
"clang",
"clang++",
"lld-link",
"llvm-addr2line",
"llvm-ar",
"llvm-nm",
"llvm-objdump",
"llvm-strip",
"llvm-objcopy",
"clangd",
"clang-format",
"clang-tidy",
"lldb",
]
discovered: Dict[str, Optional[Path]] = find_tools_in_directories(
tool_names, [extracted_bin_dir]
)
available_tools: Dict[str, Path] = {}
for name, p in discovered.items():
if p is not None:
available_tools[name] = cast(Path, p)
if not available_tools:
print(
"CRITICAL: Extracted LLVM bin directory does not contain expected tools."
)
sys.exit(1)
print(f"Creating hard links in {target_dir}...")
create_hard_links(available_tools, target_dir)
version_file = target_dir / "VERSION"
version_file.write_text("downloaded-llvm-windows\n")
bin_dir = target_dir / "bin"
existing = [p for p in bin_dir.iterdir() if p.is_file()]
print(f"OK: Tools available in {bin_dir}: {len(existing)} files")
create_breadcrumb(target_dir)
return
print("Checking for existing LLVM/Clang tools on system...")
# Check if clang is available and get its version
clang_version_output = get_tool_version("clang")
if clang_version_output:
clang_version = get_clang_version_number(clang_version_output)
print(
f"Found system clang: version {clang_version if clang_version else 'unknown'}"
)
if clang_version and clang_version >= 19:
print("OK: System clang 19+ detected - checking tools")
# Check for essential and extra tools
found_essential, found_extras = check_required_tools()
available_essential = {
name: path for name, path in found_essential.items() if path
}
missing_essential = [
name for name, path in found_essential.items() if not path
]
available_extras = {
name: path for name, path in found_extras.items() if path
}
missing_extras = [name for name, path in found_extras.items() if not path]
# Check if all essential tools are present
if len(available_essential) == len(found_essential):
print("OK: All essential tools found - using system tools")
# Combine available tools for linking
all_available_tools = {**available_essential, **available_extras}
print(f"Creating hard links in {target_dir}...")
create_hard_links(all_available_tools, target_dir)
# Create a simple version file
version_file = target_dir / "VERSION"
version_file.write_text(f"system-clang-{clang_version}\n")
print(
f"OK: Successfully linked {len(all_available_tools)} system tools"
)
# Warn about missing extras in one line
if missing_extras:
print(f"WARN: Missing extra tools: {', '.join(missing_extras)}")
return
else:
# Missing essential tools - need to install LLVM
print(f"ERROR: Missing essential tools: {', '.join(missing_essential)}")
if platform_name == "linux":
print("Installing LLVM package to provide missing tools...")
download_and_install_llvm(target_dir)
return
else:
print(
f"WARN: Cannot auto-install on {platform_name} - please install manually"
)
return
else:
print(f"System clang version {clang_version} < 19")
if platform_name == "linux":
print("Installing LLVM 18 package...")
download_and_install_llvm(target_dir)
return
else:
print(
f"WARN: Cannot auto-install on {platform_name} - please install manually"
)
return
else:
print("No system clang found")
if platform_name == "linux":
print("Installing LLVM package...")
download_and_install_llvm(target_dir)
else:
print(
f"WARN: Cannot auto-install on {platform_name} - please install manually"
)
if __name__ == "__main__":
try:
main()
finally:
# Clean up download directory at the end
_cleanup_download_dir()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/symbol_analysis_runner.py
+74
View File
@@ -0,0 +1,74 @@
#!/usr/bin/env python3
"""
Generic Symbol Analysis Runner for GitHub Actions
Wrapper script to run symbol analysis with proper error handling for any platform
"""
import argparse
import sys
from pathlib import Path
from ci.util.symbol_analysis import main as symbol_analysis
def main():
parser = argparse.ArgumentParser(
description="Run symbol analysis for any platform with GitHub Actions support"
)
parser.add_argument("--board", required=True, help="Board name being analyzed")
parser.add_argument("--example", default="Blink", help="Example that was compiled")
parser.add_argument(
"--skip-on-failure",
action="store_true",
help="Don't fail the build if symbol analysis fails",
)
args = parser.parse_args()
print(f"Symbol Analysis Runner - Board: {args.board}, Example: {args.example}")
# Check if build_info.json exists
build_info_path = Path(".build") / args.board / "build_info.json"
if not build_info_path.exists():
# Fall back to nested pio structure
build_info_path = Path(".build") / "pio" / args.board / "build_info.json"
if not build_info_path.exists():
message = (
f"Build info not found at {build_info_path}. Skipping symbol analysis."
)
if args.skip_on_failure:
print(f"Warning: {message}")
return 0
else:
print(f"Error: {message}")
return 1
try:
# Import and run the generic symbol analysis
print("Running symbol analysis...")
# Override sys.argv to pass the board argument to the symbol analysis script
original_argv = sys.argv
sys.argv = ["symbol_analysis.py", "--board", args.board]
try:
symbol_analysis()
print("Symbol analysis completed successfully!")
return 0
finally:
sys.argv = original_argv
except ImportError as e:
print(f"Failed to import symbol_analysis module: {e}")
if args.skip_on_failure:
return 0
return 1
except Exception as e:
print(f"Symbol analysis failed: {e}")
if args.skip_on_failure:
return 0
return 1
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/test_integration/test_archive_creation.py
+1155
View File
File diff suppressed because it is too large. Load diff
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/test_integration/test_xcache.py
+464
View File
@@ -0,0 +1,464 @@
#!/usr/bin/env python3
"""
Integration tests for xcache.py using real clang compiler.
This test suite verifies that xcache works correctly with actual compilation
scenarios, including response files and direct execution modes.
"""
import os
import shutil
import subprocess
import sys
import tempfile
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
@dataclass
class TestResults:
"""Test results container."""
passed: int = 0
failed: int = 0
errors: Optional[list[str]] = None
def __post_init__(self) -> None:
if self.errors is None:
self.errors = []
def find_clang() -> Optional[str]:
"""Find clang compiler in PATH."""
clang_path = shutil.which("clang")
if clang_path:
return clang_path
# Check common locations
common_paths = [
"/usr/bin/clang",
"/usr/local/bin/clang",
"/opt/local/bin/clang",
"C:/Program Files/LLVM/bin/clang.exe",
"C:/msys64/mingw64/bin/clang.exe",
]
for path in common_paths:
if os.path.isfile(path) and os.access(path, os.X_OK):
return path
return None
def find_sccache() -> Optional[str]:
"""Find sccache in PATH."""
return shutil.which("sccache")
def create_test_source_file(temp_dir: Path) -> Path:
"""Create a simple C source file for testing."""
source_file = temp_dir / "test.c"
source_content = """
#include <stdio.h>
int main() {
printf("Hello from xcache test!\\n");
return 0;
}
"""
source_file.write_text(source_content)
return source_file
def create_response_file(temp_dir: Path, args: list[str]) -> Path:
"""Create a response file with the given arguments."""
response_file = temp_dir / "compile_args.rsp"
# Write arguments to response file (one per line or space-separated)
content = " ".join(args)
response_file.write_text(content)
return response_file
def test_xcache_direct_execution(results: TestResults) -> None:
"""Test xcache with direct execution (no response files)."""
print("🧪 Testing xcache direct execution...")
clang_path = find_clang()
if not clang_path:
results.errors.append(
"clang compiler not found - skipping direct execution test"
)
return
sccache_path = find_sccache()
if not sccache_path:
results.errors.append("sccache not found - skipping direct execution test")
return
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create test source file
source_file = create_test_source_file(temp_path)
output_file = temp_path / "test_direct.o"
# Test xcache with direct arguments (no response files)
env = os.environ.copy()
env["XCACHE_DEBUG"] = "1"
cmd = [
sys.executable,
"ci/util/xcache.py",
clang_path,
"-c",
str(source_file),
"-o",
str(output_file),
]
try:
result = subprocess.run(
cmd, capture_output=True, text=True, env=env, timeout=30
)
if result.returncode == 0 and output_file.exists():
print(" ✅ Direct execution test passed")
print(f" Output file created: {output_file.stat().st_size} bytes")
results.passed += 1
else:
print(
f" ❌ Direct execution test failed (return code: {result.returncode})"
)
print(f" Stderr: {result.stderr}")
results.failed += 1
results.errors.append(f"Direct execution failed: {result.stderr}")
except subprocess.TimeoutExpired:
print(" ❌ Direct execution test timed out")
results.failed += 1
results.errors.append("Direct execution test timed out")
except Exception as e:
print(f" ❌ Direct execution test error: {e}")
results.failed += 1
results.errors.append(f"Direct execution error: {e}")
def test_xcache_response_file_execution(results: TestResults) -> None:
"""Test xcache with response file execution."""
print("\n🧪 Testing xcache response file execution...")
clang_path = find_clang()
if not clang_path:
results.errors.append("clang compiler not found - skipping response file test")
return
sccache_path = find_sccache()
if not sccache_path:
results.errors.append("sccache not found - skipping response file test")
return
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create test source file
source_file = create_test_source_file(temp_path)
output_file = temp_path / "test_response.o"
# Create response file with compilation arguments
compile_args = ["-c", str(source_file), "-o", str(output_file), "-O2", "-Wall"]
response_file = create_response_file(temp_path, compile_args)
# Test xcache with response file
env = os.environ.copy()
env["XCACHE_DEBUG"] = "1"
cmd = [sys.executable, "ci/util/xcache.py", clang_path, f"@{response_file}"]
try:
result = subprocess.run(
cmd, capture_output=True, text=True, env=env, timeout=30
)
if result.returncode == 0 and output_file.exists():
print(" ✅ Response file execution test passed")
print(f" Output file created: {output_file.stat().st_size} bytes")
print(f" Response file handled: @{response_file}")
# Verify that response file was passed through (not expanded)
if f"@{response_file}" in result.stderr:
print(" ✅ Response file passed through correctly")
else:
print(
" ⚠️ Response file passthrough not clearly visible in debug output"
)
results.passed += 1
else:
print(
f" ❌ Response file execution test failed (return code: {result.returncode})"
)
print(f" Stderr: {result.stderr}")
results.failed += 1
results.errors.append(
f"Response file execution failed: {result.stderr}"
)
except subprocess.TimeoutExpired:
print(" ❌ Response file execution test timed out")
results.failed += 1
results.errors.append("Response file execution test timed out")
except Exception as e:
print(f" ❌ Response file execution test error: {e}")
results.failed += 1
results.errors.append(f"Response file execution error: {e}")
def test_xcache_mixed_arguments(results: TestResults) -> None:
"""Test xcache with mixed regular arguments and response files."""
print("\n🧪 Testing xcache mixed arguments...")
clang_path = find_clang()
if not clang_path:
results.errors.append(
"clang compiler not found - skipping mixed arguments test"
)
return
sccache_path = find_sccache()
if not sccache_path:
results.errors.append("sccache not found - skipping mixed arguments test")
return
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create test source file
source_file = create_test_source_file(temp_path)
output_file = temp_path / "test_mixed.o"
# Create response file with some arguments
response_args = ["-O2", "-Wall", "-Wextra"]
response_file = create_response_file(temp_path, response_args)
# Test xcache with mixed arguments
env = os.environ.copy()
env["XCACHE_DEBUG"] = "1"
cmd = [
sys.executable,
"ci/util/xcache.py",
clang_path,
"-c",
str(source_file),
f"@{response_file}",
"-o",
str(output_file),
]
try:
result = subprocess.run(
cmd, capture_output=True, text=True, env=env, timeout=30
)
if result.returncode == 0 and output_file.exists():
print(" ✅ Mixed arguments test passed")
print(f" Output file created: {output_file.stat().st_size} bytes")
print(f" Mixed args: regular + @{response_file}")
results.passed += 1
else:
print(
f" ❌ Mixed arguments test failed (return code: {result.returncode})"
)
print(f" Stderr: {result.stderr}")
results.failed += 1
results.errors.append(f"Mixed arguments failed: {result.stderr}")
except subprocess.TimeoutExpired:
print(" ❌ Mixed arguments test timed out")
results.failed += 1
results.errors.append("Mixed arguments test timed out")
except Exception as e:
print(f" ❌ Mixed arguments test error: {e}")
results.failed += 1
results.errors.append(f"Mixed arguments error: {e}")
def test_xcache_cache_effectiveness(results: TestResults) -> None:
"""Test that xcache actually caches compilation results."""
print("\n🧪 Testing xcache cache effectiveness...")
clang_path = find_clang()
if not clang_path:
results.errors.append(
"clang compiler not found - skipping cache effectiveness test"
)
return
sccache_path = find_sccache()
if not sccache_path:
results.errors.append("sccache not found - skipping cache effectiveness test")
return
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create test source file
source_file = create_test_source_file(temp_path)
# Set up environment for caching
env = os.environ.copy()
env["XCACHE_DEBUG"] = "1"
cache_dir = temp_path / "sccache"
cache_dir.mkdir()
env["SCCACHE_DIR"] = str(cache_dir)
# First compilation (should be cache miss)
output_file1 = temp_path / "test_cache1.o"
cmd1 = [
sys.executable,
"ci/util/xcache.py",
clang_path,
"-c",
str(source_file),
"-o",
str(output_file1),
]
# Second compilation (should be cache hit)
output_file2 = temp_path / "test_cache2.o"
cmd2 = [
sys.executable,
"ci/util/xcache.py",
clang_path,
"-c",
str(source_file),
"-o",
str(output_file2),
]
try:
# First compilation
result1 = subprocess.run(
cmd1, capture_output=True, text=True, env=env, timeout=30
)
# Second compilation
result2 = subprocess.run(
cmd2, capture_output=True, text=True, env=env, timeout=30
)
if (
result1.returncode == 0
and output_file1.exists()
and result2.returncode == 0
and output_file2.exists()
):
print(" ✅ Cache effectiveness test passed")
print(f" First compilation: {output_file1.stat().st_size} bytes")
print(f" Second compilation: {output_file2.stat().st_size} bytes")
# Check if sccache cache directory has content
cache_files = list(cache_dir.rglob("*"))
if cache_files:
print(f" Cache directory has {len(cache_files)} entries")
else:
print(" ⚠️ Cache directory appears empty")
results.passed += 1
else:
print(f" ❌ Cache effectiveness test failed")
print(
f" First result: {result1.returncode}, {output_file1.exists()}"
)
print(
f" Second result: {result2.returncode}, {output_file2.exists()}"
)
results.failed += 1
results.errors.append("Cache effectiveness test failed")
except subprocess.TimeoutExpired:
print(" ❌ Cache effectiveness test timed out")
results.failed += 1
results.errors.append("Cache effectiveness test timed out")
except Exception as e:
print(f" ❌ Cache effectiveness test error: {e}")
results.failed += 1
results.errors.append(f"Cache effectiveness error: {e}")
def run_all_tests() -> TestResults:
"""Run all xcache tests."""
print("🚀 XCACHE INTEGRATION TESTS WITH CLANG")
print("=" * 60)
results = TestResults()
# Check prerequisites
clang_path = find_clang()
sccache_path = find_sccache()
print(f"📋 Test Environment:")
print(f" Clang: {clang_path or 'NOT FOUND'}")
print(f" Sccache: {sccache_path or 'NOT FOUND'}")
print(f" xcache: ci/util/xcache.py")
print()
if not clang_path:
print("❌ Clang compiler not found - install clang to run tests")
results.errors.append("Clang compiler not available")
return results
if not sccache_path:
print("❌ Sccache not found - install sccache to run tests")
results.errors.append("Sccache not available")
return results
# Run tests
test_xcache_direct_execution(results)
test_xcache_response_file_execution(results)
test_xcache_mixed_arguments(results)
test_xcache_cache_effectiveness(results)
return results
def main() -> int:
"""Main test function."""
results = run_all_tests()
print("\n" + "=" * 60)
print("📊 TEST RESULTS SUMMARY")
print("=" * 60)
total_tests = results.passed + results.failed
if total_tests > 0:
success_rate = (results.passed / total_tests) * 100
print(f" Tests passed: {results.passed}")
print(f" Tests failed: {results.failed}")
print(f" Success rate: {success_rate:.1f}%")
else:
print(" No tests were executed")
if results.errors:
print(f"\n🚨 Errors encountered:")
for i, error in enumerate(results.errors, 1):
print(f" {i}. {error}")
if results.failed == 0 and results.passed > 0:
print("\n✅ All xcache tests passed! Ready for ESP32S3 integration.")
return 0
elif results.passed > 0:
print(
f"\n⚠️ Some tests passed but {results.failed} failed. Check errors above."
)
return 1
else:
print("\n❌ No tests passed. Check prerequisites and errors above.")
return 1
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/test_integration/test_xcache_standalone.py
+437
View File
@@ -0,0 +1,437 @@
#!/usr/bin/env python3
"""
Standalone tests for xcache.py using clang compiler without requiring sccache server.
This test suite verifies that xcache works correctly by testing the wrapper
functionality and ensuring proper command construction and execution.
"""
import os
import shutil
import subprocess
import sys
import tempfile
from dataclasses import dataclass
from pathlib import Path
from typing import Optional
@dataclass
class TestResults:
"""Test results container."""
passed: int = 0
failed: int = 0
errors: Optional[list[str]] = None
def __post_init__(self) -> None:
if self.errors is None:
self.errors = []
def find_clang() -> Optional[str]:
"""Find clang compiler in PATH."""
clang_path = shutil.which("clang")
if clang_path:
return clang_path
# Check common locations
common_paths = [
"/usr/bin/clang",
"/usr/local/bin/clang",
"/opt/local/bin/clang",
"C:/Program Files/LLVM/bin/clang.exe",
"C:/msys64/mingw64/bin/clang.exe",
]
for path in common_paths:
if os.path.isfile(path) and os.access(path, os.X_OK):
return path
return None
def create_test_source_file(temp_dir: Path) -> Path:
"""Create a simple C source file for testing."""
source_file = temp_dir / "test.c"
source_content = """
#include <stdio.h>
int main() {
printf("Hello from xcache test!\\n");
return 0;
}
"""
source_file.write_text(source_content)
return source_file
def create_response_file(temp_dir: Path, args: list[str]) -> Path:
"""Create a response file with the given arguments."""
response_file = temp_dir / "compile_args.rsp"
# Write arguments to response file (space-separated)
# Properly quote arguments that contain spaces (Windows paths)
# Use forward slashes for Windows paths to avoid backslash escape issues
quoted_args: list[str] = []
for arg in args:
# Convert Windows backslashes to forward slashes for response files
normalized_arg = arg.replace("\\", "/")
if " " in normalized_arg and not (
normalized_arg.startswith('"') and normalized_arg.endswith('"')
):
quoted_args.append(f'"{normalized_arg}"')
else:
quoted_args.append(normalized_arg)
content = " ".join(quoted_args)
response_file.write_text(content)
return response_file
def test_clang_direct_compilation(results: TestResults) -> None:
"""Test that clang compilation works directly (baseline test)."""
print("🧪 Testing direct clang compilation (baseline)...")
clang_path = find_clang()
if not clang_path:
results.errors.append("clang compiler not found - skipping baseline test")
return
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create test source file
source_file = create_test_source_file(temp_path)
output_file = temp_path / "test_baseline.o"
# Test direct clang compilation
cmd = [clang_path, "-c", str(source_file), "-o", str(output_file)]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
if result.returncode == 0 and output_file.exists():
print(" ✅ Direct clang compilation works")
print(f" Output file created: {output_file.stat().st_size} bytes")
results.passed += 1
else:
print(
f" ❌ Direct clang compilation failed (return code: {result.returncode})"
)
print(f" Stderr: {result.stderr}")
results.failed += 1
results.errors.append(
f"Direct clang compilation failed: {result.stderr}"
)
except subprocess.TimeoutExpired:
print(" ❌ Direct clang compilation timed out")
results.failed += 1
results.errors.append("Direct clang compilation timed out")
except Exception as e:
print(f" ❌ Direct clang compilation error: {e}")
results.failed += 1
results.errors.append(f"Direct clang compilation error: {e}")
def test_clang_response_file_compilation(results: TestResults) -> None:
"""Test that clang can handle response files directly."""
print("\n🧪 Testing clang response file compilation...")
clang_path = find_clang()
if not clang_path:
results.errors.append("clang compiler not found - skipping response file test")
return
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create test source file
source_file = create_test_source_file(temp_path)
output_file = temp_path / "test_response.o"
# Create response file with compilation arguments
compile_args = ["-c", str(source_file), "-o", str(output_file)]
response_file = create_response_file(temp_path, compile_args)
# Test clang with response file
cmd = [clang_path, f"@{response_file}"]
try:
result = subprocess.run(cmd, capture_output=True, text=True, timeout=30)
if result.returncode == 0 and output_file.exists():
print(" ✅ Clang response file compilation works")
print(f" Output file created: {output_file.stat().st_size} bytes")
print(f" Response file: @{response_file}")
results.passed += 1
else:
print(
f" ❌ Clang response file compilation failed (return code: {result.returncode})"
)
print(f" Stderr: {result.stderr}")
results.failed += 1
results.errors.append(
f"Clang response file compilation failed: {result.stderr}"
)
except subprocess.TimeoutExpired:
print(" ❌ Clang response file compilation timed out")
results.failed += 1
results.errors.append("Clang response file compilation timed out")
except Exception as e:
print(f" ❌ Clang response file compilation error: {e}")
results.failed += 1
results.errors.append(f"Clang response file compilation error: {e}")
def test_xcache_wrapper_script_creation(results: TestResults) -> None:
"""Test that xcache creates wrapper scripts correctly."""
print("\n🧪 Testing xcache wrapper script creation...")
clang_path = find_clang()
if not clang_path:
results.errors.append("clang compiler not found - skipping wrapper script test")
return
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create test source file and response file
source_file = create_test_source_file(temp_path)
compile_args = ["-c", str(source_file), "-o", "test.o"]
response_file = create_response_file(temp_path, compile_args)
# Create a mock sccache script that just echoes its arguments
mock_sccache = (
temp_path / "sccache.bat" if os.name == "nt" else temp_path / "sccache"
)
if os.name == "nt":
mock_content = """@echo off
echo MOCK_SCCACHE_CALLED: %*
"""
else:
mock_content = """#!/bin/bash
echo "MOCK_SCCACHE_CALLED: $@"
"""
mock_sccache.write_text(mock_content)
if os.name != "nt":
mock_sccache.chmod(0o755)
# Set up environment to use mock sccache
env = os.environ.copy()
env["PATH"] = f"{temp_path}{os.pathsep}{env.get('PATH', '')}"
env["XCACHE_DEBUG"] = "1"
# Test xcache with response file (should create wrapper script)
cmd = [sys.executable, "ci/util/xcache.py", clang_path, f"@{response_file}"]
try:
result = subprocess.run(
cmd, capture_output=True, text=True, env=env, timeout=30
)
# Check if wrapper script was created and executed
if "Created compiler wrapper:" in result.stderr:
print(" ✅ Wrapper script creation detected")
# Check if wrapper was executed
if "Executing with wrapper:" in result.stderr:
print(" ✅ Wrapper script execution detected")
# Check if mock sccache was called
if "MOCK_SCCACHE_CALLED:" in result.stdout:
print(" ✅ Mock sccache was called through wrapper")
print(f" Sccache called with: {result.stdout.strip()}")
results.passed += 1
else:
print(" ❌ Mock sccache was not called")
results.failed += 1
results.errors.append(
"Mock sccache was not called through wrapper"
)
else:
print(" ❌ Wrapper script was not executed")
results.failed += 1
results.errors.append("Wrapper script was not executed")
else:
print(" ❌ Wrapper script was not created")
print(f" Stderr: {result.stderr}")
results.failed += 1
results.errors.append(
f"Wrapper script creation failed: {result.stderr}"
)
except subprocess.TimeoutExpired:
print(" ❌ Wrapper script test timed out")
results.failed += 1
results.errors.append("Wrapper script test timed out")
except Exception as e:
print(f" ❌ Wrapper script test error: {e}")
results.failed += 1
results.errors.append(f"Wrapper script test error: {e}")
def test_xcache_direct_mode(results: TestResults) -> None:
"""Test xcache direct mode (no response files)."""
print("\n🧪 Testing xcache direct mode...")
clang_path = find_clang()
if not clang_path:
results.errors.append("clang compiler not found - skipping direct mode test")
return
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create test source file
source_file = create_test_source_file(temp_path)
# Create a mock sccache script that just passes through to real compiler
mock_sccache = (
temp_path / "sccache.bat" if os.name == "nt" else temp_path / "sccache"
)
if os.name == "nt":
mock_content = f'''@echo off
echo MOCK_SCCACHE_DIRECT: %*
"{clang_path}" %*
'''
else:
mock_content = f'''#!/bin/bash
echo "MOCK_SCCACHE_DIRECT: $@"
"{clang_path}" "$@"
'''
mock_sccache.write_text(mock_content)
if os.name != "nt":
mock_sccache.chmod(0o755)
# Set up environment to use mock sccache
env = os.environ.copy()
env["PATH"] = f"{temp_path}{os.pathsep}{env.get('PATH', '')}"
env["XCACHE_DEBUG"] = "1"
output_file = temp_path / "test_direct.o"
# Test xcache without response files (direct mode)
cmd = [
sys.executable,
"ci/util/xcache.py",
clang_path,
"-c",
str(source_file),
"-o",
str(output_file),
]
try:
result = subprocess.run(
cmd, capture_output=True, text=True, env=env, timeout=30
)
if result.returncode == 0 and output_file.exists():
print(" ✅ Direct mode compilation succeeded")
print(f" Output file created: {output_file.stat().st_size} bytes")
# Check if direct execution was used (no wrapper)
if "Direct execution:" in result.stderr:
print(" ✅ Direct execution mode confirmed")
results.passed += 1
else:
print(" ⚠️ Direct execution mode not clearly indicated")
results.passed += 1 # Still pass since compilation worked
else:
print(
f" ❌ Direct mode compilation failed (return code: {result.returncode})"
)
print(f" Stderr: {result.stderr}")
results.failed += 1
results.errors.append(
f"Direct mode compilation failed: {result.stderr}"
)
except subprocess.TimeoutExpired:
print(" ❌ Direct mode test timed out")
results.failed += 1
results.errors.append("Direct mode test timed out")
except Exception as e:
print(f" ❌ Direct mode test error: {e}")
results.failed += 1
results.errors.append(f"Direct mode test error: {e}")
def run_all_tests() -> TestResults:
"""Run all xcache standalone tests."""
print("🚀 XCACHE STANDALONE TESTS WITH CLANG")
print("=" * 60)
results = TestResults()
# Check prerequisites
clang_path = find_clang()
print(f"📋 Test Environment:")
print(f" Clang: {clang_path or 'NOT FOUND'}")
print(f" xcache: ci/util/xcache.py")
print(f" Note: Using mock sccache for testing wrapper functionality")
print()
if not clang_path:
print("❌ Clang compiler not found - install clang to run tests")
results.errors.append("Clang compiler not available")
return results
# Run tests
test_clang_direct_compilation(results)
test_clang_response_file_compilation(results)
test_xcache_wrapper_script_creation(results)
test_xcache_direct_mode(results)
return results
def main() -> int:
"""Main test function."""
results = run_all_tests()
print("\n" + "=" * 60)
print("📊 TEST RESULTS SUMMARY")
print("=" * 60)
total_tests = results.passed + results.failed
if total_tests > 0:
success_rate = (results.passed / total_tests) * 100
print(f" Tests passed: {results.passed}")
print(f" Tests failed: {results.failed}")
print(f" Success rate: {success_rate:.1f}%")
else:
print(" No tests were executed")
if results.errors:
print(f"\n🚨 Errors encountered:")
for i, error in enumerate(results.errors, 1):
print(f" {i}. {error}")
if results.failed == 0 and results.passed > 0:
print(
"\n✅ All xcache standalone tests passed! Wrapper functionality verified."
)
return 0
elif results.passed > 0:
print(
f"\n⚠️ Some tests passed but {results.failed} failed. Check errors above."
)
return 1
else:
print("\n❌ No tests passed. Check prerequisites and errors above.")
return 1
if __name__ == "__main__":
sys.exit(main())
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/test_unified_build.py
+227
View File
@@ -0,0 +1,227 @@
#!/usr/bin/env python3
"""
Test script to validate the new unified build system.
This script tests the core functionality of the BuildAPI to ensure
the refactored build system works correctly.
"""
import tempfile
from pathlib import Path
from ci.build_api import BuildType, create_example_builder, create_unit_test_builder
def test_build_api_creation():
"""Test that BuildAPI instances can be created successfully"""
print("Testing BuildAPI creation...")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Test unit test builder creation
unit_builder = create_unit_test_builder(
build_dir=temp_path / "unit", use_pch=True, parallel=True, clean=False
)
assert unit_builder.build_type == BuildType.UNIT_TEST
assert unit_builder.build_dir == temp_path / "unit"
assert unit_builder.use_pch == True
print("✓ Unit test builder created successfully")
# Test example builder creation
example_builder = create_example_builder(
build_dir=temp_path / "examples", use_pch=True, parallel=True, clean=False
)
assert example_builder.build_type == BuildType.EXAMPLE
assert example_builder.build_dir == temp_path / "examples"
assert example_builder.use_pch == True
print("✓ Example builder created successfully")
def test_build_configuration_loading():
"""Test that build configurations are loaded correctly"""
print("\nTesting build configuration loading...")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Test unit test configuration
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
# Check that unit test specific defines are present
unit_defines = unit_builder.compiler_settings.defines or []
assert "FASTLED_FORCE_NAMESPACE=1" in unit_defines
assert "FASTLED_TESTING=1" in unit_defines
assert "STUB_PLATFORM" in unit_defines
print("✓ Unit test configuration loaded correctly")
# Test example configuration
example_builder = create_example_builder(build_dir=temp_path / "examples")
# Check that example specific defines are present
example_defines = example_builder.compiler_settings.defines or []
assert "STUB_PLATFORM" in example_defines
assert "ARDUINO=10808" in example_defines
# Unit test specific defines should NOT be present
assert "FASTLED_FORCE_NAMESPACE=1" not in example_defines
assert "FASTLED_TESTING=1" not in example_defines
print("✓ Example configuration loaded correctly")
def test_build_directory_structure():
"""Test that build directory structure is created correctly"""
print("\nTesting build directory structure...")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
# Check that all required directories are created
assert unit_builder.build_dir.exists()
assert unit_builder.artifacts_dir.exists()
assert unit_builder.targets_dir.exists()
assert unit_builder.cache_dir.exists()
# Check directory organization
expected_structure = [
unit_builder.build_dir / "artifacts",
unit_builder.build_dir / "targets",
unit_builder.build_dir / "cache",
]
for directory in expected_structure:
assert directory.exists(), f"Directory {directory} should exist"
print("✓ Build directory structure created correctly")
def test_build_info():
"""Test that build info is returned correctly"""
print("\nTesting build info...")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
info = unit_builder.get_build_info()
# Check required fields
assert "config_file" in info
assert "build_dir" in info
assert "build_type" in info
assert "use_pch" in info
assert "parallel" in info
# Check values
assert info["build_type"] == "unit_test"
assert info["use_pch"] == True
assert str(temp_path / "unit") in info["build_dir"]
assert "build_unit.toml" in info["build_flags_toml"]
print("✓ Build info returned correctly")
def test_clean_functionality():
"""Test that clean functionality works"""
print("\nTesting clean functionality...")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
# Create some fake files to test cleaning
test_file = unit_builder.build_dir / "test_file.txt"
test_file.write_text("test content")
assert test_file.exists()
# Test clean functionality
unit_builder.clean_build_artifacts()
# Build directory should still exist but be empty (except for subdirs)
assert unit_builder.build_dir.exists()
assert not test_file.exists()
# Subdirectories should be recreated
assert unit_builder.artifacts_dir.exists()
assert unit_builder.targets_dir.exists()
assert unit_builder.cache_dir.exists()
print("✓ Clean functionality works correctly")
def test_configuration_separation():
"""Test that unit test and example configurations are properly separated"""
print("\nTesting configuration separation...")
with tempfile.TemporaryDirectory() as temp_dir:
temp_path = Path(temp_dir)
# Create both builders
unit_builder = create_unit_test_builder(build_dir=temp_path / "unit")
example_builder = create_example_builder(build_dir=temp_path / "examples")
# Get build info for comparison
unit_info = unit_builder.get_build_info()
example_info = example_builder.get_build_info()
# Configurations should use different files
assert "build_unit.toml" in unit_info["build_flags_toml"]
assert "build_example.toml" in example_info["build_flags_toml"]
# Build types should be different
assert unit_info["build_type"] == "unit_test"
assert example_info["build_type"] == "example"
# Build directories should be different
assert unit_info["build_dir"] != example_info["build_dir"]
print("✓ Configuration separation works correctly")
def run_all_tests():
"""Run all validation tests"""
print("=" * 60)
print("FASTLED UNIFIED BUILD SYSTEM VALIDATION")
print("=" * 60)
try:
test_build_api_creation()
test_build_configuration_loading()
test_build_directory_structure()
test_build_info()
test_clean_functionality()
test_configuration_separation()
print("\n" + "=" * 60)
print("ALL TESTS PASSED ✓")
print("=" * 60)
print("The unified build system is working correctly!")
print("\nKey features validated:")
print("✓ BuildAPI creation for both unit tests and examples")
print("✓ Proper configuration loading from separate TOML files")
print("✓ Organized build directory structure")
print("✓ Clean separation between unit test and example builds")
print("✓ Build info reporting")
print("✓ Clean functionality")
return True
except Exception as e:
print(f"\n❌ TEST FAILED: {e}")
import traceback
traceback.print_exc()
return False
if __name__ == "__main__":
success = run_all_tests()
exit(0 if success else 1)
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/tests/check_namespace_includes.py
+114
View File
@@ -0,0 +1,114 @@
# pyright: reportUnknownMemberType=false, reportMissingParameterType=false
"""Test for checking namespace includes in C++ headers."""
#!/usr/bin/env python3
"""
Script to check for includes after namespace declarations in C++ files.
This is used in CI/CD to prevent bad code patterns.
"""
import os
import re
import sys
from pathlib import Path
from typing import Any, Dict, List
def find_includes_after_namespace(file_path: Path) -> List[int]:
"""
Check if a C++ file has #include directives after namespace declarations.
Args:
file_path (Path): Path to the C++ file to check
Returns:
List[int]: List of line numbers where includes appear after namespaces
"""
try:
with open(file_path, "r", encoding="utf-8") as f:
content = f.readlines()
violations: List[int] = []
namespace_started = False
# Basic patterns
namespace_pattern = re.compile(r"^\s*namespace\s+\w+\s*\{")
include_pattern = re.compile(r"^\s*#\s*include")
for i, line in enumerate(content, 1):
line = line.strip()
# Skip empty lines and comments
if not line or line.startswith("//") or line.startswith("/*"):
continue
# Check for namespace declaration
if namespace_pattern.match(line):
namespace_started = True
# Check for #include after namespace started
if namespace_started and include_pattern.match(line):
violations.append(i)
return violations
except (UnicodeDecodeError, IOError):
# Skip files that can't be read
return []
def scan_cpp_files(directory: str = ".") -> Dict[str, Any]:
"""
Scan all C++ files in a directory for includes after namespace declarations.
Args:
directory (str): Directory to scan for C++ files
Returns:
Dict[str, Any]: Dictionary mapping file paths to lists of violation line numbers
"""
cpp_extensions = [
".cpp",
".cc",
".cxx",
".c++",
".hpp",
".h",
".hh",
".hxx",
".h++",
]
violations: Dict[str, Any] = {}
for root, dirs, files in os.walk(directory):
for file in files:
file_path = os.path.join(root, file)
# Check if it's a C++ file
if any(file.endswith(ext) for ext in cpp_extensions):
line_numbers: List[int] = find_includes_after_namespace(Path(file_path))
if line_numbers:
violations[file_path] = line_numbers
return violations
def main() -> None:
violations: Dict[str, Any] = scan_cpp_files()
if violations:
print("Found #include directives after namespace declarations:")
for file_path, line_numbers in violations.items():
print(f"\n{file_path}:")
for line_num in line_numbers:
print(f" Line {line_num}")
failing: List[str] = list(violations.keys())
print(f"\nFailing files: {failing}")
sys.exit(1)
else:
print("No violations found.")
sys.exit(0)
if __name__ == "__main__":
main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/tests/no_using_namespace_fl_in_headers.py
+63
View File
@@ -0,0 +1,63 @@
#!/usr/bin/env python3
# pyright: reportUnknownMemberType=false, reportMissingParameterType=false
import os
import unittest
from concurrent.futures import ThreadPoolExecutor
from typing import List
from ci.util.paths import PROJECT_ROOT
SRC_ROOT = PROJECT_ROOT / "src"
PLATFORMS_DIR = os.path.join(SRC_ROOT, "platforms")
NUM_WORKERS = 1 if os.environ.get("NO_PARALLEL") else (os.cpu_count() or 1) * 4
class NoUsingNamespaceFlInHeaderTester(unittest.TestCase):
def check_file(self, file_path: str) -> List[str]:
if "FastLED.h" in file_path:
return []
failings: List[str] = []
with open(file_path, "r", encoding="utf-8") as f:
for line_number, line in enumerate(f, 1):
if line.startswith("//"):
continue
if "using namespace fl;" in line:
failings.append(f"{file_path}:{line_number}: {line.strip()}")
return failings
def test_no_using_namespace(self) -> None:
"""Searches through the program files to check for banned headers, excluding src/platforms."""
files_to_check: List[str] = []
for root, _, files in os.walk(SRC_ROOT):
for file in files:
if file.endswith(
(".h", ".hpp")
): # Add or remove file extensions as needed
file_path = os.path.join(root, file)
files_to_check.append(file_path)
all_failings: List[str] = []
with ThreadPoolExecutor(max_workers=NUM_WORKERS) as executor:
futures = [
executor.submit(self.check_file, file_path)
for file_path in files_to_check
]
for future in futures:
all_failings.extend(future.result())
if all_failings:
msg = (
f'Found {len(all_failings)} header file(s) "using namespace fl": \n'
+ "\n".join(all_failings)
)
for failing in all_failings:
print(failing)
self.fail(msg)
else:
print("No using namespace fl; found in headers.")
if __name__ == "__main__":
unittest.main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/tests/test_bin_2_elf.py
+74
View File
@@ -0,0 +1,74 @@
import subprocess
import unittest
import warnings
from pathlib import Path
from ci.util.bin_2_elf import bin_to_elf
from ci.util.elf import dump_symbol_sizes
from ci.util.paths import PROJECT_ROOT
from ci.util.tools import Tools, load_tools
HERE = Path(__file__).resolve().parent.absolute()
UNO = HERE / "uno"
OUTPUT = HERE / "output"
BUILD_INFO_PATH = PROJECT_ROOT / ".build" / "examples" / "uno" / "build_info.json"
BUILD_INFO_PATH2 = (
PROJECT_ROOT / ".build" / "fled" / "examples" / "uno" / "build_info.json"
)
DISABLED = True
class TestBinToElf(unittest.TestCase):
@classmethod
def setUpClass(cls):
if DISABLED:
return
uno_build = PROJECT_ROOT / ".build" / "uno"
print(f"Checking for Uno build in: {uno_build}")
if not uno_build.exists():
print("Uno build not found. Running compilation...")
try:
subprocess.run(
"uv run python -m ci.ci-compile uno --examples Blink",
shell=True,
check=True,
)
print("Compilation completed successfully.")
except subprocess.CalledProcessError as e:
print(f"Error during compilation: {e}")
raise
@unittest.skip("Skip bin to elf conversion test")
def test_bin_to_elf_conversion(self) -> None:
if DISABLED:
return
tools: Tools
try:
tools = load_tools(BUILD_INFO_PATH)
except FileNotFoundError as e:
warnings.warn(f"Error while loading tools: {e}")
tools = load_tools(BUILD_INFO_PATH2)
bin_file = UNO / "firmware.hex"
map_file = UNO / "firmware.map"
output_elf = OUTPUT / "output.elf"
try:
bin_to_elf(
bin_file,
map_file,
tools.as_path,
tools.ld_path,
tools.objcopy_path,
output_elf,
)
stdout = dump_symbol_sizes(tools.nm_path, tools.cpp_filt_path, output_elf)
print(stdout)
except Exception as e:
warnings.warn(f"Error while converting binary to ELF: {e}")
if __name__ == "__main__":
unittest.main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/tests/test_board_to_platformio_ini.py
+53
View File
@@ -0,0 +1,53 @@
import unittest
from ci.boards import Board
class TestBoardToPlatformioIni(unittest.TestCase):
"""Tests for Board.to_platformio_ini()."""
def _ini_to_set(self, ini: str) -> set[str]:
"""Return a set with each non-empty, stripped line of the ini snippet."""
return {line.strip() for line in ini.splitlines() if line.strip()}
def test_basic_fields(self):
board = Board(board_name="uno", platform="atmelavr", framework="arduino")
ini = board.to_platformio_ini()
lines = self._ini_to_set(ini)
expected = {
"[env:uno]",
"board = uno",
"platform = atmelavr",
"framework = arduino",
}
self.assertTrue(expected.issubset(lines))
# Should not reference internal attributes
self.assertNotIn("platform_needs_install", ini)
def test_real_board_name(self):
board = Board(
board_name="esp32c3",
real_board_name="esp32-c3-devkitm-1",
platform="espressif32",
)
ini = board.to_platformio_ini()
lines = self._ini_to_set(ini)
self.assertIn("[env:esp32c3]", lines)
self.assertIn("board = esp32-c3-devkitm-1", lines)
def test_flags_and_unflags(self):
board = Board(
board_name="custom",
defines=["FASTLED_TEST=1"],
build_flags=["-O2"],
)
ini = board.to_platformio_ini()
lines = self._ini_to_set(ini)
# The build_flags are in multi-line format - check that both flags are present as separate lines
self.assertIn("build_flags =", lines)
self.assertIn("-DFASTLED_TEST=1", lines)
self.assertIn("-O2", lines)
if __name__ == "__main__":
unittest.main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/tests/test_build_flags_toml.py
+357
View File
@@ -0,0 +1,357 @@
#!/usr/bin/env python3
"""
Unit tests for BuildFlags TOML parsing and serialization functionality.
Tests the BuildFlags and BuildTools classes for parsing TOML build configuration
files and serializing back to TOML format.
"""
import tempfile
import unittest
from pathlib import Path
from ci.compiler.clang_compiler import ArchiveOptions, BuildFlags, BuildTools
class TestBuildFlagsToml(unittest.TestCase):
"""Test BuildFlags TOML parsing and serialization"""
def setUp(self) -> None:
"""Set up test fixtures"""
self.temp_dir = Path(tempfile.mkdtemp())
def tearDown(self) -> None:
"""Clean up test fixtures"""
# Clean up any test files
for file in self.temp_dir.glob("*.toml"):
file.unlink()
self.temp_dir.rmdir()
def create_test_toml(self, content: str) -> Path:
"""Create a test TOML file with the given content"""
test_file = self.temp_dir / "test_build_flags.toml"
with open(test_file, "w", encoding="utf-8") as f:
f.write(content)
return test_file
def test_build_tools_defaults(self) -> None:
"""Test BuildTools default values"""
tools = BuildTools(
cpp_compiler=[],
archiver=[],
linker=[],
c_compiler=[],
objcopy=[],
nm=[],
strip=[],
ranlib=[],
)
# Test modern command-based fields (should be empty by default)
self.assertEqual(tools.cpp_compiler, [])
self.assertEqual(tools.archiver, [])
self.assertEqual(tools.linker, [])
# Test other important fields
self.assertEqual(tools.c_compiler, [])
self.assertEqual(tools.objcopy, [])
self.assertEqual(tools.nm, [])
self.assertEqual(tools.strip, [])
self.assertEqual(tools.ranlib, [])
def test_parse_minimal_toml(self) -> None:
"""Test parsing minimal TOML file with required tools section"""
toml_content = """
[all]
defines = ["-DTEST=1"]
compiler_flags = ["-Wall"]
include_flags = ["-I."]
[tools]
cpp_compiler = ["uv", "run", "python", "-m", "ziglang", "c++"]
linker = ["uv", "run", "python", "-m", "ziglang", "c++"]
c_compiler = ["clang"]
objcopy = ["uv", "run", "python", "-m", "ziglang", "objcopy"]
nm = ["uv", "run", "python", "-m", "ziglang", "nm"]
strip = ["uv", "run", "python", "-m", "ziglang", "strip"]
ranlib = ["uv", "run", "python", "-m", "ziglang", "ranlib"]
archiver = ["uv", "run", "python", "-m", "ziglang", "ar"]
[archive]
flags = "rcsD"
[linking.base]
flags = ["-pthread"]
[strict_mode]
flags = ["-Werror"]
"""
test_file = self.create_test_toml(toml_content)
flags = BuildFlags.parse(test_file, quick_build=False, strict_mode=False)
# Check basic flags
self.assertEqual(flags.defines, ["-DTEST=1"])
self.assertEqual(flags.compiler_flags, ["-Wall"])
self.assertEqual(flags.include_flags, ["-I."])
self.assertEqual(flags.link_flags, ["-pthread"])
self.assertEqual(flags.strict_mode_flags, ["-Werror"])
# Check tools (from [tools] section) - modern command-based approach
self.assertEqual(flags.tools.c_compiler, ["clang"])
self.assertEqual(
flags.tools.linker, ["uv", "run", "python", "-m", "ziglang", "c++"]
)
self.assertEqual(
flags.tools.cpp_compiler,
["uv", "run", "python", "-m", "ziglang", "c++"],
)
self.assertEqual(
flags.tools.objcopy, ["uv", "run", "python", "-m", "ziglang", "objcopy"]
)
self.assertEqual(flags.tools.nm, ["uv", "run", "python", "-m", "ziglang", "nm"])
self.assertEqual(
flags.tools.strip, ["uv", "run", "python", "-m", "ziglang", "strip"]
)
self.assertEqual(
flags.tools.ranlib, ["uv", "run", "python", "-m", "ziglang", "ranlib"]
)
self.assertEqual(
flags.tools.archiver, ["uv", "run", "python", "-m", "ziglang", "ar"]
)
def test_parse_toml_with_tools(self) -> None:
"""Test parsing TOML file with [tools] section"""
toml_content = """
[all]
defines = ["-DTEST=1"]
compiler_flags = ["-Wall"]
include_flags = ["-I."]
[tools]
cpp_compiler = ["g++"]
archiver = ["gcc-ar"]
linker = ["ld.gold"]
c_compiler = ["gcc"]
objcopy = ["arm-objcopy"]
nm = ["arm-nm"]
strip = ["arm-strip"]
ranlib = ["arm-ranlib"]
[archive]
flags = "rcsD"
[linking.base]
flags = ["-pthread"]
"""
test_file = self.create_test_toml(toml_content)
flags = BuildFlags.parse(test_file, quick_build=False, strict_mode=False)
# Check that tools were parsed correctly - modern command-based approach
self.assertEqual(flags.tools.cpp_compiler, ["g++"])
self.assertEqual(flags.tools.archiver, ["gcc-ar"])
self.assertEqual(flags.tools.linker, ["ld.gold"])
self.assertEqual(flags.tools.c_compiler, ["gcc"])
self.assertEqual(flags.tools.objcopy, ["arm-objcopy"])
self.assertEqual(flags.tools.nm, ["arm-nm"])
self.assertEqual(flags.tools.strip, ["arm-strip"])
self.assertEqual(flags.tools.ranlib, ["arm-ranlib"])
def test_parse_partial_tools_section(self) -> None:
"""Test parsing TOML with all required [tools] section fields"""
toml_content = """
[all]
defines = ["-DTEST=1"]
[tools]
cpp_compiler = ["custom-clang++"]
archiver = ["custom-ar"]
linker = ["custom-linker"]
c_compiler = ["clang"]
objcopy = ["custom-objcopy"]
nm = ["custom-nm"]
strip = ["custom-strip"]
ranlib = ["custom-ranlib"]
# All tools must be provided - no defaults allowed
[archive]
flags = "rcsD"
"""
test_file = self.create_test_toml(toml_content)
flags = BuildFlags.parse(test_file, quick_build=False, strict_mode=False)
# Check all tools are set as specified - strict validation
self.assertEqual(flags.tools.cpp_compiler, ["custom-clang++"])
self.assertEqual(flags.tools.archiver, ["custom-ar"])
self.assertEqual(flags.tools.linker, ["custom-linker"])
self.assertEqual(flags.tools.c_compiler, ["clang"])
self.assertEqual(flags.tools.objcopy, ["custom-objcopy"])
self.assertEqual(flags.tools.nm, ["custom-nm"])
self.assertEqual(flags.tools.strip, ["custom-strip"])
self.assertEqual(flags.tools.ranlib, ["custom-ranlib"])
def test_serialize_build_flags_with_tools(self) -> None:
"""Test serializing BuildFlags with tools to TOML"""
# Create BuildFlags with custom tools
custom_tools = BuildTools(
linker=["arm-none-eabi-ld"],
c_compiler=["arm-none-eabi-gcc"],
objcopy=["arm-none-eabi-objcopy"],
nm=["arm-none-eabi-nm"],
strip=["arm-none-eabi-strip"],
ranlib=["arm-none-eabi-ranlib"],
cpp_compiler=["arm-none-eabi-g++"],
archiver=["arm-none-eabi-ar"],
)
flags = BuildFlags(
defines=["-DARM_BUILD=1"],
compiler_flags=["-mcpu=cortex-m4", "-mthumb"],
include_flags=["-I.", "-Iarm"],
link_flags=["-nostdlib"],
strict_mode_flags=["-Werror"],
tools=custom_tools,
archive=ArchiveOptions(flags="rcsD"),
)
# Serialize to TOML
toml_output = flags.serialize()
# Check that tools section is present with new field names
self.assertIn("[tools]", toml_output)
self.assertIn("cpp_compiler = ['arm-none-eabi-g++']", toml_output)
self.assertIn("archiver = ['arm-none-eabi-ar']", toml_output)
self.assertIn("linker = ['arm-none-eabi-ld']", toml_output)
self.assertIn("c_compiler = ['arm-none-eabi-gcc']", toml_output)
self.assertIn("objcopy = ['arm-none-eabi-objcopy']", toml_output)
self.assertIn("nm = ['arm-none-eabi-nm']", toml_output)
self.assertIn("strip = ['arm-none-eabi-strip']", toml_output)
self.assertIn("ranlib = ['arm-none-eabi-ranlib']", toml_output)
def test_serialize_with_none_linker(self) -> None:
"""Test serializing BuildFlags when linker is None"""
flags = BuildFlags(
defines=["-DTEST=1"],
compiler_flags=[],
include_flags=[],
link_flags=[],
strict_mode_flags=[],
tools=BuildTools(
linker=[], # Empty list instead of None
cpp_compiler=["clang++"],
c_compiler=["clang"],
archiver=[],
objcopy=[],
nm=[],
strip=[],
ranlib=[],
),
archive=ArchiveOptions(flags="rcsD"),
)
toml_output = flags.serialize()
# Check that tools section is present but linker is omitted
self.assertIn("[tools]", toml_output)
self.assertIn("cpp_compiler = ['clang++']", toml_output)
self.assertNotIn("linker =", toml_output) # Should be omitted when empty
def test_round_trip_toml_parsing(self) -> None:
"""Test that parse -> serialize -> parse maintains data integrity"""
# Create original flags
original_tools = BuildTools(
linker=["test-ld"],
c_compiler=["test-gcc"],
cpp_compiler=["test-compiler"],
archiver=["test-ar"],
objcopy=["test-objcopy"],
nm=["test-nm"],
strip=["test-strip"],
ranlib=["test-ranlib"],
)
original_flags = BuildFlags(
defines=["-DROUND_TRIP=1"],
compiler_flags=["-Wall", "-O2"],
include_flags=["-I.", "-Itest"],
link_flags=["-pthread"],
strict_mode_flags=["-Werror"],
tools=original_tools,
archive=ArchiveOptions(flags="rcsD"),
)
# Serialize to TOML
toml_content = original_flags.serialize()
# Write to temporary file
temp_file = self.temp_dir / "roundtrip.toml"
original_flags.to_toml_file(temp_file)
# Parse back from file
parsed_flags = BuildFlags.parse(temp_file, quick_build=False, strict_mode=False)
# Check that all data is preserved
self.assertEqual(parsed_flags.defines, original_flags.defines)
self.assertEqual(parsed_flags.compiler_flags, original_flags.compiler_flags)
self.assertEqual(parsed_flags.include_flags, original_flags.include_flags)
self.assertEqual(parsed_flags.link_flags, original_flags.link_flags)
self.assertEqual(
parsed_flags.strict_mode_flags, original_flags.strict_mode_flags
)
# Check tools - modern command-based approach
self.assertEqual(
parsed_flags.tools.cpp_compiler, original_flags.tools.cpp_compiler
)
self.assertEqual(parsed_flags.tools.archiver, original_flags.tools.archiver)
self.assertEqual(parsed_flags.tools.linker, original_flags.tools.linker)
self.assertEqual(parsed_flags.tools.c_compiler, original_flags.tools.c_compiler)
def test_parse_missing_file(self) -> None:
"""Test parsing non-existent TOML file raises FileNotFoundError"""
nonexistent_file = self.temp_dir / "does_not_exist.toml"
# Should raise FileNotFoundError when file is missing
with self.assertRaises(FileNotFoundError) as context:
BuildFlags.parse(nonexistent_file, quick_build=False, strict_mode=False)
self.assertIn("Required build_flags.toml not found", str(context.exception))
def test_from_toml_file_alias(self) -> None:
"""Test that from_toml_file() is an alias for parse()"""
toml_content = """
[all]
defines = ["-DALIAS_TEST=1"]
[tools]
cpp_compiler = ["alias-compiler"]
linker = ["alias-linker"]
c_compiler = ["clang"]
objcopy = ["alias-objcopy"]
nm = ["alias-nm"]
strip = ["alias-strip"]
ranlib = ["alias-ranlib"]
archiver = ["alias-archiver"]
[archive]
flags = "rcsD"
"""
test_file = self.create_test_toml(toml_content)
# Parse using both methods
flags_parse = BuildFlags.parse(test_file, quick_build=True, strict_mode=False)
flags_alias = BuildFlags.from_toml_file(
test_file, quick_build=True, strict_mode=False
)
# Should be identical
self.assertEqual(flags_parse.defines, flags_alias.defines)
self.assertEqual(flags_parse.tools.cpp_compiler, flags_alias.tools.cpp_compiler)
self.assertEqual(flags_parse.tools.cpp_compiler, ["alias-compiler"])
if __name__ == "__main__":
unittest.main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/tests/test_buildflags.py
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env python3
"""
Tests for BuildFlags TOML parsing in ci/ directory.
This test discovers all top-level *.toml files under the project ci/ folder
and verifies they can be parsed by BuildFlags.parse without throwing.
"""
import unittest
from pathlib import Path
from typing import List
from ci.compiler.clang_compiler import BuildFlags
def find_ci_toml_files(ci_dir: Path) -> List[Path]:
"""Return a list of top-level *.toml files in ci/.
Args:
ci_dir: Absolute path to the ci/ directory
Returns:
List of absolute paths to .toml files directly under ci/.
"""
toml_files: List[Path] = []
for entry in ci_dir.iterdir():
if entry.is_file() and entry.suffix == ".toml":
toml_files.append(entry.resolve())
return toml_files
class TestBuildFlagsParsing(unittest.TestCase):
def test_parse_all_ci_tomls(self) -> None:
project_root: Path = Path(__file__).resolve().parent.parent.parent
ci_dir: Path = project_root / "ci"
self.assertTrue(ci_dir.exists(), f"ci directory not found at {ci_dir}")
toml_files: List[Path] = find_ci_toml_files(ci_dir)
self.assertTrue(len(toml_files) > 0, "No .toml files found in ci/")
for toml_path in toml_files:
with self.subTest(toml=str(toml_path)):
# Ensure each ci/*.toml parses without throwing
_ = BuildFlags.parse(toml_path, quick_build=True, strict_mode=False)
if __name__ == "__main__":
unittest.main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/tests/test_elf.py
+65
View File
@@ -0,0 +1,65 @@
import subprocess
import unittest
import warnings
from pathlib import Path
from ci.util.elf import dump_symbol_sizes
from ci.util.paths import PROJECT_ROOT
from ci.util.tools import Tools, load_tools
HERE = Path(__file__).resolve().parent.absolute()
UNO = HERE / "uno"
OUTPUT = HERE / "output"
ELF_FILE = UNO / "firmware.elf"
BUILD_INFO_PATH = PROJECT_ROOT / ".build" / "uno" / "build_info.json"
BUILD_INFO_PATH2 = (
PROJECT_ROOT / ".build" / "fled" / "examples" / "uno" / "build_info.json"
)
PLATFORMIO_PATH = Path.home() / ".platformio"
PLATFORMIO_PACKAGES_PATH = PLATFORMIO_PATH / "packages"
TOOLCHAIN_AVR = PLATFORMIO_PACKAGES_PATH / "toolchain-atmelavr"
def init() -> None:
uno_build = PROJECT_ROOT / ".build" / "uno"
print(f"Checking for Uno build in: {uno_build}")
if not BUILD_INFO_PATH.exists() or not TOOLCHAIN_AVR.exists():
print("Uno build not found. Running compilation...")
try:
subprocess.run(
"uv run python -m ci.ci-compile uno --examples Blink",
shell=True,
check=True,
cwd=str(PROJECT_ROOT),
)
print("Compilation completed successfully.")
except subprocess.CalledProcessError as e:
print(f"Error during compilation: {e}")
raise
class TestBinToElf(unittest.TestCase):
def test_bin_to_elf_conversion(self) -> None:
# Skip test if required UNO build directory is missing
uno_build_dir = PROJECT_ROOT / ".build" / "uno"
if not uno_build_dir.exists():
warnings.warn(
"Skipping TestBinToElf::test_bin_to_elf_conversion because .build/uno does not exist. "
"Run 'uv run ci/ci-compile.py uno --examples Blink' to generate it."
)
self.skipTest(".build/uno missing; skipping ELF conversion test")
tools: Tools
try:
tools = load_tools(BUILD_INFO_PATH)
except Exception as e:
warnings.warn(f"Error while loading tools: {e}")
tools = load_tools(BUILD_INFO_PATH2)
msg = dump_symbol_sizes(tools.nm_path, tools.cpp_filt_path, ELF_FILE)
print(msg)
if __name__ == "__main__":
unittest.main()
v1.2/Shibo-main-TOMMY/lib/FastLED/ci/tests/test_fingerprint_cache.py
+450
View File
@@ -0,0 +1,450 @@
"""
Comprehensive test suite for fingerprint cache functionality.
Tests cover:
- Core functionality (cache hits, misses, content changes)
- Edge cases (touched files, copied files, corruption)
- Performance requirements
- Integration scenarios
"""
import json
import os
import shutil
import tempfile
import time
from pathlib import Path
from typing import List, Optional
from unittest import TestCase
# Import the fingerprint cache module
from ci.ci.fingerprint_cache import CacheEntry, FingerprintCache, has_changed
class TestFingerprintCache(TestCase):
"""Test suite for fingerprint cache functionality."""
def setUp(self) -> None:
"""Set up test environment with temporary directory."""
self.test_dir = Path(tempfile.mkdtemp())
self.cache_dir = self.test_dir / "cache"
self.cache_dir.mkdir(exist_ok=True)
self.temp_dir = self.test_dir / "temp"
self.temp_dir.mkdir(exist_ok=True)
def tearDown(self) -> None:
"""Clean up test environment."""
shutil.rmtree(self.test_dir, ignore_errors=True)
def create_test_file(
self, path: Path, content: str, modtime: Optional[float] = None
) -> Path:
"""Create a test file with specific content and optional modification time."""
with open(path, "w") as f:
f.write(content)
if modtime:
# Set specific modification time
os.utime(path, (modtime, modtime))
return path
def touch_file(self, path: Path) -> float:
"""Touch a file to update its modification time without changing content."""
# Wait a small amount to ensure modtime changes
time.sleep(0.01)
path.touch()
return os.path.getmtime(path)
def modify_file_content(self, path: Path, new_content: str) -> float:
"""Modify file content and return new modification time."""
time.sleep(0.01) # Ensure modtime changes
with open(path, "w") as f:
f.write(new_content)
return os.path.getmtime(path)
def test_cache_hit_modtime_unchanged(self) -> None:
"""Test that identical modtime returns False immediately."""
cache_file = self.cache_dir / "test_cache.json"
cache = FingerprintCache(cache_file)
# Create test file
test_file = self.temp_dir / "test.txt"
self.create_test_file(test_file, "original content")
original_modtime = os.path.getmtime(test_file)
# Same modtime should return False (no change detected)
result1 = cache.has_changed(test_file, original_modtime)
self.assertFalse(result1, "Same modtime should return False")
# Verify fast performance (cache hit)
start_time = time.time()
result2 = cache.has_changed(test_file, original_modtime)
elapsed = time.time() - start_time
self.assertFalse(result2, "Same modtime should return False")
self.assertLess(
elapsed, 0.002, f"Cache hit should be <2ms, took {elapsed * 1000:.2f}ms"
)
def test_cache_hit_with_existing_cache(self) -> None:
"""Test cache hit when file exists in cache with current modtime."""
cache_file = self.cache_dir / "test_cache.json"
cache = FingerprintCache(cache_file)
# Create and cache a file
test_file = self.temp_dir / "cached.txt"
self.create_test_file(test_file, "cached content")
current_modtime = os.path.getmtime(test_file)
# Prime the cache
cache.has_changed(
test_file, current_modtime - 1
) # Different modtime to force caching
# Test cache hit
result = cache.has_changed(test_file, current_modtime)
self.assertFalse(
result, "File with current modtime in cache should return False"
)
def test_modtime_changed_content_same(self) -> None:
"""Critical test: modtime changes but content identical should return False."""
cache_file = self.cache_dir / "test_cache.json"
cache = FingerprintCache(cache_file)
# Create test file
test_file = self.temp_dir / "touched.txt"
original_content = "unchanged content"
self.create_test_file(test_file, original_content)
original_modtime = os.path.getmtime(test_file)
# Prime cache with original state
cache.has_changed(test_file, original_modtime - 1) # Force caching
# Touch file (change modtime but not content)
new_modtime = self.touch_file(test_file)
self.assertNotEqual(
new_modtime, original_modtime, "Touch should change modtime"
)
# Verify content is still the same
with open(test_file, "r") as f:
current_content = f.read()
self.assertEqual(
current_content, original_content, "Content should be unchanged"
)
# Test the critical behavior: should return False despite modtime change
result = cache.has_changed(test_file, original_modtime)
self.assertFalse(
result, "File with changed modtime but same content should return False"
)
def test_file_copy_different_timestamp(self) -> None:
"""Test same file with different timestamps but identical content."""
cache_file = self.cache_dir / "test_cache.json"
cache = FingerprintCache(cache_file)
# Create original file
test_file = self.temp_dir / "timestamp_test.txt"
content = "file content for timestamp testing"
self.create_test_file(test_file, content)
original_modtime = os.path.getmtime(test_file)
# Prime cache with original state
cache.has_changed(test_file, original_modtime - 1)
# Touch file to change timestamp but keep content same
time.sleep(0.01)
new_modtime = time.time()
os.utime(test_file, (new_modtime, new_modtime))
# Verify content is still the same
with open(test_file, "r") as f:
current_content = f.read()
self.assertEqual(current_content, content, "Content should be unchanged")
# Test: different modtime, same content should return False
result = cache.has_changed(test_file, original_modtime)
self.assertFalse(
result, "File with different timestamp but same content should return False"
)
def test_content_actually_changed(self) -> None:
"""Test that actual content changes are detected."""
cache_file = self.cache_dir / "test_cache.json"
cache = FingerprintCache(cache_file)
# Create test file
test_file = self.temp_dir / "modified.txt"
original_content = "original content"
self.create_test_file(test_file, original_content)
original_modtime = os.path.getmtime(test_file)
# Prime cache
cache.has_changed(test_file, original_modtime - 1)
# Actually modify content
new_content = "modified content"
new_modtime = self.modify_file_content(test_file, new_content)
# Should detect change
result = cache.has_changed(test_file, original_modtime)
self.assertTrue(result, "File with changed content should return True")
def test_incremental_changes(self) -> None:
"""Test detecting changes at different points in time."""
cache_file = self.cache_dir / "test_cache.json"
cache = FingerprintCache(cache_file)
test_file = self.temp_dir / "incremental.txt"
# Version 1 - baseline
content_v1 = "version 1"
self.create_test_file(test_file, content_v1)
modtime_v1 = os.path.getmtime(test_file)
# Test v1 against earlier time (should be changed)
result = cache.has_changed(test_file, modtime_v1 - 1)
self.assertTrue(result, "File should be detected as changed from earlier time")
# Test v1 against same time (should be unchanged)
result = cache.has_changed(test_file, modtime_v1)
self.assertFalse(result, "File should be unchanged from same time")
# Version 2 - modify content
content_v2 = "version 2"
modtime_v2 = self.modify_file_content(test_file, content_v2)
# Test v2 against v1 time (should be changed)
result = cache.has_changed(test_file, modtime_v1)
self.assertTrue(result, "File should be detected as changed from v1 time")
# Test v2 against v2 time (should be unchanged)
result = cache.has_changed(test_file, modtime_v2)
self.assertFalse(result, "File should be unchanged from v2 time")
def test_cache_persistence(self) -> None:
"""Test that cache persists across FingerprintCache instances."""
cache_file = self.cache_dir / "persistent.json"
# First cache instance
cache1 = FingerprintCache(cache_file)
test_file = self.temp_dir / "persistent.txt"
self.create_test_file(test_file, "persistent content")
modtime = os.path.getmtime(test_file)
# Prime cache
cache1.has_changed(test_file, modtime - 1)
# Second cache instance (reload from disk)
cache2 = FingerprintCache(cache_file)
# Should use cached data
result = cache2.has_changed(test_file, modtime)
self.assertFalse(result, "New cache instance should load existing cache")
def test_cache_corruption_recovery(self) -> None:
"""Test graceful handling of corrupted cache files."""
cache_file = self.cache_dir / "corrupted.json"
# Create corrupted cache file
with open(cache_file, "w") as f:
f.write("{ invalid json content")
# Should handle corruption gracefully
cache = FingerprintCache(cache_file)
test_file = self.temp_dir / "recovery.txt"
self.create_test_file(test_file, "recovery content")
modtime = os.path.getmtime(test_file)
# Should work despite corrupted cache
result = cache.has_changed(test_file, modtime - 1)
self.assertTrue(result, "Should work with corrupted cache")
def test_file_not_found_error(self) -> None:
"""Test FileNotFoundError for non-existent files."""
cache_file = self.cache_dir / "test_cache.json"
cache = FingerprintCache(cache_file)
non_existent_file = self.temp_dir / "does_not_exist.txt"
with self.assertRaises(FileNotFoundError):
cache.has_changed(non_existent_file, time.time())
def test_performance_cache_hit(self) -> None:
"""Benchmark cache hit performance."""
cache_file = self.cache_dir / "perf.json"
cache = FingerprintCache(cache_file)
# Create test files
test_files: List[Path] = []
for i in range(10): # Reduced for faster test execution
test_file = self.temp_dir / f"perf_{i}.txt"
self.create_test_file(test_file, f"content {i}")
test_files.append(test_file)
# Measure cache hit performance
start_time = time.time()
for test_file in test_files:
modtime = os.path.getmtime(test_file)
cache.has_changed(test_file, modtime) # Cache hit
elapsed = time.time() - start_time
avg_time = elapsed / len(test_files) * 1000 # ms per file
self.assertLess(
avg_time, 1.0, f"Cache hit average {avg_time:.2f}ms should be <1.0ms"
)
def test_performance_large_files(self) -> None:
"""Test performance with moderately large files."""
cache_file = self.cache_dir / "large.json"
cache = FingerprintCache(cache_file)
# Create moderately large test file (100KB)
large_file = self.temp_dir / "large.txt"
large_content = "x" * (100 * 1024)
self.create_test_file(large_file, large_content)
modtime = os.path.getmtime(large_file)
# Measure MD5 computation time
start_time = time.time()
result = cache.has_changed(large_file, modtime - 1) # Force MD5 computation
elapsed = time.time() - start_time
self.assertTrue(result, "Large file should be detected as changed")
self.assertLess(
elapsed,
0.5,
f"Large file processing {elapsed * 1000:.2f}ms should be <500ms",
)
def test_convenience_function(self) -> None:
"""Test the convenience has_changed function."""
cache_file = self.cache_dir / "convenience.json"
test_file = self.temp_dir / "convenience.txt"
self.create_test_file(test_file, "convenience test")
modtime = os.path.getmtime(test_file)
# Test convenience function
result = has_changed(test_file, modtime, cache_file)
self.assertFalse(result, "Convenience function should work correctly")
def test_cache_stats(self) -> None:
"""Test cache statistics functionality."""
cache_file = self.cache_dir / "stats.json"
cache = FingerprintCache(cache_file)
# Initially empty cache
stats = cache.get_cache_stats()
self.assertEqual(stats["total_entries"], 0)
self.assertFalse(stats["cache_file_exists"])
# Add some entries
test_file = self.temp_dir / "stats.txt"
self.create_test_file(test_file, "stats content")
cache.has_changed(test_file, time.time() - 1)
# Check stats after entries
stats = cache.get_cache_stats()
self.assertEqual(stats["total_entries"], 1)
self.assertTrue(stats["cache_file_exists"])
self.assertGreater(stats["cache_file_size_bytes"], 0)
def test_clear_cache(self) -> None:
"""Test cache clearing functionality."""
cache_file = self.cache_dir / "clear.json"
cache = FingerprintCache(cache_file)
# Add an entry
test_file = self.temp_dir / "clear.txt"
self.create_test_file(test_file, "clear content")
cache.has_changed(test_file, time.time() - 1)
# Verify cache has entry
self.assertEqual(len(cache.cache), 1)
self.assertTrue(cache_file.exists())
# Clear cache
cache.clear_cache()
# Verify cache is cleared
self.assertEqual(len(cache.cache), 0)
self.assertFalse(cache_file.exists())
def test_build_system_workflow(self) -> None:
"""Test complete build system workflow."""
cache_file = self.cache_dir / "build.json"
cache = FingerprintCache(cache_file)
# Simulate source files
source_files = [
self.temp_dir / "main.cpp",
self.temp_dir / "utils.cpp",
self.temp_dir / "config.h",
]
# Create initial files
for i, src_file in enumerate(source_files):
self.create_test_file(src_file, f"source content {i}")
# First build - all files should be "changed" (new)
changed_files: List[Path] = []
baseline_time = time.time() - 3600 # 1 hour ago
for src_file in source_files:
if cache.has_changed(src_file, baseline_time):
changed_files.append(src_file)
self.assertEqual(
len(changed_files), 3, "All new files should be detected as changed"
)
# Second build - no changes
current_modtimes = [os.path.getmtime(f) for f in source_files]
changed_files: List[Path] = []
for src_file, modtime in zip(source_files, current_modtimes):
if cache.has_changed(src_file, modtime):
changed_files.append(src_file)
self.assertEqual(
len(changed_files), 0, "Unchanged files should not be detected as changed"
)
# Third build - modify one file
self.modify_file_content(source_files[0], "modified main.cpp")
changed_files: List[Path] = []
for src_file, modtime in zip(source_files, current_modtimes):
if cache.has_changed(src_file, modtime):
changed_files.append(src_file)
self.assertEqual(len(changed_files), 1, "Only modified file should be detected")
self.assertEqual(
changed_files[0], source_files[0], "Should detect the correct modified file"
)
def run_all_tests() -> bool:
"""Execute all test scenarios."""
import unittest
# Create test suite
suite = unittest.TestLoader().loadTestsFromTestCase(TestFingerprintCache)
# Run tests
runner = unittest.TextTestRunner(verbosity=2)
result = runner.run(suite)
if result.wasSuccessful():
print("✅ All fingerprint cache tests passed!")
return True
else:
print("❌ Some tests failed!")
return False
if __name__ == "__main__":
run_all_tests()
Loaded 100 of 1839 files, more files were not shown because too many files have changed in this diff. Show more